Skip to content
Star16Hire me
01 / StartXP 0%
Sep 8, 2026 · 1 min · by Ahmed Mamdouh

What actually breaks when AI agents go to production

#ai#agents#mcp#distributed-systems

The gap between an agent demo and an agent in production shows up on the third tool call, when the API returns a 500 and the agent has already written half a row to your database.

Demos are one happy path

A demo shows one happy path. Production means partial failure, retries that must not double-charge anyone, and a clear rule for when the agent stops and asks a person.

I see plenty of agent architecture diagrams. I rarely see anyone talk about the three things that actually break.

State

An agent that runs for ninety seconds and calls six tools is a long-running transaction with no transaction manager. If it dies on step four, something has to know that steps one to three happened. Most demos keep that in memory and hope. In production the progress needs to be persisted somewhere a restarted run can read it.

Partial failure

A tool call times out. Maybe it never ran, or maybe it ran and the response got lost. Those need different handling, and the agent usually can't tell them apart, so treat a timeout as an unknown outcome. That's why every side-effecting tool needs an idempotency key, the same way every queue consumer does.

Knowing when to stop

This one is the hardest. An agent that asks a human too often is a slow form, and an agent that never asks is a liability. Where the threshold sits is a product decision. It belongs in code where you can test it, and a line in the prompt is a poor place to keep it.

Nothing new

None of this is new. It's retries, idempotency and backpressure, which backend engineers have been doing for twenty years, applied to a new kind of caller. In my experience the model is the easy part.

Building something like this?

I'm Ahmed Mamdouh, a senior full-stack and AI engineer. I reply within one working day.

More posts