Everyone is building agents right now. Very few people are asking whether their agent should be allowed anywhere near production. I spend my days on exactly that boundary, in industries where a bad automated decision is not a funny screenshot but a stopped plant or a failed audit. Over time I’ve boiled my go/no-go thinking down to five questions. If you can’t answer all five, your agent is a demo. That’s fine. Demos are useful. Just don’t connect a demo to systems that matter.
1. What is the worst thing this agent can do?
Not the average case. The worst case. Sit down and write it out.
If your answer sounds like “well, it has access to the ticketing system, the asset database, and email, so worst case it could…” and you trail off, you already have your answer. The problem is not that you need better guardrails. The problem is that your agent’s scope is too big. An agent whose worst case you cannot state in one sentence is an agent doing too many jobs.
The fix is boring and it works: split the job. Give each piece of the work its own narrow agent with only the access that piece needs. Suddenly the worst-case question has small answers. “It could write a wrong record to the staging table” is a risk you can live with and monitor. “It could do anything” is not.
2. Is your data actually ready for this?
This is the question nobody wants to hear, because the answer is usually no.
An agent reasons over whatever you feed it. If your asset inventory has duplicates, your master data has three names for the same thing, and your source systems disagree with each other, the agent will produce confident nonsense on top of that mess. It will be fluent nonsense, which is worse than an error message, because fluent nonsense gets trusted.
In every serious agent project I have seen, the unglamorous data cleanup consumed more effort than anything to do with prompts or models. Budget for that upfront. If you cannot, fix the data first and build the agent later. The model can wait. Bad decisions made at scale cannot be un-made.
3. Where does a human stay on the hook?
Notice I did not ask “where do we put a human in the loop.” That framing leads teams to sprinkle approval steps everywhere, which feels safe and is actually dangerous. A person asked to approve two hundred routine items a day stops reading them by day three. You have not added safety. You have added a rubber stamp and a scapegoat.
The better question is about accountability. Which decisions must a named person own? In my world the answers are things like accepting a risk, approving a change to a production system, or closing a finding. Those keep a human, always. Everything else gets automated checks instead of human approvals. Fewer gates, but real ones, staffed by people who have the time to actually think at each one.
4. What does it do when it fails?
Every agent fails. The input is malformed, the source system is down, the situation is genuinely ambiguous. The only question is what happens next.
There is exactly one acceptable answer: it stops and asks. It puts the item in a queue for a person, with everything it knows attached, and moves on. The unacceptable answer is that it guesses, because a guessing agent fails silently, and silent failure in production is how small problems become incidents.
And watch that queue. A growing human queue is your early warning system. It tells you the automation is hitting something it does not understand, and it tells you before anything breaks. If nobody is looking at the queue, you have turned your safety net into a hiding place.
5. What does one decision cost, and can you say it out loud to your CFO?
Here is the conversation that kills more AI projects than any technical failure: someone senior asks what this will cost at full scale, and the room goes quiet.
If your agent design cannot tell you the cost per item processed, you do not have a production design yet. The good news is that the same discipline that answers question one answers this one too. Narrow agents doing narrow jobs consume a predictable amount of model usage per item. Predictable per-item cost times expected volume is a budget. A budget is something a business can approve. “It depends” is not.
There is a bonus hiding in here. Once you can see the cost per role, you notice that most of the routine work does not need your most expensive model at all. Save the big model for the few steps that genuinely need judgment. The bill drops, and nothing about the outcome gets worse.
The pattern behind the questions
Read the five questions again and you’ll notice none of them are really about AI. They are about scope, data, accountability, failure, and money. Which is the point I keep coming back to in everything I write: the hard part of AI is not the model, it is the thinking you do before you touch the model. Teams that skip the thinking build impressive demos that never ship. Teams that do the thinking ship boring systems that quietly run for years.
Boring systems that quietly run for years are the whole game.
This thinking-before-tooling argument is the subject of the book I am currently writing. More on that soon. If you are wrestling with an agent project and one of these five questions is giving you trouble, that discomfort is information. Listen to it before production does the listening for you.
SappersAI 
Leave a comment