The Review Layer Is the Bottleneck in Agentic Coding
11 min read#AI#Engineering#Product#Culture
For a while, the question around AI-generated code was simple:
Can I trust it?
My answer used to be: not really. Not without reading every line, running the tests myself, and keeping one hand close to the rollback button.
That answer is changing.
I do trust AI to write useful code now. I have seen it take on real work, not just toy examples and one-file demos.
But that does not mean I trust every change it makes.
The question I keep coming back to now is harder:
How do I build a pipeline that makes AI-generated code trustworthy enough to review, merge, and eventually ship to production?
That is the problem I have been testing inside my own agent sandbox.
I tried it on two very different kinds of work
First, I used the sandbox on a greenfield project.
Greenfield work is the friendliest possible setting for an agent. There is less history to misunderstand, fewer invisible contracts, and more room to choose a clean path forward. The agent can help set up the project, build a first slice of the product, add tests, and move quickly.
Then I gave it a much less friendly problem: a large migration in an existing codebase.
That is where things get real.
Migrations have old assumptions everywhere. A type that looks safe to rename might be used by a job nobody has touched in two years. A test may pass while a customer-facing edge case quietly changes shape. The agent can read a lot, search a lot, and make a very convincing plan. But it still does not automatically know which pieces of history matter.
In both cases, the agents produced useful work.
The bottleneck was not code generation.
It was review.
The greenfield work was usually easy to reason about. The project was young, the surface area was small, and a lot of the decisions were still mine to make.
The migration felt different.
Nothing had to be obviously broken for me to slow down. The pull request could look tidy. The tests could be green. Copilot could leave a useful review. And I could still find myself asking, “Okay, but what old behavior did we just assume does not matter anymore?”
That is the kind of question that turns a five-minute review into an afternoon of detective work.
It also made the problem very clear: I do not need the agent to write less code. I need the system to make the important parts of a change easier to see.
More code means more things to understand
This sounds obvious, but I think it changes the workflow more than people admit.
When an agent can produce a substantial pull request in a short amount of time, the problem is no longer waiting for implementation. The problem is deciding whether the implementation is actually safe.
If I ask an agent to make a migration, it can give me:
- a plan
- a set of changed files
- new tests
- a pull request description
- a summary of its assumptions
That is useful.
But it is also a lot to absorb.
Reading a large AI-generated diff line by line is not a scalable review strategy. It turns the human into a slow parser of code the machine just produced at high speed.
That is not where I want my attention to go.
The human reviewer should be spending time on the parts that require context and judgment:
- is this the right product behavior?
- did the migration preserve the important old behavior?
- are we changing more than the ticket asked for?
- what could fail in production even if the tests are green?
- is the code making the next change easier or harder?
The system needs to make those questions easier to answer.
A review is not one thing
Inside the sandbox, I started treating review as a layer instead of a final step.
The first layer is mechanical verification.
Can the project build? Do the tests pass? Did the type checker, linter, formatter, and any relevant security checks run? Are there migration checks or contract tests that should be part of the definition of done?
None of that proves the change is correct. But it removes a lot of avoidable noise before a person has to look at it.
The second layer is an independent AI review.
I have GitHub Copilot review every pull request. The point is not to collect another green checkmark. The point is to get a second pass from outside the implementation context.
The agent that wrote the code usually has a story about why the code makes sense. A reviewer is more useful when it is asked to look for the things that story may have missed:
- missing edge cases
- unsafe assumptions
- tests that prove the happy path but not the behavior we care about
- unexpected scope changes
- places where the migration is incomplete
This catches useful things. It also produces comments that need judgment. A review bot can be confident and still be wrong. It can find a real style issue while missing the important product risk sitting next to it.
So AI review is not a replacement for human review.
It is a way to give human review a better starting point.
It also needs to be a genuinely different pass.
If the same agent writes the code, explains the code, reviews the code, and tells me it is all good, that is not much of a review system. That is one very confident coworker talking to itself.
For now, I want separate jobs, separate prompts, and a review checklist that asks a different set of questions from the implementation prompt. For sensitive changes, I would rather use another model or another kind of check as well. The point is not to create fake independence. The point is to reduce the chance that every layer misses the same thing.
Not every change deserves the same amount of trust
One thing I am adding to the sandbox is a simple risk tier before the work starts.
A small copy change, an isolated UI fix, or a well-tested internal tool can move through a lighter path. The agent can make the change, run the checks, get an automated review, and leave a clean pull request for a quick human look.
A migration, authentication change, billing flow, permissions update, or anything that touches production data belongs in a different lane.
For those changes, I want more proof:
- a migration rehearsal against realistic data
- contract and integration checks, not just unit tests
- a staging environment or a safe preview
- a rollback plan before merge, not after something goes sideways
- an explicit human approval before anything reaches production
This is not me being anti-agent.
It is just matching the amount of trust to the amount of damage a bad change could cause.
The agent should be allowed to move quickly on low-risk work. It should also know when it has reached a boundary where it needs to stop and ask.
Cost is part of whether the workflow works
There is another constraint that gets ignored when people talk about agentic coding: most of us do not have an Anthropic-sized token budget.
Someone asked me a version of this recently:
“Why are you building all of that? Why not just buy more credits from Anthropic or OpenAI and let the agents run?”
It is a fair question.
But I do not think more credits are the whole answer.
When we write software by hand, we already think about complexity. We break down a messy feature before it becomes a messy implementation. We do not hand an unclear problem to five engineers, tell them to work on it for a week, and hope the total bill means the answer must be good.
Agent work is the same kind of problem in a different form.
The resource is no longer only developer time. It is also context, model calls, retries, and human attention at the end of the run. If we do not manage that complexity, buying more credits just gives an unclear task a larger allowance to wander around.
It is easy to say, “just run more agents” or “buy more credits.” That might work for a team with a large budget and a system that is already well understood.
For a normal developer or a small team, it is not a strategy.
Every long investigation, retry, and parallel agent run has a real cost. And expensive work is not automatically good work. An agent can spend a lot of tokens confidently exploring the wrong part of a codebase.
So I want the sandbox to make a cost decision before it makes a code decision.
Not a perfect prediction. Just a useful first estimate based on the shape of the task:
- How many files and systems are likely involved?
- Is this a small, well-defined fix or an open-ended investigation?
- Does it touch a stable area of the codebase or a legacy system with unclear dependencies?
- Are the acceptance criteria and tests already clear?
- Does the task involve a migration, production data, or external services?
- Would a short human investigation save the agent from guessing for an hour?
That gives the task a rough complexity and risk level before an agent starts burning through a large context window.
For a small task, the right answer might be one implementation agent, one focused review, and a hard time limit.
For a medium task, it might be a short research pass first. The agent can map the files, dependencies, and tests, then come back with a plan and an estimate before it gets permission to edit anything.
For a large migration, the right answer may be to split the work into small checkpoints or ask for a human decision before the implementation run begins.
The important thing is that the system has a budget.
Each run should know its time limit, token or credit limit, retry limit, and the condition that makes it stop. When it reaches that boundary, it should leave behind the useful context it gathered instead of continuing because it has no idea how to quit.
This is not about squeezing every last token until the number looks good on a dashboard.
It is about getting the most verified progress for the amount of money and human attention available.
If a task is too ambiguous or too expensive to automate well, the system should say that early. A five-minute human clarification can be much cheaper than a very determined agent spending an afternoon making increasingly creative guesses.
What I want the pull request to explain
The pull request itself should do more of the work.
I do not want to open a large diff and reconstruct the entire change from scratch. I want a review packet that answers the basic questions before I start reading implementation details.
For each agent-created pull request, I want to see:
- The goal — what problem the change is meant to solve, in plain language.
- The plan — the approach the agent chose and why.
- The changed surface area — the main files, systems, or behaviors touched.
- The evidence — tests, checks, screenshots, logs, or migration results.
- The assumptions — what the agent inferred because the ticket or codebase did not state it clearly.
- The risk — what could still be wrong, what was not tested, and where human attention is most valuable.
That last one matters.
I do not need an agent to pretend it is certain. I need it to be honest about uncertainty.
An agent that says, “I changed these twelve files and all tests passed,” is helpful.
An agent that says, “I changed these twelve files, the tests passed, but I could not verify whether this legacy job runs against the old schema in production,” is much more helpful.
That is a reviewable change.
The metrics I care about are review metrics
I am less interested in counting how many lines of code an agent can generate. That number is easy to make look impressive and not very useful on its own.
The numbers I want to watch are closer to these:
- How long does it take from pull request opened to a human being confident enough to approve it?
- How many Copilot findings turn into real fixes?
- How many human review comments find something the automated layer missed?
- How often does a change need another agent pass after review?
- How many issues are found after merge or after release?
- Which kinds of tasks create the most review work: greenfield features, refactors, or migrations?
Those numbers would tell me whether the pipeline is earning trust.
I do not have a clean dashboard full of numbers yet. I am still early enough that I would rather be honest about that than pretend one green PR proves the whole workflow.
For now, I am collecting the boring evidence: what the agent changed, what Copilot found, what I found, what took a long time to verify, and what I would ask the system to do differently next time.
If agents are producing ten pull requests a day but each one takes an hour of tense human review, then I have not really solved anything. I have just moved the bottleneck.
On the other hand, if the system can make a small, well-scoped change easy to understand, show me the evidence, surface its own uncertainty, and give me a focused review, then it is doing something valuable.
My target has changed
At first, I wanted an agent sandbox that could keep working when I was away from my laptop.
I still want that.
But I do not think the important part is making the agents run longer or write more code. The important part is building a reliable review layer around them.
The sandbox should help turn generated code into a decision a human can make with enough confidence.
And every review should leave the system a little better than it was before.
If a human finds a missing edge case, that should become a regression test. If Copilot keeps flagging the same repository convention, that should become an instruction the agent sees before it starts. If a migration review takes too long because nobody can tell which jobs depend on an old table, that is not just a review problem. It is a clue that the codebase needs better documentation or a better check.
That feedback loop is probably the most important part.
Otherwise, I am just hiring the same agent to make the same kind of mistake faster every week.
That means:
- clear task boundaries before work starts
- isolated branches and repeatable environments
- tests and checks that match the real risk
- an independent AI review on every pull request
- a concise evidence-based summary for the human reviewer
- logs and an audit trail that explain what the agent actually did
- an escalation path when the system cannot prove enough
- no automatic path from a generated change to production
This is less exciting than talking about a swarm of agents coding all night.
It is also the part I think will matter more.
The future is not “AI writes the code, so we stop reviewing it.”
The future is that review becomes a system: part automation, part independent critique, part evidence, and part human judgment.
That is the layer I am trying to build now.