Be honest: what do you actually do while the agent types?

I used to just watch. Not read, watch. Scroll the diff as it streamed in, nod at code I hadn't fully parsed, and tell myself I'd review it properly at the end. Sometimes I did. Often I didn't. The question "what do you do while AI codes?" keeps showing up in the dev.to feed, and the comments make it clear I'm not alone. The struggle to stop mid-task and just wait is real.

So I tried to fill the wait with something useful. I built a system where a second model tries to break the first model's work. Then I learned the thing that changed how I think about AI review: 89% of what I built was theater.

The first time I ran it, the debates were fake

I pointed two LLMs at the same pull request and asked them to argue. It worked beautifully: confident claims, point-by-point rebuttals, a clean verdict at the end.

Then I read the raw logs.

The second model wasn't analyzing anything. It was replaying pre-generated text. Both models were exchanging messages, no positions were changing, no new evidence was being cited. The transcript sounded like a debate. It was a recording of a debate.

If I hadn't looked at the raw output, I would have shipped it and called it a working system. That's the part that still bothers me.

The flaw in almost every "AI second opinion"

Here's what most multi-model review workflows actually do:

Model A reviews the artifact.
Model B reviews the artifact plus Model A's output.
You call the result independent review.

It isn't. It's a validation. The moment Model B's context contains Model A's verdict, B is anchored. It's doing social-pressure resistance, not analysis. Humans fail this constantly. That's why double-blind review exists. We invented that safeguard decades ago and keep leaving it out of AI pipelines.

The fix is mechanical: Model B cannot see Model A's output until B has fully committed its own position. Not prompted into independence. Structurally prevented from anchoring.

That one constraint is the entire project: AdversarialDebate.

What it looks like when the independence is real

Two models analyze in parallel with zero shared context. Each commits a verdict with structured claims and evidence. Then a bounded debate opens: every objection must reference a specific counter-claim, every claim must cite specific text from the artifact. When they can't resolve, the output is a structured disagreement report, both positions preserved and unresolved points documented.

That last part matters. Most multi-agent systems are engineered toward consensus. Disagreement is treated as a failure to be smoothed over. But if two independent reviewers reach different conclusions with different evidence, that tension is the signal. Collapsing it into one confident verdict throws away the exact thing you built two reviewers to find.

Theater rate across 217 debates in v0.2.0: 0%.

What the numbers actually looked like

I ran the first real field test on 70 actual pull requests from public repos, 411 debates. The core question: do the claims correspond to what actually went wrong in those PRs?

81% of debate claims matched documented PR outcomes. When it found an issue, it was usually the issue.
But I fixed 13 bugs before I trusted any of that: CSV parsers breaking on commas in PR descriptions, model slugs silently routing to the wrong provider, a join bug that collapsed the dataset from 2,333 rows to 359. None were architecture bugs. All were "you haven't run this on real repositories at scale" bugs.

By v0.2.0: 150 artifacts across four domains, 88.7% binary match against the corrected 2,333-row dataset, near-zero fabrications. Total cost: $0.42 for 360 reviewer runs.

Less than lunch. Compute was never the bottleneck. Prompt engineering and ground-truth measurement were.

The finding I didn't expect: pairing matters more than capability

The best pairing wasn't the two smartest models. It was GPT-4o-mini + Mistral Small 3.2, a small OpenAI model and a small European one.

GPT + Gemini produced the worst debates: lots of rounds, almost no concessions, no resolution. Two RLHF-heavy models from big US labs agree fast because they were trained to prefer harmony. GPT-mini + Mistral produced genuine disputes, real concessions, and eventual convergence.

My working theory: it's not raw capability that predicts debate quality, it's diversity of training objective. A later run refined this: the effect may be as much about non-Mistral models rubber-stamping each other, so the practical rule became "always include Mistral." I hold the theory loosely.

I have not proven this rigorously. It matches everything in my data, and it's why my model-selection advice starts with "pick from different labs" before anything about benchmark scores.

The 11.3% I can't explain away

88.7% sounds good. It is good, compared to single-pass review. But there's a problem:

False negatives are invisible.

If both models miss a real issue and converge on "looks fine," you get a clean verdict and no indication anything is wrong until it surfaces downstream. The 11.3% that's partial or wrong clusters in the narrative domains: incident reports, change proposals, where ground truth is fuzzier. Generic prompts that work on code don't port cleanly to a postmortem.

Requiring two independent models to both miss an issue is a much higher bar than single-pass review. It is not a guarantee, and I won't pretend it is.

So, back to the question

What do I do while AI codes? I don't watch anymore. I let a second model try to tear the first one's work apart while I do something that actually needs a human: reading the failure modes, writing the eval, deciding what "good" means.

It's not passive. It's a different kind of attention. And honestly, it's less lonely than watching a cursor.

So what do you actually do while the model types? Do you review live, review after, or have you found a way to use the wait?

Repo: adversarial-debate (MIT)
Prior posts: Most AI Second Opinions Are Fake · Most AI Second Opinions Are Theater

StackFeed

What Do You Do While AI Codes? I Make Mine Argue With Itself.