Two weeks ago, I published "Why Agent Evaluation Is Harder Than Model Evaluation." The core argument: with agents, you are not just judging an answer. You are judging a run. The path matters. The tools matter. The safety boundaries matter. I ended it by saying I would share the repo when it was ready.

It is ready now. AgentEval Forge is public, on PyPI, and this launch is really a report on what I learned while trying to build a believable way to validate agents.

I thought I was mostly building a scoring system. Real agents turned into the tsunami I was not prepared for, and the project became an integration reality check much faster than I expected.

What I Actually Built

I did not want to build a thin wrapper around an existing eval framework and call it a launch. So I went deep.

The PRD defines 20 Critical User Journeys across launch scenarios, regression workflows, adversarial case generation, and CI gating. The spec details an architecture with five core components: a scenario pack engine, a runner, a scoring layer with 17 deterministic checks and 11 LLM-as-judge metrics, a regression engine, and an adversarial generator. The WBS breaks into 118 tasks across 12 milestones: M0 scaffold through M12 launch.

I built adapters for five agent surfaces: subprocess, Python import, HTTP, LangGraph, and PydanticAI. Each adapter implements a thin contract: the agent receives a restricted invocation payload — scenario input, allowed tools, disallowed tools, and a budget. It never sees the expected answer or the scoring thresholds. No ground-truth leakage.

I built a security model: sandbox mode, trust policies, audit trail, API key sanitization. I built CI integration: GitHub Actions, GitLab CI, Docker sandbox. I built the docs: a user guide, a scoring guide, a scenario authoring guide, and the full field test reports with raw data.

The full capability list is in the repo README. If you want the deeper product/design docs, they are in the PRD, spec, scoring guide, and scenario guide. But the headline is: 20 launch scenarios across 10 families, 8 security scenarios, 17 deterministic scorers, 11 LLM-as-judge metrics, five framework adapters, and a product hierarchy where safety failures trump everything else.

Field Testing Changed The Project

I did not want to ship a test harness that only works against examples I wrote myself. But field testing was not part of the original plan. It got added ad hoc, late in the build, because I started getting nervous that unit tests and mock agents were hiding real integration problems. They were.

Sourcing real agents from GitHub sounded straightforward. Search for "langgraph agent" and "pydantic-ai agent," pick a few, run them. It was not straightforward.

I searched 150+ repos. Most were not viable. Large frameworks and platforms were too heavy. Database-bound agents required infrastructure I could not provision per-agent. Agents with no clear entry point, no run() method, no message surface, those just sat there silently. Agents behind an API gateway needed keys I did not have. I landed on 19 that looked testable: 11 LangGraph, 8 PydanticAI. Even those needed work.

Why I Bucketed Agents By Stars

I also had to create field-test buckets so the results would mean something. High-star agents were the reliability bucket: widely used repos that should mostly test my tool more than I test them. If EvalForge cannot even integrate with those, that is my problem first. Medium-star agents were the mutual-friction bucket: mature enough to be real, messy enough that the test cuts both ways. Sometimes I would find a weakness in their packaging or bootstrap; sometimes I would find a bad assumption in my harness. Low-star agents were the stretch bucket: experimental repos where I expected more chaos, but also where EvalForge could show obvious value by making a rough agent easier to evaluate, compare, and improve.

Each agent got its own configuration file, its own scenario pack, its own virtual environment. I tested across three tiers: local, cheap, and better. Because I was worried about token costs from the beginning, I put real effort into a local tier instead of treating it like a fake demo mode. On my machine that meant MLX serving Qwen3.5-9B-MLX-4bit, which fits comfortably on my Apple Silicon setup and was good enough to make local sweeps worth doing when the endpoint was healthy. The cloud tiers were gpt-4o-mini for cheap and gpt-4o for better.

That was the moment the clean version of the story broke. I thought I was mostly building a scoring system. Real agents forced the project to become an integration reality check.

What I Focused On — And What I Learned From The PRD

The PRD was not an afterthought. It is the thing that kept me honest.

The Product DNA section forced me to answer uncomfortable questions before I wrote a line of code. Who is this for, in order? Solo OSS builders first, small teams second, platform teams third. What is the smallest meaningful adoption outcome? Catch one regression before merge. What is the evaluation hierarchy? Safety > Correctness > Efficiency.

That last one was a forcing function. It meant I could not design a scoring system where a clean answer at the end washes away a policy violation on the way. Safety failures always fail the run. Correctness regressions warn by default but do not block unless you configure them to. Efficiency regressions are informational. That hierarchy is now baked into every scorecard the system produces.

The 20 CUJs were the other forcing function. A CUJ is not a feature. It is a specific moment where the product either earns its keep or does not. "An individual developer wants to evaluate a candidate agent version against a baseline." "A team lead wants to add a new scenario pack for a domain-specific tool." "A CI pipeline needs to block a PR when safety scores drop below threshold." Every milestone had to justify itself against at least one CUJ.

That discipline slowed me down in ways I am now grateful for. It stopped me from building a general-purpose eval library that does everything adequately and nothing well.

That was one of the earliest real learnings in the project: if the product cannot help someone catch one meaningful regression before merge, then the rest of the architecture is mostly decoration.

What the field test taught me

Here is what I expected, what I did not, and what surprised me most.

What I expected

I expected some agents to struggle with specific scenario families. I expected the better-tier judge to be more discriminating than the cheap-tier judge. I expected most agents to pass most scenarios, with the interesting signal being which ones failed and why.

None of those turned out to be true.

What I did not expect

The pass rate was 9%. Across all 19 agents and 95 scenarios, only 9 passes. That does not mean the roster is full of weak agents. It means the field test is currently measuring adapter realism more than agent quality. When a wrapper keeps the harness alive but returns blank completions, the judge scores it zero. That is a compatibility failure, not a quality failure. But it still dominates the headline number.

The most expensive model added nothing. I ran both cloud tiers against all 19 agents. Cheap and better produced identical outcomes: 9/95 passes on both. The better tier did not surface a single regression or improvement that the cheap tier missed. The bottleneck is not the judge model. The bottleneck is how faithfully the harness exercises real agent logic. When the adapter quality is low, the judge sees weak output and scores it low — regardless of which model is behind the judge.

Config chaos was the real blocker. One agent hardcoded ChatOpenAI() at module scope with a model name my local MLX endpoint does not serve. Patching required a monkeypatch on __init__ that runs before import. Another wrote to /root at import time. Three agents brought ormsgpack whose C extension ABI did not match my Python runtime — they were quarantined. One repo had pyproject.toml in a subdirectory; uv sync at the root was a silent no-op. I ended up writing eight compatibility wrapper modules, creating three pyproject.toml files for repos with broken packaging, and removing one agent entirely because it was stuck on an outdated PydanticAI API version.

This is just what happens when you run software you did not write, on a machine you did not set up. It is also why most agent evaluation stays inside the author's environment, against the author's examples, with the author's model. That is comfortable. It is the worst possible way to evaluate whether an agent actually works.

The one that worked

lg-mcp-agents, a LangGraph multi-agent repo that was previously non-runnable from outside its original Streamlit app, achieved 5/5 passes on both cloud tiers after targeted adapter work.

That result matters because it tells you the adapter is exercising real agent behavior. When you see a 5/5 from an agent you could not even import two days earlier, the integration is real.

Why field testing was a late discovery — and should not have been

The field harness was not in the WBS. I added it because unit tests and mock agents were passing cleanly and that felt wrong. The mock agents were too well-behaved. They did exactly what the adapter contract said. Real agents do not.

Real agents transitively import ffmpeg. They create venvs with a different Python version than yours. They write to absolute paths. They hardcode API keys at module scope. They nest their project files in subdirectories. Mock coverage caught none of this because mock coverage does not run real code.

The lesson is not "add field testing." The lesson is that unit and mock coverage is necessary but not sufficient for an evaluation harness. Plan for a real-agent field layer from the start. It will be the only test layer that catches integration reality.

That is probably the biggest learning from the launch: agent evaluation gets discussed like a scoring problem, but in practice it becomes an environment, adapter, and realism problem much faster than most teams expect.

I am not done

This is v0.1.0. I want to be clear about that.

The field test proves that AgentEval Forge can import, configure, invoke, and score real third-party agents from GitHub at scale. But it does not yet prove that the scorer can rank agents meaningfully across a large, heterogeneous roster. Some of the PydanticAI wrappers showed blank completions across all scenarios — the harness ran, but the agent did not produce real output. That is a compatibility achievement, not an evaluation one. I am keeping it honest because I think this is the kind of thing that gets hand-waved away in launch announcements and then discovered painfully by users.

The blank-completion behavior in some PydanticAI wrappers needs investigation. The ormsgpack C extension mismatch quarantined three otherwise viable agents and needs a containerized workaround. The field test roster needs to grow: more agents, more diversity, more corner cases. The adapter quality needs to improve until field-level compatibility crosses over into evaluation-level usefulness. The SWE-bench and WebArena connectors are built and verified end-to-end, but they are not yet integrated into the default CI path.

And the article series is only 1/5 done. I wrote about why agent evaluation is harder than model evaluation. Now the launch has given me better material for the next pieces: what scenario packs should actually look like, how to think about trajectory scoring, where adversarial testing becomes useful, and why cost-quality tradeoffs are often less important than integration realism.

What comes next

The repo is at github.com/deghosal-2026/agent-eval-forge. It installs with pip install agent-eval-forge. The README has the full capability list, and the repo docs include the user guide, field test report, and hard-won lessons.

If you are building agents, I have a question for you: what is your current evaluation workflow? Are you still eyeballing a few demos and calling it done? Have you tried scoring trajectories yet, or are you still just checking the final answer?

And if you have tried to evaluate agents at scale — especially third-party agents you did not write yourself — what broke first?

StackFeed

I Built an Agent Eval Harness. Real Agents Broke the Clean Version of the Story