Was that the model, or was it the harness?

An agent solves 62% of the tasks in a benchmark. A different agent solves 71%. The obvious conclusion is that the second one uses a better model.

Sometimes true. Often not — and the difference decides where your next month of effort should go.

Two systems, one number

Every agent result is the output of two things working together:

• The model — how well it reasons, plans, and writes correct code on the first attempt.
• The harness — everything wrapped around it. What tools it can call, how failures are retried, how context is managed as the run gets long, whether the work is verified before being called done.

A benchmark score is their product, and multiplication is lossy in a specific way: you cannot recover the factors from the result. A strong model in a thin harness and a weaker model in a well-built one can land on the same number, and the two situations call for opposite responses.

Why the harness contributes more than expected

Consider what a good harness does on a task the model gets wrong on the first try. It notices the test failed. It feeds the error back with enough context to be actionable. It tries a different approach rather than the same one. It verifies before reporting success.

Each of those is scaffolding, not intelligence, and together they convert a class of first-attempt failures into eventual successes. On multi-step tasks — where the chance of getting every step right first time is small — that recovery loop can account for a large share of the total.

Which means a meaningful part of what looks like model capability is error recovery you could have engineered yourself.

Separating them

The experiment is simple and rarely run. Vary one thing at a time.

Hold the model, vary the harness. Same model, four configurations: no retries; retries; retries plus tool access; retries plus tools plus a verification step. The spread across those four is your harness contribution, in your codebase, on your tasks.

Hold the harness, vary the model. Your best configuration, three models. That spread is the model contribution — and it is usually smaller than the first, which surprises people who have been buying their way up the tier list.

One number worth collecting while you are there: how often does the harness recover from a model mistake? A high recovery rate means your scaffolding is doing real work and a model upgrade will disappoint. A low one means the model is failing in ways your harness cannot catch, and better scaffolding is the cheaper fix.

Reading published scores

A public agent benchmark reports a pair — this model, in that harness — and only names one of them. Reproducing the score in your own setup can move it substantially in either direction, because your tools, retry policy and context management are not theirs.

Treat published numbers as evidence that a combination can reach a level under conditions you do not share. Your own two-axis test, on your own tasks, is worth more than any leaderboard, and it takes an afternoon.

What to do with the answer

If most of your gap is harness: it is engineering, you own it, and it compounds across every model you use afterwards. Better verification, better error feedback, better context management.

If most of it is model: the upgrade is worth buying, and you will get it without touching your code.

Guessing which one you are in is expensive in both directions — paying for capability you cannot use, or engineering around a limit the next release removes.