Syed Tanveer Jishan.

Why the same AI model gets three different scores

Claude Opus 4.5 was tested with the same coding benchmark under three different harnesses and the scores were 45.9, 50.2, and 55.4, respectively. The harness is the software wrapper that sits between the model and the test, and it turns out that wrapper is worth more points than most model upgrades. So the leaderboard you are reading is partly a ranking of wrappers.

Published ·Updated

01 / 04

The same AI model, Claude Opus 4.5, took the same benchmark and came back with three different scores: 45.9, 50.2, and 55.4. Nothing about the model changed between those runs, and the test did not change either, because SWE-bench Pro is a fixed set of real software engineering tasks. The only thing that changed was the harness, i.e. the software wrapper that runs the model against the test. And that one change moved the score by 9.5 points, which is a bigger jump than most model upgrades deliver.

02 / 04

Basically, the model never takes the test alone. A layer of software sits between the model and the task, and on every single step that layer makes four decisions: what the model gets to see, which tools it is allowed to call, whether a failed attempt gets retried, and when to declare the work finished. Every one of those decisions moves the final score. So a benchmark score is always produced by two things working together, the model and the harness, but the leaderboard shows only the model's name, which means majority of the harness's work gets silently credited to the model.

03 / 04

Across the industry, swapping the harness moves scores far more than the model improvements everyone argues about. Holding the model fixed and changing only the harness shifts GPT-5 by 11 points and Kimi K2 by 15 points, and one leaderboard that ran the same models through many harnesses measured swings of up to 48 points. Meanwhile, when a paper announces a meaningful model improvement, it is usually announcing a gain of 2 to 4 points. So the item that nobody talks much about is routinely several times larger than the item everyone discusses.

04 / 04

So, what is the key takeaway? It is that the harness is not a detail you can ignore, it is part of what the score is measuring. So when you are reading a leaderboard, treat two models as comparable only if the same harness ran both of them, because otherwise the ranking is partly a ranking of wrappers. On top of that, when you are building something real, look at your harness before you reach for a bigger model, because the wrapper is worth 10 to 20 points, a model upgrade is usually worth 2 to 4, and the wrapper is the part your team fully controls. Basically, before you pay for a smarter model, check whether you are wasting the one you already have.

Sources and method

Every score in this piece is a published number, not an estimate of mine. The three Opus 4.5 scores are the same model run on SWE-bench Pro by three different harnesses: Scale's standardized SEAL scaffold at 45.9, a third agent system at 50.2, and Claude Code at 55.4. The 11 and 15-point figures hold the model fixed and change only the harness, and the 48-point figure is the largest single-model swing the Holistic Agent Leaderboard measured across harnesses. The 2-to-4-point reference is the size of gain that agent papers typically report as a meaningful model advance, as characterized by the cited position paper, and the 10-to-20-point figure is Scale's own attribution of score movement to harness choices.