Fancy harnesses make coding agents more expensive but not better, study finds

A new controlled study evaluates how different coding-agent harnesses perform on standardized tasks. The harnesses examined include Claude Code, OpenCode, and mini-SWE-agent, which wrap large language models with different scaffolding, tool integrations, and system prompts to guide code generation and debugging.

The key finding is that pass rates across harnesses are remarkably similar. Switching from one harness to another moves results only marginally, often within the noise of simply rerunning the same configuration. This suggests that the underlying model capability, rather than the surrounding scaffolding, is the dominant factor in coding-agent performance.

Cost, however, tells a very different story. Per-task expenses can triple depending on the harness chosen. The primary driver is the system prompt, which is resent with every agent step. Longer or more complex prompts accumulate token costs rapidly over multi-step tasks, inflating bills without delivering proportional quality gains.

For teams deploying coding agents at scale, the study implies that harness selection should be guided more by cost efficiency and workflow fit than by expected quality improvements. Organizations may want to audit prompt lengths and step counts to control spending, rather than assuming a more elaborate harness will yield better results.

The broader takeaway is that the coding-agent market may be over-investing in scaffolding differentiation. As models improve, the marginal value of harness-level innovation could shrink further, shifting competitive advantage toward cost optimization and integration depth rather than raw performance claims.

This page provides an editorial summary based on publicly available information. It is not a republished article. Use the source link below for the original report.