Blog · AI agents

FrontierHarness Eval v1.1: which agent harness works best with which model

FrontierHarness Eval is a benchmark that runs many coding-agent harnesses, such as Codex CLI, Claude Code, OpenCode and Pi, on the same terminal tasks and compares pass rate, cost and time. Version 1.1 came out on November 17, 2026, from Runta, the company whose runtime ran every trial. It tested 22 harness setups on five open-weight models, 27,280 runs in all. It has no results for Claude, GPT or Gemini models. This guide covers what changed, the authors' pick for each model, the numbers across all models, and how far to trust them.

FrontierHarness Eval v1.1 at a glance

FrontierHarness Eval v1.1
Made byRunta, which calls itself the execution layer for AI agents. Post by Shilin Zhu. Baseten served the models as a partner; the leaderboard also lists Z.ai as a sponsor
PublishedNovember 17, 2026 (v1.0 was September 1, 2026)
Versionv1.1, Harbor dataset frontier-harness-eval/frontier-harness-eval@v1.1.0
Runs27,280: 62 tasks × 22 configurations × 4 attempts × 5 models, using 83B tokens
Tasks62 command-line tasks (software engineering, science, data, systems, ML and security), with time limits from 15 minutes to 8 hours
ModelsFive open-weight models, all served by Baseten: GLM-5.3, GLM-5.3-Flash, DeepSeek-V4.1-Flash, Muse Spark Open and Kimi-K3
Harnesses22 configurations of 18 harnesses (DeepSeek Harness in 4 profiles, OpenCode in 2 versions), each on its default settings at maximum reasoning effort
What's measuredPass rate, average cost per task, cost per pass and time per task
Data and codeThe eval repo (branch release/v1.1) has the suite, adapters and launcher; tasks are public on Harbor Hub. No license is stated

What's new in v1.1

Version 1.0 tested one model, Kimi K3, with 12 harness configurations on 30 tasks, one attempt each: 360 trials. Version 1.1 is much bigger. The authors call it 76× the size of v1.0.

  • Five models instead of one, served by Baseten instead of Fireworks.
  • 22 harness configurations instead of 12.
  • 62 tasks instead of 30, and 4 attempts per task instead of 1.
  • New cost metrics: average cost per task and cost per pass (v1.0 used median cost per pass).
  • The main question changed too: for each model, which harness gives the best value.

Results by model: the authors' best-value pick

The authors' rule: the harness with the lowest cost per pass wins, unless another one costs at most 30% more per pass and passes at least 5 points more often. To test each pick, they resampled the tasks 4,000 times. The last column shows how often the pick stayed on top.

ModelBest-value harnessCost per passPass rateHow firm
GLM-5.3Codex CLI$0.80357.7%Firm: on top in 99.6% of resamples
GLM-5.3-FlashPi$0.22358.1%Weak: 42%
DeepSeek-V4.1-FlashZCode$0.22269.0%Weak: 47%. Picked over OpenCode v1, which cost 25% less per pass and passed 7.7 points less
Muse Spark OpenOh My Pi$1.21258.9%About even: 53%
Kimi-K3Alma$1.57964.1%Weak: 25%. Picked over Oh My Pi, which cost 9% less per pass and passed 6.5 points less
All modelsNo single pick. On the cost and pass-rate frontier: OpenCode v2, Pi, Codex CLI, ZCodeLowest: Codex CLI, $0.937Highest: ZCode, 61.9%-

Across all models: pass rate, cost and time by harness

Averaged over all five models, 1,240 scored attempts per harness. Sorted by cost per pass. Costs are estimates from Baseten list prices, not invoices. The authors say a gap of a few points in pass rate is a tie: the 95% interval for one harness on one model is about ±10 points.

This is 13 of the 22 rows. The full table, with each model separately and the other harnesses (Cline CLI, MiniMax Code, mini-SWE-agent, Maka, Exo and the four DeepSeek Harness profiles), is in the repo and the post.

HarnessPass rateCost per taskCost per passTime per task
Codex CLI61.1%$0.573$0.93726.8 min
Pi57.7%$0.563$0.97631.2 min
OpenCode v257.3%$0.561$0.97825.1 min
Oh My Pi57.3%$0.564$0.98334.0 min
ZCode61.9%$0.659$1.06335.4 min
KimiCode60.2%$0.653$1.08533.7 min
OpenCode v157.4%$0.646$1.12535.9 min
Hermes Agent59.9%$0.692$1.15535.0 min
Grok Build60.0%$0.723$1.20536.2 min
Alma59.7%$0.731$1.22434.1 min
GitHub Copilot CLI56.9%$0.787$1.38336.8 min
Claude Code58.9%$0.915$1.55541.3 min
Antigravity CLI55.0%$0.897$1.63236.8 min

What the numbers say

  • The model sets most of the cost. In the authors' analysis, the model explains 92.8% of the variance in cost per task and the harness 2.3%.
  • With the model fixed, the harness still matters a lot. It changed cost per task by 1.8–5.0× and cost per pass by 1.9–4.9×. The biggest gap was on DeepSeek, where Cline CLI cost 5.0× as much per task as OpenCode v1 for 2.0 more points.
  • Paying more doesn't buy a higher pass rate. Per model, the rank correlation between cost per task and pass rate ran from −0.39 to +0.22.
  • Steps predict cost; size doesn't. Steps per pass tracked cost per pass (+0.51). Start tokens, tool count and cache hit rate didn't. mini-SWE-agent, the smallest harness, ranked 19th of 22 on cost per task.
  • Most money goes to runs that fail. 64.5% of spend went to trials that didn't pass: 52.7% to failures and 11.8% to timeouts. With the five best-value harnesses, 68% (Pi) to 82% (Codex CLI) of failed runs ended with a confident final answer, not at the time limit.
  • Pass rates sit close together. Across all models they range from 50.6% (Exo) to 61.9% (ZCode), and most harnesses are within a few points of each other, which the authors treat as ties.
  • The five best-value harnesses have things in common: one shell tool doing most of the work, context that only grows, no subagents and almost no summarizing. The authors call these observations, not tests.

Claude Code and Codex in this eval

Both ran here only with open-weight models through Baseten, never with Anthropic's or OpenAI's own models. Read the results with that in mind.

  • Codex CLI (0.164.0) had the lowest cost per pass across all models, $0.937, and a top-tier pass rate, 61.1%. It was also the fastest of the high-pass-rate group at 26.8 minutes per task. With GLM-5.3 it was the authors' best-value pick and held in 99.6% of resamples.
  • Claude Code (2.1.308) passed 58.9% of tasks, mid-pack, but was slow and costly per pass with these models. It had the second-highest cost per pass ($1.555, after Antigravity CLI), the longest time per task (41.3 minutes) and the second-highest timeout rate (14.2%).
  • Its single most expensive pairing was with Muse Spark Open: $3.753 per pass, the highest of all 110 harness and model cells.
  • With DeepSeek-V4.1-Flash, Claude Code did well: 68.1%, second on that model after ZCode's 69.0%, at $0.285 per pass.

What it doesn't tell you

  • Nothing about closed models. v1.1 tested only open-weight models. It has no results for Claude Opus or Fable, GPT or Gemini models, and its rankings may not carry over to them. The authors list Opus 5.5, GPT-6 Astra and Gemini 4 Carbon as future work. For those models, see GPT-6.2 vs Claude.
  • Some harnesses are missing, including Cursor, Gemini CLI and Droid.
  • Costs are estimates. They're Baseten list prices times the tokens the provider reported. Baseten's caching and prices drive the cost ranks, which the authors say themselves. Your bill on another provider or plan may differ.
  • The maker ran it. Every trial ran on Runta's own runtime, and Runta's founder said so in the Hacker News thread. The data and code are public, so others can check.
  • Small gaps are noise. The interval for one harness on one model is about ±10 points, so most pass-rate differences in the tables are ties.
  • One kind of task. All 62 tasks are command-line tasks, each run in a fresh, short-lived runtime with default harness settings. Your repo, your prompts and your settings may favor a different harness.

How to read it for your team

  • Pick the model first. It drives most of the cost.
  • When you change models, test the harness again. The authors' own advice is to think about changing the harness too.
  • Use the per-model pick, not the overall average, and check how firm it is. Codex CLI on GLM-5.3 held up; the picks for GLM-5.3-Flash, DeepSeek and Kimi-K3 often didn't.
  • Compare cost per pass, not cost per task. A cheap run that fails still costs money.
  • Run your own tasks. The authors suggest a private set of your own tasks to test candidates, which tells you more than any public table.
  • Watch steps and failed runs. Harnesses that take fewer steps cost less, and checking work before an agent stops may save some of the money spent on failures. The authors didn't test that last idea.
With Crewmark

Compare harness and model pairs on your own issues

FrontierHarness Eval measures one agent harness at a time. Crewmark is the harness for a team of agents: each AI employee is its own coding-agent session with a role, a goal, working hours and a heartbeat. So you can run different harness and model pairs side by side on one board and see which one closes your issues.

  1. Add employees on different harnesses

    Claude Code and Codex employees run natively. Other agents, such as OpenCode and DeepSeek Harness, connect through ACP (via acpx) on your computer. See Claude Code vs Codex and OpenCode alternatives for the options.

  2. Give them the same kind of work

    Use the same role for each, such as the AI software engineer, and assign similar issues from your backlog.

  3. Compare on the board

    Each employee wakes on its heartbeat, works its issues and keeps their status current. Check which issues each one finished and what it cost on your own provider or plan.

FAQ

What is FrontierHarness Eval?

A benchmark that runs coding-agent harnesses on the same 62 command-line tasks and compares pass rate, cost and time. Version 1.1 tested 22 harness configurations on five open-weight models, 27,280 runs in all.

Who made FrontierHarness Eval?

Runta, a company that builds a runtime for AI agents. Every trial ran on Runta's own runtime, which its founder disclosed on Hacker News. Baseten served the models, and the leaderboard lists Z.ai as a sponsor.

Which agent harness is best?

The authors don't name one overall winner. They pick a best-value harness per model: Codex CLI for GLM-5.3, Pi for GLM-5.3-Flash, ZCode for DeepSeek-V4.1-Flash, Oh My Pi for Muse Spark Open and Alma for Kimi-K3. Only the Codex CLI pick held firmly when they resampled tasks. Across all models, Codex CLI had the lowest cost per pass and ZCode the highest pass rate.

Does FrontierHarness Eval include Claude or GPT models?

No. v1.1 tested only open-weight models: GLM-5.3, GLM-5.3-Flash, DeepSeek-V4.1-Flash, Muse Spark Open and Kimi-K3. Claude Code and Codex CLI were tested as harnesses, but only with those models. The authors list Opus 5.5, GPT-6 Astra and Gemini 4 Carbon as future work.

Is FrontierHarness Eval open source? Where is the data?

The code and dataset manifest are on GitHub at frontier-harness-eval/eval, branch release/v1.1, and the tasks are public on Harbor Hub. No license is stated for the repo or the data.

What is an agent harness?

The software around a model that makes it a working agent: the loop, the tools, the context and the permissions. Claude Code, Codex CLI and OpenCode are harnesses. FrontierHarness Eval keeps the model fixed and swaps the harness to see how much it changes the result.

How much did FrontierHarness Eval cost to run?

The authors don't state a total. They report 83B tokens over 27,280 runs, and per-task costs estimated from Baseten list prices, from about $0.11 to $1.98 per task depending on the harness and model.

Checked against Runta's v1.1 post, the eval repo (release/v1.1) and the leaderboard on November 18, 2026.

Run your harnesses as a team.

Crewmark turns Claude Code, Codex and other coding-agent sessions into AI employees with roles, goals, working hours and a heartbeat.