FrontierHarness Eval v1.1 at a glance
| FrontierHarness Eval v1.1 | |
|---|---|
| Made by | Runta, which calls itself the execution layer for AI agents. Post by Shilin Zhu. Baseten served the models as a partner; the leaderboard also lists Z.ai as a sponsor |
| Published | November 17, 2026 (v1.0 was September 1, 2026) |
| Version | v1.1, Harbor dataset frontier-harness-eval/frontier-harness-eval@v1.1.0 |
| Runs | 27,280: 62 tasks × 22 configurations × 4 attempts × 5 models, using 83B tokens |
| Tasks | 62 command-line tasks (software engineering, science, data, systems, ML and security), with time limits from 15 minutes to 8 hours |
| Models | Five open-weight models, all served by Baseten: GLM-5.3, GLM-5.3-Flash, DeepSeek-V4.1-Flash, Muse Spark Open and Kimi-K3 |
| Harnesses | 22 configurations of 18 harnesses (DeepSeek Harness in 4 profiles, OpenCode in 2 versions), each on its default settings at maximum reasoning effort |
| What's measured | Pass rate, average cost per task, cost per pass and time per task |
| Data and code | The eval repo (branch release/v1.1) has the suite, adapters and launcher; tasks are public on Harbor Hub. No license is stated |
What's new in v1.1
Version 1.0 tested one model, Kimi K3, with 12 harness configurations on 30 tasks, one attempt each: 360 trials. Version 1.1 is much bigger. The authors call it 76× the size of v1.0.
- Five models instead of one, served by Baseten instead of Fireworks.
- 22 harness configurations instead of 12.
- 62 tasks instead of 30, and 4 attempts per task instead of 1.
- New cost metrics: average cost per task and cost per pass (v1.0 used median cost per pass).
- The main question changed too: for each model, which harness gives the best value.
Results by model: the authors' best-value pick
The authors' rule: the harness with the lowest cost per pass wins, unless another one costs at most 30% more per pass and passes at least 5 points more often. To test each pick, they resampled the tasks 4,000 times. The last column shows how often the pick stayed on top.
| Model | Best-value harness | Cost per pass | Pass rate | How firm |
|---|---|---|---|---|
| GLM-5.3 | Codex CLI | $0.803 | 57.7% | Firm: on top in 99.6% of resamples |
| GLM-5.3-Flash | Pi | $0.223 | 58.1% | Weak: 42% |
| DeepSeek-V4.1-Flash | ZCode | $0.222 | 69.0% | Weak: 47%. Picked over OpenCode v1, which cost 25% less per pass and passed 7.7 points less |
| Muse Spark Open | Oh My Pi | $1.212 | 58.9% | About even: 53% |
| Kimi-K3 | Alma | $1.579 | 64.1% | Weak: 25%. Picked over Oh My Pi, which cost 9% less per pass and passed 6.5 points less |
| All models | No single pick. On the cost and pass-rate frontier: OpenCode v2, Pi, Codex CLI, ZCode | Lowest: Codex CLI, $0.937 | Highest: ZCode, 61.9% | - |
Across all models: pass rate, cost and time by harness
Averaged over all five models, 1,240 scored attempts per harness. Sorted by cost per pass. Costs are estimates from Baseten list prices, not invoices. The authors say a gap of a few points in pass rate is a tie: the 95% interval for one harness on one model is about ±10 points.
This is 13 of the 22 rows. The full table, with each model separately and the other harnesses (Cline CLI, MiniMax Code, mini-SWE-agent, Maka, Exo and the four DeepSeek Harness profiles), is in the repo and the post.
| Harness | Pass rate | Cost per task | Cost per pass | Time per task |
|---|---|---|---|---|
| Codex CLI | 61.1% | $0.573 | $0.937 | 26.8 min |
| Pi | 57.7% | $0.563 | $0.976 | 31.2 min |
| OpenCode v2 | 57.3% | $0.561 | $0.978 | 25.1 min |
| Oh My Pi | 57.3% | $0.564 | $0.983 | 34.0 min |
| ZCode | 61.9% | $0.659 | $1.063 | 35.4 min |
| KimiCode | 60.2% | $0.653 | $1.085 | 33.7 min |
| OpenCode v1 | 57.4% | $0.646 | $1.125 | 35.9 min |
| Hermes Agent | 59.9% | $0.692 | $1.155 | 35.0 min |
| Grok Build | 60.0% | $0.723 | $1.205 | 36.2 min |
| Alma | 59.7% | $0.731 | $1.224 | 34.1 min |
| GitHub Copilot CLI | 56.9% | $0.787 | $1.383 | 36.8 min |
| Claude Code | 58.9% | $0.915 | $1.555 | 41.3 min |
| Antigravity CLI | 55.0% | $0.897 | $1.632 | 36.8 min |
What the numbers say
- The model sets most of the cost. In the authors' analysis, the model explains 92.8% of the variance in cost per task and the harness 2.3%.
- With the model fixed, the harness still matters a lot. It changed cost per task by 1.8–5.0× and cost per pass by 1.9–4.9×. The biggest gap was on DeepSeek, where Cline CLI cost 5.0× as much per task as OpenCode v1 for 2.0 more points.
- Paying more doesn't buy a higher pass rate. Per model, the rank correlation between cost per task and pass rate ran from −0.39 to +0.22.
- Steps predict cost; size doesn't. Steps per pass tracked cost per pass (+0.51). Start tokens, tool count and cache hit rate didn't. mini-SWE-agent, the smallest harness, ranked 19th of 22 on cost per task.
- Most money goes to runs that fail. 64.5% of spend went to trials that didn't pass: 52.7% to failures and 11.8% to timeouts. With the five best-value harnesses, 68% (Pi) to 82% (Codex CLI) of failed runs ended with a confident final answer, not at the time limit.
- Pass rates sit close together. Across all models they range from 50.6% (Exo) to 61.9% (ZCode), and most harnesses are within a few points of each other, which the authors treat as ties.
- The five best-value harnesses have things in common: one shell tool doing most of the work, context that only grows, no subagents and almost no summarizing. The authors call these observations, not tests.
Claude Code and Codex in this eval
Both ran here only with open-weight models through Baseten, never with Anthropic's or OpenAI's own models. Read the results with that in mind.
- Codex CLI (0.164.0) had the lowest cost per pass across all models, $0.937, and a top-tier pass rate, 61.1%. It was also the fastest of the high-pass-rate group at 26.8 minutes per task. With GLM-5.3 it was the authors' best-value pick and held in 99.6% of resamples.
- Claude Code (2.1.308) passed 58.9% of tasks, mid-pack, but was slow and costly per pass with these models. It had the second-highest cost per pass ($1.555, after Antigravity CLI), the longest time per task (41.3 minutes) and the second-highest timeout rate (14.2%).
- Its single most expensive pairing was with Muse Spark Open: $3.753 per pass, the highest of all 110 harness and model cells.
- With DeepSeek-V4.1-Flash, Claude Code did well: 68.1%, second on that model after ZCode's 69.0%, at $0.285 per pass.
What it doesn't tell you
- Nothing about closed models. v1.1 tested only open-weight models. It has no results for Claude Opus or Fable, GPT or Gemini models, and its rankings may not carry over to them. The authors list Opus 5.5, GPT-6 Astra and Gemini 4 Carbon as future work. For those models, see GPT-6.2 vs Claude.
- Some harnesses are missing, including Cursor, Gemini CLI and Droid.
- Costs are estimates. They're Baseten list prices times the tokens the provider reported. Baseten's caching and prices drive the cost ranks, which the authors say themselves. Your bill on another provider or plan may differ.
- The maker ran it. Every trial ran on Runta's own runtime, and Runta's founder said so in the Hacker News thread. The data and code are public, so others can check.
- Small gaps are noise. The interval for one harness on one model is about ±10 points, so most pass-rate differences in the tables are ties.
- One kind of task. All 62 tasks are command-line tasks, each run in a fresh, short-lived runtime with default harness settings. Your repo, your prompts and your settings may favor a different harness.
How to read it for your team
- Pick the model first. It drives most of the cost.
- When you change models, test the harness again. The authors' own advice is to think about changing the harness too.
- Use the per-model pick, not the overall average, and check how firm it is. Codex CLI on GLM-5.3 held up; the picks for GLM-5.3-Flash, DeepSeek and Kimi-K3 often didn't.
- Compare cost per pass, not cost per task. A cheap run that fails still costs money.
- Run your own tasks. The authors suggest a private set of your own tasks to test candidates, which tells you more than any public table.
- Watch steps and failed runs. Harnesses that take fewer steps cost less, and checking work before an agent stops may save some of the money spent on failures. The authors didn't test that last idea.
Compare harness and model pairs on your own issues
FrontierHarness Eval measures one agent harness at a time. Crewmark is the harness for a team of agents: each AI employee is its own coding-agent session with a role, a goal, working hours and a heartbeat. So you can run different harness and model pairs side by side on one board and see which one closes your issues.
Add employees on different harnesses
Claude Code and Codex employees run natively. Other agents, such as OpenCode and DeepSeek Harness, connect through ACP (via acpx) on your computer. See Claude Code vs Codex and OpenCode alternatives for the options.
Give them the same kind of work
Use the same role for each, such as the AI software engineer, and assign similar issues from your backlog.
Compare on the board
Each employee wakes on its heartbeat, works its issues and keeps their status current. Check which issues each one finished and what it cost on your own provider or plan.
FAQ
What is FrontierHarness Eval?
A benchmark that runs coding-agent harnesses on the same 62 command-line tasks and compares pass rate, cost and time. Version 1.1 tested 22 harness configurations on five open-weight models, 27,280 runs in all.
Who made FrontierHarness Eval?
Runta, a company that builds a runtime for AI agents. Every trial ran on Runta's own runtime, which its founder disclosed on Hacker News. Baseten served the models, and the leaderboard lists Z.ai as a sponsor.
Which agent harness is best?
The authors don't name one overall winner. They pick a best-value harness per model: Codex CLI for GLM-5.3, Pi for GLM-5.3-Flash, ZCode for DeepSeek-V4.1-Flash, Oh My Pi for Muse Spark Open and Alma for Kimi-K3. Only the Codex CLI pick held firmly when they resampled tasks. Across all models, Codex CLI had the lowest cost per pass and ZCode the highest pass rate.
Does FrontierHarness Eval include Claude or GPT models?
No. v1.1 tested only open-weight models: GLM-5.3, GLM-5.3-Flash, DeepSeek-V4.1-Flash, Muse Spark Open and Kimi-K3. Claude Code and Codex CLI were tested as harnesses, but only with those models. The authors list Opus 5.5, GPT-6 Astra and Gemini 4 Carbon as future work.
Is FrontierHarness Eval open source? Where is the data?
The code and dataset manifest are on GitHub at frontier-harness-eval/eval, branch release/v1.1, and the tasks are public on Harbor Hub. No license is stated for the repo or the data.
What is an agent harness?
The software around a model that makes it a working agent: the loop, the tools, the context and the permissions. Claude Code, Codex CLI and OpenCode are harnesses. FrontierHarness Eval keeps the model fixed and swaps the harness to see how much it changes the result.
How much did FrontierHarness Eval cost to run?
The authors don't state a total. They report 83B tokens over 27,280 runs, and per-task costs estimated from Baseten list prices, from about $0.11 to $1.98 per task depending on the harness and model.
Checked against Runta's v1.1 post, the eval repo (release/v1.1) and the leaderboard on November 18, 2026.