Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't.
Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.
Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.
Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.
That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.
vidarh
One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.
Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.
nsingh2
This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed.
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks for making this and filling a real gap!
markbao
Putting only the tasks and results in the repo is a poor decision. These conclusions would be far more credible if anyone could re-run the benchmark.
kaishin
I'm not sure i agree with these. I've been testing a simple prompt with different model and harness combinations to work out which one produces best results. I do this in '/goal' mode.
Prompt -
"Build a single-page Three.js sci-fi hangar with hovering drones, animated warning lights, emissive runway strips, and subtle volumetric-style fog planes. Include drone formation toggle and cinematic camera path. Output one self-contained HTML file with inline JavaScript."
OMP i would say comes out on this prompt the best out of all the harnesses i've tried. I suspect possibly a lot of this is to do with the base system prompt that each harness injects.
benw214
Is Github Copilot (integrated in VS Code) not a thing? I use it all the time and don't know what more I could wish for.
Disclaimer: I don't let agents run on huge tasks for hours. Almost all tasks I give them are done in under 30 min.
knombertus
I was excited until I saw cost was only provided as the median. Your provider will bill you for all your tasks and one can get back to that total from the mean by multiplying by the number of tasks. This isn't possible with the median and I suspect the median is likely below the mean so this understates the actual costs.
joshheitzman
There really should be error bars on those measurements. With just 30 samples, the 95% confidence intervals for accuracy should all be more than 35 percentage points wide, so comfortably overlap. For cost it's harder to say, because outcomes aren't constrained to {0, 1}, but I also expect a lot of variability there.
yorwba
Good start, but needs to be harness x model to be useful. 3 top harnesses x 3 top models would be more interesting.
xnx
With all the hype around the latest Gemini 3.8 Flash/Cyber release, will Antigravity CLI [1] be supported?
comments (10)
Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.
Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.
Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.
That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.
vidarh
Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.
nsingh2
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks for making this and filling a real gap!
markbao
kaishin
Prompt -
"Build a single-page Three.js sci-fi hangar with hovering drones, animated warning lights, emissive runway strips, and subtle volumetric-style fog planes. Include drone formation toggle and cinematic camera path. Output one self-contained HTML file with inline JavaScript."
You can see the outputs here - https://alvins82.github.io/hangar-harness-model-tests/ https://github.com/alvins82/hangar-harness-model-tests/
OMP i would say comes out on this prompt the best out of all the harnesses i've tried. I suspect possibly a lot of this is to do with the base system prompt that each harness injects.
benw214
Disclaimer: I don't let agents run on huge tasks for hours. Almost all tasks I give them are done in under 30 min.
knombertus
joshheitzman
yorwba
xnx
1/ https://antigravity.google/product/antigravity-cli
kakugawa