A few days ago I published a benchmark that ran five frontier models through 410 coding-agent tasks. The short version: with a test suite to run against, all five fixed every bug, and what separated them was cost, speed, and one model’s refusals, not raw coding ability. The one place capability actually split the field was a single bug diagnosed blind, with no tests to check the answer against. Every Claude model mostly failed it and both GPT-5.6 models solved it every time. You can read the full writeup here.
The obvious follow-up question came up more than once: what about the open-weights models? So I added one. Kimi K3, Moonshot’s new open-weights flagship, ran through the same harness and traced into Latitude like everything else. I gave it the parts that actually separated the field, the hard tier with tests and the blind test, rather than the full 82-run gauntlet. Here is what happened.
The bug that broke every Claude model
The blind test reuses one task: an event bus whose once helper fires its handler twice during a re-entrant emit. The model gets the bug report and the source, no test files and no way to run the code, and a hidden suite scores the fix afterward. In the original run every Claude model mostly failed it, and both GPT-5.6 models went a perfect 13 of 13.
Kimi K3 went 12 of 13. An open-weights model with no lineage to either lab landed with the GPT-5.6 family and well clear of the entire Claude field.
Everywhere else it kept pace too. It solved all 18 of its hard-tier runs with a test suite to check against, made clean tool calls with no malformed requests, never edited a test to force a pass, and unlike Claude Fable 5 it never once refused the work.
The price is speed, not money
The catch shows up on the clock.
Per solved task Kimi costs about what Sonnet does, roughly a nickel, and a fraction of what Opus costs. But it is slow. Its median hard-tier run took 98 seconds against 10 to 20 seconds for the frontier field. On this workload Kimi K3 does not cost more in dollars. It costs in wall-clock time.
What this says about the blind result
The blind number is the one worth sitting with, because it complicates the explanation I gave the first time around. I argued that the GPT-5.6 sweep came from a shared family instinct: make the smallest change that fixes the reported bug, and treat the existing design as a contract to respect rather than something to rewrite. Kimi K3 was trained by a different lab on the other side of the world, shares no lineage with either OpenAI or Anthropic, and has the same instinct. So the discipline that cracks this bug is not proprietary to one company. Whoever did the training, the models that win here make the small fix, and the models that lose diagnose the bug correctly and then rewrite the architecture around it.
Kimi’s single miss makes the point better than its twelve solves. In that one run it identified the re-entrancy perfectly, then rewrote the event bus’s dispatch loop instead of adding the one-line guard the passing models used. That is the exact move that sank all three Claude models: correct diagnosis, more ambitious fix, failed hidden test. The thing separating the models here is not who understands the bug. It is who trusts the existing design enough to leave most of it alone.
The takeaway holds, and widens
The conclusions from the original benchmark still stand, and Kimi stretches one of them. Capability is now table stakes, even for open weights. If your pipeline hands agents a failing test to iterate against, an open-weights model produces the same green checkmark as a frontier flagship for about the same money, though on this harness you will wait longer for it. The gap that remains is judgment under uncertainty, the case where nothing can tell the model it is wrong, and that gap does not line up with price or with whether the weights are open.
The harness, the tasks, and every per-run result are in the repo, including the Kimi runs, and the original writeup is here.