Klavis AI has published Multi-step SWE, a benchmark for coding agents that repeatedly work on their own implementations. The preprint was submitted on November 4, 2026. Its 69 tasks contain chains of 5–15 dependent engineering requests. The strongest evaluated combination solves 18.5% of complete rollouts; none of the nine tested models solves 38 tasks. These are author-reported results, not an independent SIExpo measurement.
Why the next request matters
A typical demonstration asks whether an agent can build one feature. Real projects continue with changed requirements, migrations, or optimization. If an initial design is awkward, subsequent work begins from that design. Multi-step SWE makes this dependency part of evaluation instead of providing a clean starting point for every request.
The repository contains 574 steps across nine languages and uses Harbor's task format. A step is a complete engineering request, not a tool call. Checks run in separate verifier containers. Running the public benchmark requires a Docker environment, computing resources, and access to the selected model; open evaluation artifacts do not make compute or external services free.
Klavis's accompanying post distinguishes individual-step performance from whole-task success. For Claude Opus 5.5 with Claude Code, it reports 56.8% of steps passed and 18.5% of complete rollouts solved. Later steps continue after failures, but an earlier failure still prevents full task success. Pass@4 is a different measure: at least one success across four attempts.
Reading the comparison
Results evaluate a model together with its harness and configuration. They are not an isolated ranking of model weights. The authors acknowledge limitations, including C/C++-heavy coverage, possible familiarity with public code, and the absence of an isolated-step baseline separating accumulated errors from later steps' inherent difficulty. A longer budget alone is not an established solution.
That supports a more measured conclusion than saying agents cannot program. A system may help with many individual changes yet fail to complete a sequence under every criterion. The metrics answer different questions. When selecting a tool for an extended project, whole-chain completion matters alongside the quality of individual edits.
An editorial example for an internal project
Imagine a small application that first stores classified listings, then adds search, and later introduces roles and moderation. The initial version might look complete, while later requirements reveal whether data and authorization have clear boundaries. This is our example of dependent requests, not another task reported by the benchmark authors.
For such an internal experiment, expectations can be written before work starts. After each step, evaluation checks both the new requirement and previously promised behavior. Search working correctly is insufficient if an edit lets someone change another user's listing. The verification procedure should follow the original conditions, rather than only the agent's own plan.
Preserving the work history is important. Replacing problematic code with a reference solution between steps would measure another capability. Repeated attempts also need accounting: the best attempt's outcome is not the same as the probability that an ordinary session finishes successfully.
The reviewer should also record whether the agent identifies unfinished work correctly. A final message saying everything is complete is evidence of the agent's report, not evidence that every requirement passed. Comparing the report with independent checks makes an operationally useful distinction between successful work, an acknowledged partial result, and an incorrectly claimed completion.
Public tasks provide a starting point for this kind of research while leaving licensing, resource requirements, and safe execution to the evaluator. SIExpo checked the available materials and methodology, without installing the benchmark or running model trials. The significance of the release lies in its more realistic question: can an agent keep building on the decisions it has already made?
