The November 17 Resolution Hypothesis preprint examines a limitation in agent evolution: evaluation may fail to distinguish a helpful modification from a lucky run. The authors study seven methods and six models with more than ten million language-model calls. Their central distinction is between proposing improvements and reliably identifying them. The work is neither a verdict against every form of self-improvement nor evidence that other researchers acted in bad faith.

Two different parts of one loop

A system might propose a new instruction, skill or workflow structure. Another mechanism then decides whether the proposal should survive. If the selection mechanism is unreliable, good proposals do not guarantee a better next version. This is the analytical problem underlying the hypothesis.

The project page presents repeated evaluations and examples of evaluation shortcuts. The authors distinguish transferable benefit from adaptation to a grader's conventions. They argue for reporting variation more clearly and studying proposal and selection separately. These conclusions arise from their experimental program, not a universal measurement of every commercial agent.

How luck can look like progress

Consider an editorial example. Two versions attempt the same difficult task. One happens to find the relevant file earlier, while the other spends time following an unhelpful path. If only that run is retained, the difference might be attributed to an instruction change even when the change has no stable benefit.

Accepting whichever version looks better after each such attempt can accumulate luck rather than dependable skill. The opposite extreme also has costs: a highly conservative selector might reject every small change. The useful question is how strong an observed signal needs to be before it justifies selection.

The distinction does not imply that difficult measurements are worthless. It means that their resolving power belongs in the interpretation. A tool can be useful for detecting a large change and still be poorly suited to ranking several very similar candidates. Treating both decisions as equally certain obscures the engineering problem.

A result includes uncertainty

Reporting a result requires more than a best percentage. Repetition, the spread of outcomes and test conditions all matter. A difference of a few points means something different when one unchanged version naturally varies by more than that amount. This statistical reading makes the research relevant to practical development.

A team could separate two questions: do useful proposals exist in the candidate pool, and does its selector choose them? Failed selection would then not be confused with a lack of good ideas. The distinction also directs improvement work. Better checking might be more valuable than a more complicated candidate generator.

An everyday engineering example

Imagine an agent preparing a spreadsheet report. If the grader checks only that a file and expected heading exist, a change that creates both more reliably might score higher while leaving incorrect numbers inside. Evaluating the actual objective requires examining content and data provenance. This is our illustration of signal versus goal, not an experiment attributed to the paper.

The same logic applies to time and cost. Faster completion is valuable when the same useful work is still performed. If an optimization quietly changes what counts as completion, the comparison becomes a different task. The evaluation mechanism is therefore part of the engineered system rather than an administrative attachment.

A useful development record should retain the starting version, accepted changes, rejected candidates and repeated outcomes. Such records make it possible to ask whether apparent progress survives fresh work instead of merely improving the history from which selection was made. These are editorial assessment suggestions, not claims that the researchers certify a particular deployment procedure.

The Resolution Hypothesis directs attention toward better measurement rather than abandoning agents. For readers, the practical question is whether a claimed gain remains visible on repetition and on separate work. Evidence of progress is a stable change in useful behaviour, not only the highest point in one evolution log.