METR published a new assessment on November 20 of capabilities relevant to AI research. It reports strong performance in coding and optimization, alongside weaker judgment on tasks without clear empirical feedback. That distinction matters when deciding how to evaluate the usefulness of agents rather than describing all capabilities with a single headline.

The new TH2 suite contains 36 tasks in 12 families. Among 34 score-and-time baselines, three come from successful human attempts and 31 are estimates. The comparison should therefore not be presented as though every human reference were directly measured under the same experimental procedure.

Reading the time horizon

Earlier METR methodology uses human reference time for tasks a system completes with a specified probability of success. This is not the duration of continuous agent operation. The metric alone does not establish that an entire profession or working day can automatically be transferred to a system with the corresponding time label.

Our interpretation is to connect an assessment to a concrete kind of work. A well-specified change with regression tests and an architectural decision under incomplete requirements have different verification methods. These are original examples, rather than tasks quoted from TH2. Persuasive final text is not equally strong evidence of success in both cases.

Define acceptance before the pilot

For internal assessment, we suggest grouping tasks by feedback: automatic testing, delayed measurement or expert judgment. Each group can have an identified person or procedure that confirms success. This prevents easily checked tasks from hiding uncertainty elsewhere behind a single average result that looks more general than the underlying evidence.

The human reference should also retain a category. Measurement, estimation and assumption can be reported separately, with realistic precision. A reader can then see which parts of the comparison derive from observation and which require an unavoidable but uncertain judgment. That distinction is useful even when the numerical result remains attractive.

Research rather than a deployment promise

SIExpo has not reproduced the suite. A particular organization could next run a bounded pilot with a predefined scope and retained reasons for each assessment. For decisions without a simple test, assumptions and consequences of error are especially important to document. Successful coding output does not remove that obligation.

The report adds useful evidence partly because it exposes differences between strengths and limitations. Better assessment does not require giving every task the same level of trust. It requires knowing what was verified, which comparisons were estimated, and where human judgment remains necessary for the decision being made.