Anthropic published guidance on building AI safety evaluations on November 13. Its central message concerns measurement: a higher score does not establish safer behavior if the test captures the wrong property. The company recommends defining criteria in advance, validating graders and reporting uncertainty. This is a new methodological publication, rather than independent certification of Claude.
What question does the test answer?
A test score compresses an answer to a particular question. Whether an assistant preserves specified constraints over a conversation is different from whether it refuses every request containing a particular word. In our editorial example, a business assistant could reject a harmless account-protection question while missing a risk phrased differently. An aggregate percentage could conceal that distinction.
The concern predates the new guidance and extends beyond model vendors. The 2024 BetterBench paper proposes a framework for assessing benchmarks themselves and reports substantial differences in quality. Statistical reporting and reproducibility are among the problems it describes. It does not measure the safety of a current product version; it explains why the measuring instrument deserves its own review.
Another paper, the 2025 Measuring what Matters study, examines the relationship between an asserted abstract property, the chosen tasks and the scoring metrics. Its systematic review of LLM benchmarks provides useful background: a measure needs to represent the phenomenon about which conclusions will be drawn. Both studies predate this news and are cited as context, rather than being presented as newly released research.
From measurement to a deployment decision
NIST's AI Risk Management Framework separates connected activities for governance, understanding context, measurement and risk management. It treats risk across a system's lifecycle and involves different kinds of expertise. That perspective helps explain why publishing a performance table does not complete deployment assessment.
Our practical interpretation is to begin by describing the actual service and the consequences of a mistake. Test cases can then be selected around that use, alongside a separate set of ordinary, permitted requests. Human review of some grades helps establish whether an automated grader interprets the criteria correctly. This is a suggested application of the principles, rather than an experiment conducted by SIExpo.
When a model, instruction set or connected tool changes, the comparison needs to identify precisely what changed. Replacing several components simultaneously makes it difficult to attribute an observed result to just one of them. Retained configurations, test cases and unsuccessful responses provide more useful evidence for later analysis than a standalone headline percentage.
The next question is what decision the evidence supports. It might justify a bounded pilot, further tests or avoiding a particular use. An operational system's safety also depends on access permissions, monitoring and a way to stop execution. A benchmark cannot take responsibility for those surrounding controls.
For a buyer comparing assistants, this means asking for the evaluated configuration and the intended setting, rather than treating every published score as interchangeable. A result obtained with different tools or a different system prompt may answer another question. The new guidance is a useful starting point for that conversation, while independent assessment and local validation remain necessary parts of a credible deployment decision.
The cover is an AI-generated editorial illustration, not a documentary photograph.



