On November 17, Vals AI released an evaluation of safeguards in general-purpose models under simulated scenarios involving risks to civilians. Researchers found that some models produced unacceptable assessments about civilians or civilian objects. The finding concerns the tested systems under specific conditions and matters for the protection of people when AI is deployed.

What the evaluation measured

Vals tested whether safeguards responded consistently when a task was presented in different ways. That consistency matters because requests with the same underlying meaning can arrive in many different forms. The results show gaps in the tested models' behavior. They remain dependent on the version, environment and evaluation setup.

The authors draw explicit boundaries around the result. They did not validate targeting accuracy, compliance with the law of armed conflict or performance in real military workflows. The research does not establish how any actual decision was made in a specific incident. It cannot serve as an assessment of a particular system's fitness for the use of force.

What this means for evaluating AI

A useful safety criterion is whether an underlying boundary holds across varied forms of interaction. A single demonstrated refusal gives limited evidence about this. Evaluations should be repeated as versions change, with the context and the metric made explicit.

A separate question is what a surrounding application can do with a model output. In high-consequence settings, constraints also need to operate outside the model through clear permissions, independent review and accountable human oversight. A poor assessment of an image should never acquire the status of a verified fact just because it arrives in a structured format. The report is a reason to scrutinize safeguards more closely and make the risks to civilians explicit in decisions about deployment.

The International Committee of the Red Cross warns that loss of human control makes it harder to anticipate and limit the effects of autonomous weapons. That is a separate institutional position. The Vals evaluation contributes a specific question about the reliability of safeguards in models offered for other purposes.

Editorial verification — 2026-11-18T10:21:35.465083+00:00