FINAL-Bench introduced ViKernel Scanner on November 18 alongside its own evaluation of 20 open-weight models. The authors say all received an F under their methodology. Testing uses mock tools and records attempted calls. These are the scanner creators' results, not an independently reproduced ranking or evidence of 20 actual breaches.

What is actually evaluated

The public repository describes simulated file, communication and shell actions. It specifies that real actions are not executed. It also says the checks are supplied by a central server. Open client code and a complete fixed test set are therefore different components of reproducibility.

That distinction matters when interpreting a finding. An attempted dangerous call shows behavior in a particular scenario; it does not establish whether a real application would execute the request. Permissions, validators, isolation and human confirmation can stand between model output and a side effect. Neither the presence nor absence of those controls should be assumed without inspection.

Under the described scheme, the overall result follows the weakest evaluated domain. An F can therefore arise from a single severe case; it does not mean a model complied with every dangerous request. This property should accompany the headline count of tested models.

Comparing two evaluations

Our editorial criteria would require the exact model version, provider, configuration, check-set identifier and raw record of attempted calls. If the central set changes, an unchanged scanner name does not guarantee identical conditions across dates. Preserving context is part of the measurement.

Ordinary permitted tasks matter too. A system that blocks everything can avoid dangerous actions while being unusable. We would therefore compare protective behavior alongside successful completion of allowed work. This is a proposed evaluation framework, not our own result from running ViKernel.

An application-level review should also separate the model's decision from the executor's decision. Recording both makes it possible to see whether a restriction prevented an attempted action or whether the model never proposed that action. Those outcomes can require different follow-up work.

What a letter grade cannot establish

A letter alone does not show frequency, severity, conditions or whether a side effect was genuinely reachable. Two models may receive the same overall grade after different numbers of problematic situations. Deployment decisions need concrete traces and application-layer checks.

SIExpo read the announcement and code description without running adversarial checks or sending real data to the service. The development is the tool and disclosed methodology. Result claims remain explicitly attributed to the authors, including claims about protection offered by the same team.