IBM Research's CUGA team published a comparison of three agent harnesses using GLM-5.3-flash on November 3. In its sample of 90 AppWorld tasks, CUGA completed 73, Hermes 42 and Deep Agents 27. Configurations differed, however: CUGA used an AppWorld-specific setup. This is an experiment by the project's own team, not an independent ranking covering every use case.
The authors explicitly state that these runs did not use policies. The results therefore do not measure business-rule enforcement. We keep those subjects separate instead of treating a high completion rate as proof of safe deployment.
What the environment around a model does
Our editorial interpretation starts with a simple example: an assistant must find an order, read its status and propose a next step. The model chooses what to request, but the harness determines available tools, how results are returned and how the process continues. Changing that organization can change outcomes even when the underlying model stays the same.
The linked public harbor-eval repository is a fork of Harbor, a framework for evaluating agents and environments. It makes the technological foundation inspectable. Code availability does not mean SIExpo has reproduced this comparison. A replication would need recorded versions, settings, budgets and completion criteria before percentages could be compared usefully.
Instructions and pre-execution checks
CUGA's policy documentation distinguishes guidance such as Playbooks and Tool Guides from pre-execution checks and human approval. Policies can be triggered by input, context or tool use. This is the project's architectural description, rather than independent certification that every configuration enforces its rules correctly.
The ToolGuard guide explains that generated checking code can block a call before the original tool executes. A tool without an applicable rule is not automatically blocked by this mechanism. Teams therefore need to inspect both the rule's contents and whether the intended tool actually passes through it.
In an illustrative workflow, an assistant may prepare a customer-record change while committing it requires a check on permitted fields. A persuasive explanation of the change does not establish that the execution path is technically constrained. A pilot should exercise an allowed and a forbidden case, observing whether the second reaches the receiving service at all. We have not performed that test here.
The same reasoning applies when an application exposes several routes to a similar operation. Testing one visible button does not establish what happens through another integration. Our proposed review would identify the actual write paths and the evidence needed to show each is governed. This is an integration recommendation, not an additional CUGA benchmark result.
Approval has its own lifecycle
The separate Tool Approval guide describes pausing for confirmation after code generation and resuming through a response to the agent. Showing code is configurable, while timed automatic approval is a separate parameter disabled by default. An approval button consequently needs to be assessed together with its actual configuration.
Our recommendation is to keep approval connected to the precise proposed action. If arguments change, the reviewer should understand which operation is being accepted. Resumption after interruption matters too: continuing a conversation should not turn an already completed operation into a second submission. That must be checked in the surrounding workflow, rather than inferred from a successful response.
What the comparison supports
The study is a useful reason to treat harness selection as a separate engineering decision. In a real pilot, we would measure completed tasks, failures and requests for human intervention against the intended workflow. Evaluating those outcomes and evaluating constraints on actions should remain separate exercises. The reported figures motivate scrutiny of configuration; they do not provide a general guarantee that one product is best for every organization.
