IBM Research published customer-service results for ALTK-Evolve on October 20, 2026. With GPT-4.1, learned instructions raise retail Pass^4 from 57.5% to 67.5%: the share of tasks successful on all four attempts. Weights are unchanged. The leaderboard comparison uses an October 12 snapshot; these are author-reported findings that SIExpo has not independently reproduced.
What gets added to the agent
Evolve’s implementation documentation describes trajectory-derived memory, MCP integration and conflict resolution. It warns that the local dashboard API has no authentication, allowing other local processes to read or change memories; operation without the dashboard is available. The code uses Apache-2.0. These implementation properties are separate from the measured customer-service result.
The essential distinction for readers is between retained text and a demonstrably useful rule. Our explanation is that an assistant extracting the wrong procedure from an incorrectly completed case would not improve merely by repeating it. We would therefore track each instruction’s origin, the conditions under which it applies and who can revoke it. This is an editorial evaluation recommendation, rather than an additional IBM experiment or evidence that every stored instruction is reliable.
Which benchmark version is being compared
Sierra’s current repository already describes τ-bench 2.0 and warns that its English tasks and rules differ from 1.x. Results are not directly comparable; the old v1.0.1 remains available for the corresponding comparisons. The benchmark’s current documentation therefore does not turn IBM’s October measurement into a result on the new task set.
This matters to a company choosing an agent. A percentage without task versions and execution conditions provides a weak basis for a decision. We would request identical queries, the same tool permissions and a record of the final changes. That evaluates completed work instead of the confident tone of a closing response. For a real support pilot, we would also include cases correctly concluded by escalation to a person, rather than treating autonomous completion as the only desirable outcome.
What repeatability means to the customer
Our example is a product-return request. Correctly written prose and a correctly updated order are different outcomes. An assessment should establish whether the authorized action occurred and whether the customer received an accurate account of it. This example and the proposed criteria are the newsroom’s; we are not claiming to have deployed the system or inspected actual customer accounts.
A better benchmark result gives teams a reason to examine the approach carefully. It does not remove the need to review memory, limit access and retain a route back to an earlier configuration. For our own proposed pilot, we would record cost per completed task and keep failed attempts alongside successful ones. The aim would be to understand which improvement persists under the organization’s actual conditions. That would provide a more useful basis for selection than a general promise that an agent learns by itself.
Editorial verification: 2026-10-21T06:28:23.702396+00:00 (UTC). Cover: conceptual AI illustration, not a documentary photograph.
