# ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

DevFeed: [ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model](<https://devfeed.tech/articles/altk-evolve-how-learned-memory-tops-the-2-bench-leaderboard-on-customer-service-tasks-without-a-frontier-model-91098.md>)

Original publisher: [Read original article](<https://huggingface.co/blog/ibm-research/altk-evolve-bc>)

Author: Ashwath Vaithinathan Aravindan; Vatche Isahagian; Vinod Muthusamy; Jayaram Radhakrishnan; Evelyn Duesterwald; Gaodan Fang; Punleuk Oum; G Thomas; Michał Ulewicz; Merve Unuvar; Ayhan Sebin

Published: 2026-10-20T15:19:48Z

Content type: article

Language: en

Sources: [Hugging Face](<https://devfeed.tech/sources/hugging-face-blog.md>)

Topics: [AI for customer service automation](<https://devfeed.tech/topics/ai-for-customer-service-automation.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Frontier Model](<https://devfeed.tech/topics/frontier-model.md>), [Claude](<https://devfeed.tech/topics/claude.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [airline](<https://devfeed.tech/tags/airline.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [customer-service](<https://devfeed.tech/tags/customer-service.md>), [frontier-model](<https://devfeed.tech/tags/frontier-model.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [leaderboard](<https://devfeed.tech/tags/leaderboard.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model-weights](<https://devfeed.tech/tags/model-weights.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [retail](<https://devfeed.tech/tags/retail.md>)

## AI overview

ALTK-Evolve improves τ²-bench customer-service reliability by automatically distilling rules from an agent's training-task successes and failures into prompt guidelines, without changing model weights or using human annotations. On held-out tasks with GPT-4.1, guidelines raised Pass^4 by 10 percentage points in both retail and airline; retail benefited most from the full guideline set, while airline's smaller 14-rule sup3 core was more reliable than its full set. With GPT-5.5 on airline, guidelines raised accuracy by 10 points and Pass^4 by 15. The reported leaderboard results put GPT-4.1 with guidelines first in retail (67.5% Pass^4) and GPT-5.5 first in airline (82.0%); GPT-4.1 placed eighth in airline. Guidelines increased prompt input cost, but matched solved conversations were slightly shorter. Results use τ²-bench's ReAct scaffold and four trials per task; leaderboard figures cover the full base split, unlike the held-out comparisons.

## Source excerpt

# ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model *ALTK-Evolve learns and applies a playbook using GPT-4.1, outperforming the retail entries on τ²-bench, a customer service agent benchmark--without updating model weights.* Using GPT-4.1, an older, non-reasoning model, a basic customer-service agent reaches **67.5% Pass^4 on τ²-bench retail** -- meaning it solves the task on *all four* independent attempts for 67.5% of tasks.