# Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale

DevFeed: [Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale](<https://devfeed.tech/articles/screen-before-you-serve-simulation-for-production-customer-experience-ai-agents-at-140m-scale-120086.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.30137>)

Author: Edesio Alcoba\\c{c}a, Kevin Rossell, Aman Gupta, Shao Tang, Jiwoo Hong, Pabel Carrillo-Mendoza, Wanderson Concei\\c{c}\\~ao Ferreira, Alvaro Tedeschi, Zayd Simjee, Shreya Rajpal, Bruno Finardi Hime, Chri

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI for customer service automation](<https://devfeed.tech/topics/ai-for-customer-service-automation.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-workflows](<https://devfeed.tech/tags/agentic-workflows.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [also](<https://devfeed.tech/tags/also.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [backends](<https://devfeed.tech/tags/backends.md>), [brazil](<https://devfeed.tech/tags/brazil.md>), [can](<https://devfeed.tech/tags/can.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [customer](<https://devfeed.tech/tags/customer.md>), [customer-experience](<https://devfeed.tech/tags/customer-experience.md>), [experience](<https://devfeed.tech/tags/experience.md>), [production](<https://devfeed.tech/tags/production.md>), [scale](<https://devfeed.tech/tags/scale.md>), [screen](<https://devfeed.tech/tags/screen.md>), [serve](<https://devfeed.tech/tags/serve.md>), [simulation](<https://devfeed.tech/tags/simulation.md>)

## AI overview

The Snowglobe workflow screens customer experience agents before deployment by having synthetic customers react to agent responses and simulating tool outputs, enabling multi-step tests without production backends. On Nubank's Card Delivery and expanded Card Management agents, simulated and production binary evaluator scores were highly correlated across four deployed versions. Simulation-guided iteration raised transactional net promoter score by 36.69 points in a live A/B test. Screening open-weight configurations across more than 16,000 simulated conversations preceded a live test in which the selected model raised self-service rate by 8.82 percentage points to Nubank's highest observed level, with no statistically significant tNPS change. The abstract reports results for this Nubank deployment; it does not establish that the workflow generalizes to other agents.

## Source excerpt

arXiv:2609.30137v2 Announce Type: replace Abstract: Customer experience (CX) agents use tools and large language models to address customer requests and guide conversational interactions with an organization's products. Improving these agents, especially in regulated industries, is difficult: they must detect intent, follow complex operational policies and use tools reliably. Manual end-to-end testing offers limited coverage, while live experiments expose customers to failures that can erode trust. We present a hypothesis-driven simulation workflow for screening candidate CX agents before deployment. Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends. We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Management - Nubank's highest-volume chat-support agent in Brazil. Across 4 deployed versions, simulated and production version-level binary evaluator scores show high correlation. Simulation-guided iteration increased transactional net promoter score (tNPS) by 36.69 points in a live A/B test. We also screened open-weight configurations in over 16,000 simulated conversations. In a subsequent live A/B test, the selected model increased self-service rate (SSR) by 8.82 percentage points to the highest level observed at Nubank, with no statistically significant change in tNPS. Simulation made broad exploration of models, reasoning settings, and prompts feasible without customer exposure, enabling production improvements that would have been impractical to pursue through live experimentation alone.