# How a Bank Serving 100 Million Users Tests Agentic AI in Production

DevFeed: [How a Bank Serving 100 Million Users Tests Agentic AI in Production](<https://devfeed.tech/articles/how-a-bank-serving-100-million-users-tests-agentic-ai-in-production-109110.md>)

Original publisher: [Read original article](<https://hackernoon.com/how-a-bank-serving-100-million-users-tests-agentic-ai-in-production?source=rss>)

Author: sarahevans

Published: 2026-10-30T13:05:58Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [AI for customer service automation](<https://devfeed.tech/topics/ai-for-customer-service-automation.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [blue green and canary deployments](<https://devfeed.tech/topics/blue-green-and-canary-deployments.md>), [experiments](<https://devfeed.tech/topics/experiments.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-research](<https://devfeed.tech/tags/ai-research.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [bank](<https://devfeed.tech/tags/bank.md>), [business](<https://devfeed.tech/tags/business.md>), [customer-experience](<https://devfeed.tech/tags/customer-experience.md>), [customer-support](<https://devfeed.tech/tags/customer-support.md>), [fintech](<https://devfeed.tech/tags/fintech.md>), [llm-evaluation](<https://devfeed.tech/tags/llm-evaluation.md>), [production](<https://devfeed.tech/tags/production.md>), [tests](<https://devfeed.tech/tags/tests.md>)

## AI overview

The bank's five customer support agents improved transactional NPS in online A/B tests, but all remained below separately measured human benchmarks. To assess new versions, the team first validates automated graders against expert labels: one grader's initial 68.88% accuracy trailed a 77.78% majority-label baseline, then rose to 88.89% after prompt optimization. That score measures agreement with expert labels, not agent accuracy. New versions begin at 1% to 5% customer exposure, with the team monitoring self-service, satisfaction, escalations, and errors. Agents hand off cases when requests exceed their routines, required data is unavailable, or confidence is low; adding a card-reissue capability removed a transfer from the most common card-delivery resolution path. The excerpt stresses that satisfaction scores alone can obscure workload shifts: harder cases left for human agents may take longer, so teams need to examine why cases escalate. The graders' cross-model tests covered models from one vendor, with broader coverage left for future work.

## Source excerpt

A KDD 2026 paper shows how a bank tests agentic AI in production: graders checked against experts, 1% to 5% rollouts, and the human handoffs behind the scores.