# From Voiceprints to User Entities: Online Speaker-Memory Construction for Multi-User Spoken Agents

DevFeed: [From Voiceprints to User Entities: Online Speaker-Memory Construction for Multi-User Spoken Agents](<https://devfeed.tech/articles/from-voiceprints-to-user-entities-online-speaker-memory-construction-for-multi-user-spoken-agents-103180.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36713>)

Author: Yucong Zhang, Cancan Li, Bo Xiao, Ke Zeng, Ming Li

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [voice ai](<https://devfeed.tech/topics/voice-ai.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-sd](<https://devfeed.tech/tags/cs-sd.md>), [eess-as](<https://devfeed.tech/tags/eess-as.md>), [identity](<https://devfeed.tech/tags/identity.md>), [memory](<https://devfeed.tech/tags/memory.md>), [online](<https://devfeed.tech/tags/online.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [speaker](<https://devfeed.tech/tags/speaker.md>), [user](<https://devfeed.tech/tags/user.md>)

## AI overview

EvoSpeakerMem treats voiceprints as evolving evidence about persistent user entities, routing speech causally and selectively updating acoustic state before partitioning memory writes and retrieval by identity through a replaceable text-memory backend. The training-free layer is evaluated with LoCoMo-ID, a benchmark featuring hidden speaker identities, speaker-sensitive queries, simulated recording shifts, and four-speaker confusion tests. Gated updates improve identity-gated QA over fixed representations by 3.86 percentage points in clean conditions and 11.62 under shifts, with no wrong-user authorizations; the advantage persists in the four-speaker test. Separately, longitudinal SMIIP-TV results show genuine-probe rejection rising from 29.8% to 55.6% over about a year at a frozen near-enrollment threshold. The authors caution that less fragmentation and better QA can still accompany incorrect cross-user links, so QA alone misses the fragmentation-contamination trade-off.

## Source excerpt

arXiv:2610.36713v1 Announce Type: cross Abstract: Personalized spoken agents must identify speakers before accessing user-specific memory, yet memory benchmarks typically assume correctly attributed histories. We argue that voiceprints are observations of identity, not identities themselves, and formulate online speaker-memory construction to maintain persistent user entities as acoustic evidence evolves. We introduce EvoSpeakerMem, a training-free layer for causal speaker routing, selective acoustic-state updates, and identity-partitioned memory writes and retrieval through a replaceable text-memory backend. We also construct LoCoMo-ID, a spoken-memory benchmark with hidden speaker identities, speaker-sensitive queries, simulated recording-condition shifts, and four-speaker identity-confusion tests. On longitudinal SMIIP-TV, genuine-probe rejection under a frozen near-enrollment threshold rises from 29.8% to 55.6% over approximately one year. On LoCoMo-ID speaker-sensitive queries, gated updating improves identity-gated QA over fixed representations by 3.86 and 11.62 percentage points under clean and shifted conditions, respectively, with no wrong-user authorizations. The advantage persists in the four-speaker test. Yet reducing fragmentation and improving QA can coexist with incorrect cross-user links, revealing a fragmentation--contamination trade-off that QA alone cannot capture.