# Omni-JEV: Text and Dual-Stream Speech Decisions

DevFeed: [Omni-JEV: Text and Dual-Stream Speech Decisions](<https://devfeed.tech/articles/omni-jev-text-and-dual-stream-speech-decisions-128757.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13777>)

Author: Shengzhou Kong

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Conversational AI](<https://devfeed.tech/topics/conversational-ai.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [audio](<https://devfeed.tech/tags/audio.md>), [choice](<https://devfeed.tech/tags/choice.md>), [classification](<https://devfeed.tech/tags/classification.md>), [completion](<https://devfeed.tech/tags/completion.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [history](<https://devfeed.tech/tags/history.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [interface](<https://devfeed.tech/tags/interface.md>), [interrupt](<https://devfeed.tech/tags/interrupt.md>), [jev](<https://devfeed.tech/tags/jev.md>), [learning](<https://devfeed.tech/tags/learning.md>), [maps](<https://devfeed.tech/tags/maps.md>), [policy](<https://devfeed.tech/tags/policy.md>), [probability](<https://devfeed.tech/tags/probability.md>), [speech](<https://devfeed.tech/tags/speech.md>), [stream](<https://devfeed.tech/tags/stream.md>), [text](<https://devfeed.tech/tags/text.md>)

## AI overview

Omni-JEV predicts probabilities over five dialogue actions at collected-segment endpoints, using text or role-separated user and assistant-reference speech. Its aligned speech-to-speech backbone shares a candidate readout with text inputs. The model is trained with bilingual weak supervision and cross-entropy fine-tuning, then reinforcement learning for probability-sensitive decisions (RLCD). Comparing Gaussian-logit with categorical Choice RLCD from the same supervised initialization and matched training exposure, the authors report different trade-offs between classification and probability quality. Gaussian Omni-JEV reaches test accuracy of 84.08% for audio and 96.55% for text. The shared action interface is intended to guide whether a full-duplex assistant responds, continues listening, interrupts, or resumes an interrupted response.

## Source excerpt

arXiv:2611.13777v1 Announce Type: new Abstract: A user utterance does not uniquely determine the next action in full-duplex dialogue: its consequences depend on the assistant's response state and interaction history. Speech activity, utterance completion, and interruption intent provide related evidence, whose implications for a response require an explicit action policy. We formulate this task at collected-segment endpoints and introduce Omni-JEV, which maps text or role-indexed user and assistant-reference speech to probabilities over five interaction actions. An aligned speech-to-speech understanding backbone encodes the acoustic roles separately and shares its candidate readout with text inputs. Bilingual weak supervision supports cross-entropy fine-tuning followed by reinforcement learning for probability-sensitive decisions (RLCD). We compare Gaussian-logit and categorical Choice RLCD from the same supervised initialization under matched training exposure, observing different trade-offs between classification and probability quality. Gaussian Omni-JEV achieves test accuracies of 84.08% for audio and 96.55% for text. This common action interface supports decisions about when a full-duplex voice assistant should respond, preserve ongoing speech, interrupt, or resume an interrupted response.