# Real-time AI interactions

Real-time AI interactions are low-latency interactions in which AI responses are delivered immediately or incrementally, including conversational, streaming, and multimodal interactions.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## AURA: AUdio-dRiven streaming Avatar

DevFeed: [AURA: AUdio-dRiven streaming Avatar](<https://devfeed.tech/articles/aura-audio-driven-streaming-avatar-145682.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26540>)

Author: Nathaniel Cohen, Nicolas Dufour, Am\\'elie Royer, Alasdair Newson, Patrick P\\'erez

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Streaming](<https://devfeed.tech/topics/streaming.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [audio](<https://devfeed.tech/tags/audio.md>), [aura](<https://devfeed.tech/tags/aura.md>), [avatar](<https://devfeed.tech/tags/avatar.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [denoising](<https://devfeed.tech/tags/denoising.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-models](<https://devfeed.tech/tags/diffusion-models.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [faster](<https://devfeed.tech/tags/faster.md>), [fps](<https://devfeed.tech/tags/fps.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [latency](<https://devfeed.tech/tags/latency.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

AURA generates audio-driven avatar video chunk by chunk using a rolling KV cache, so generation can begin with audio from the current chunk rather than requiring multi-second look-ahead. Its 1.3B model streams 576p at 20.7 fps on one GPU for 16 Hz playback; the 5B model streams 720p at 9.7 fps. A two-stage distillation process aligns a causal student to a teacher with fewer diffusion steps, then uses Distribution Matching Distillation with backward simulation. On standard talking-head benchmarks, the 1.3B model is 23x faster than its bidirectional teacher and has quality comparable to TalkVerse at lower resolution; the 5B model matches its TalkVerse teacher at 720p while running 17x faster. The authors report releasing both models' weights, 30K synthetic videos, and precomputed teacher trajectories.

### Source excerpt

arXiv:2611.26540v1 Announce Type: cross Abstract: Current audio-driven video diffusion models can synthesize realistic talking-head videos but generally require dozens of diffusion denoising steps over multi-second video chunks, making them unsuitable for streaming applications such as voice assistants and virtual avatars, where low-latency and online causal generation are required properties. AURA generates video chunk by chunk with a rolling KV cache, requiring only the audio of the current chunk to start generating, rather than a multi-second audio look-ahead like bidirectional models. It comes in two sizes: AURA 1.3B streams 576p video in real time on a single GPU (20.7 fps for a 16 Hz playback rate), making it suited to interactive avatar applications, and AURA 5B streams 720p video at 9.7 fps. Our two-stage distillation pipeline first aligns the causal student with the teacher, by training it to match the teacher's trajectories with fewer diffusion steps and a chunk-wise sliding window causal attention mask, then applies Distribution Matching Distillation (DMD) with backward simulation to close the remaining quality gap. On standard talking-head benchmarks, AURA 1.3B runs 23x faster than its own bidirectional teacher and reaches a quality comparable to TalkVerse at a lower resolution, and AURA 5B matches its TalkVerse teacher at 720p while running 17x faster. To support reproducibility and further research on streaming avatar generation, we release the weights of both models along with a dataset of 30K synthetic videos and their precomputed teacher trajectories.

## FLUID: Flexible, Evidence-Grounded Situated Task Guidance

DevFeed: [FLUID: Flexible, Evidence-Grounded Situated Task Guidance](<https://devfeed.tech/articles/fluid-flexible-evidence-grounded-situated-task-guidance-145677.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26491>)

Author: Yunqi Zhao, Run Peng, Ze Dong, Ryan Lu, Yinpei Dai, Joyce Chai

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [User Experience](<https://devfeed.tech/topics/user-experience.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [flexible](<https://devfeed.tech/tags/flexible.md>), [fluid](<https://devfeed.tech/tags/fluid.md>), [framework](<https://devfeed.tech/tags/framework.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [human-ai-interaction](<https://devfeed.tech/tags/human-ai-interaction.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [latency](<https://devfeed.tech/tags/latency.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [precision](<https://devfeed.tech/tags/precision.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [response](<https://devfeed.tech/tags/response.md>), [responses](<https://devfeed.tech/tags/responses.md>), [speech](<https://devfeed.tech/tags/speech.md>)

### AI overview

FLUID models situated task guidance around the evidence for an intervention, whether it is necessary, and how flexible its timing is--including when silence is appropriate. Using human-human guidance from WTaG and HoloAssist, the framework supports both evaluation and a real-time guidance agent. The abstract reports that this agent had the strongest overall balance among evaluated systems for response correctness, intervention precision, timely guidance, and latency, while speech overlap remained an area for improvement. The work aims to move evaluation beyond matching a single fixed dialogue response and timing.

### Source excerpt

arXiv:2611.26491v1 Announce Type: new Abstract: Real-time task guidance is inherently flexible: when an agent guides a user through a task, multiple responses and timings can be appropriate, and sometimes the best action is to remain silent. Existing benchmarks often miss this flexibility by treating synthesized dialogues as fixed targets for what to say and when to say it. We introduce FLUID (Flexible, Evidence-Grounded Situated Task Guidance), a unified framework for modeling, evaluating, and enabling flexible interaction in multimodal task guidance. Using naturally occurring human-human guidance from WTaG and HoloAssist, FLUID grounds interventions in their supporting evidence, necessity, and timing flexibility, and uses this representation both to evaluate flexible guidance and to build a real-time agent for task guidance. Our agent achieves the strongest overall balance of response correctness, intervention precision, timely guidance, and response latency among evaluated systems, while still leaving room for improvement in avoiding speech overlap. Together, FLUID moves situated task guidance beyond rigid reference matching toward more natural and adaptive human-AI interaction. Project page: https://fluid-guidance.github.io.

## Disney's new Star Wars robots use AI to react to guests in real time, but can't go off-script

DevFeed: [Disney's new Star Wars robots use AI to react to guests in real time, but can't go off-script](<https://devfeed.tech/articles/disney-s-new-star-wars-robots-use-ai-to-react-to-guests-in-real-time-but-can-t-go-off-script-143632.md>)

Original publisher: [Read original article](<https://www.techspot.com/news/114620-disney-new-star-wars-robots-use-ai-react.html>)

Author: Skye Jacobs

Published: 2026-11-19T14:04:00Z

Content type: news

Language: en

Sources: [TechSpot](<https://devfeed.tech/sources/techspot.md>)

Topics: [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [Humanoid Robots](<https://devfeed.tech/topics/humanoid-robots.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [The Three Main Cloud Service Models](<https://devfeed.tech/topics/the-three-main-cloud-service-models.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [aliens](<https://devfeed.tech/tags/aliens.md>), [amazon-aws](<https://devfeed.tech/tags/amazon-aws.md>), [animatronic](<https://devfeed.tech/tags/animatronic.md>), [animatronics](<https://devfeed.tech/tags/animatronics.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloud](<https://devfeed.tech/tags/aws-cloud.md>), [aws-re-invent](<https://devfeed.tech/tags/aws-re-invent.md>), [camera](<https://devfeed.tech/tags/camera.md>), [compute](<https://devfeed.tech/tags/compute.md>), [conference](<https://devfeed.tech/tags/conference.md>), [content](<https://devfeed.tech/tags/content.md>), [context](<https://devfeed.tech/tags/context.md>), [decision-making](<https://devfeed.tech/tags/decision-making.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [disney](<https://devfeed.tech/tags/disney.md>), [electronics](<https://devfeed.tech/tags/electronics.md>), [guests](<https://devfeed.tech/tags/guests.md>), [park](<https://devfeed.tech/tags/park.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [robots](<https://devfeed.tech/tags/robots.md>), [script](<https://devfeed.tech/tags/script.md>), [star-wars](<https://devfeed.tech/tags/star-wars.md>), [tech-culture](<https://devfeed.tech/tags/tech-culture.md>), [the-mandalorian](<https://devfeed.tech/tags/the-mandalorian.md>)

### AI overview

Disney's three Anzellan robots use cameras and microphones to react to guests and one another, while staying within material approved by Imagineers. AWS cloud models divide the work: a fast model handles immediate reactions such as eye contact, while larger models plan slower exchanges using more context. The system selects prerecorded dialogue and designed animations; it does not generate new speech or motion. This preserves creative control and keeps responses predictable, though whether reactions feel immediate in a noisy crowd remains to be tested. The robots are due to appear at AWS re:Invent, and Disney has not committed to a theme park debut.

### Source excerpt

The characters are named Vilo Vele, Kado Bre'a and Jibu Bre'a. Their first public appearance will be at AWS re:Invent in Las Vegas later this month, where conference attendees can interact with them. Disney has not committed to a park debut. However, the characters and the technology behind them could... Read Entire Article

## Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models

DevFeed: [Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models](<https://devfeed.tech/articles/spool-low-latency-streaming-inference-for-hybrid-vision-language-models-141866.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23228>)

Author: Rui Ning, Om Chabra, Hari Balakrishnan

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [vllm](<https://devfeed.tech/topics/vllm.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [caching](<https://devfeed.tech/tags/caching.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [frame](<https://devfeed.tech/tags/frame.md>), [gated-deltanet](<https://devfeed.tech/tags/gated-deltanet.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [inference](<https://devfeed.tech/tags/inference.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [recurrent](<https://devfeed.tech/tags/recurrent.md>), [sliding-window](<https://devfeed.tech/tags/sliding-window.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [time-to-first-token](<https://devfeed.tech/tags/time-to-first-token.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

Spool speeds up sliding-window video inference for hybrid vision-language models by caching each frame's affine recurrent-state transformation and composing transformations as the window advances, so expired frames can be removed without replaying the window through the model. Removal is exact within each recurrent layer, though expired frames may still influence later layers through cached inputs. In tests across five open-source models and four benchmarks, Spool had the lowest p95 time-to-first-token in all 20 comparisons, reducing latency by a geometric mean of 6.6x versus the faster of two vLLM baselines and improving average accuracy by 2.3 percentage points.

### Source excerpt

arXiv:2611.23228v1 Announce Type: cross Abstract: Applications such as visual accessibility agents and robotics require answering questions about recent video content promptly, over a bounded window of past frames (e.g., the last two minutes). Hybrid vision-language models (VLMs), which interleave attention with linear recurrent layers such as Gated DeltaNet (GDN), are attractive for this setting, but maintaining a sliding window is expensive. The reason is that recurrent layers combine all past frames into a shared state, so expiring old frames normally requires re-running the entire window through the model. We observe that the recurrent update is affine in the state with input-dependent coefficients, so the effect of each frame can be captured as a transformation that composes associatively. We present Spool, a training-free framework that caches per-frame state transformations and composes them efficiently as the window advances, removing expired frames from each recurrent layer's state without replaying frames through the backbone. Removal is exact within each layer; as with key-value eviction in attention, expired frames can retain residual influence through the cached inputs computed by earlier layers. We design a dedicated data structure and GPU kernels to achieve low caching and composition overhead. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool achieves the lowest p95 time-to-first-token in all 20 model-benchmark tests, reducing it by a geometric mean of $6.6\times$ over the faster of two recent vLLM baselines, while improving accuracy by 2.3 percentage points on average.

## How Consumer AI Companions Handle Context Windows, Persistent Memory, and Voice Latency

DevFeed: [How Consumer AI Companions Handle Context Windows, Persistent Memory, and Voice Latency](<https://devfeed.tech/articles/how-consumer-ai-companions-handle-context-windows-persistent-memory-and-voice-latency-139410.md>)

Original publisher: [Read original article](<https://www.sitepoint.com/consumer-ai-companions-context-memory-voice-latency/>)

Author: SitePoint Sponsors

Published: 2026-11-17T07:00:00Z

Content type: article

Language: en

Sources: [SitePoint](<https://devfeed.tech/sources/sitepoint.md>)

Topics: [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [speech-to-speech](<https://devfeed.tech/topics/speech-to-speech.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [404-media](<https://devfeed.tech/tags/404-media.md>), [4k](<https://devfeed.tech/tags/4k.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-companions](<https://devfeed.tech/tags/ai-companions.md>), [apps](<https://devfeed.tech/tags/apps.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [consumer-ai](<https://devfeed.tech/tags/consumer-ai.md>), [context](<https://devfeed.tech/tags/context.md>), [handle](<https://devfeed.tech/tags/handle.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [memory](<https://devfeed.tech/tags/memory.md>), [persistent-memory](<https://devfeed.tech/tags/persistent-memory.md>), [stateful](<https://devfeed.tech/tags/stateful.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [voice](<https://devfeed.tech/tags/voice.md>), [windows](<https://devfeed.tech/tags/windows.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Consumer AI companions maintain continuity by combining a bounded context window with persistent memory: a running summary, extracted user facts, and retrieval of relevant older chats. These layers help but summaries can drift, and retention varies by plan; the article cites a free tier that deletes chat history after seven days of inactivity. Voice latency falls when transcription, LLM output, and text-to-speech stream in sequence, with completed sentences sent to speech synthesis before the full reply is ready; first audio can approach a second on a good connection. Voice also costs more because it uses three model calls and audio bandwidth, so examples include per-minute charges or voice limited to top tiers. The article's practical guidance is to budget prompt space, make stored facts inspectable, stream each voice stage, and protect memory stores because they contain sensitive data.

### Source excerpt

See how consumer AI companions manage context windows, persistent memory and streaming voice, with practical architecture lessons for stateful LLM apps. Continue reading How Consumer AI Companions Handle Context Windows, Persistent Memory, and Voice Latency on SitePoint.

## Decode Early, Verify Late: Streaming Offline Speech Recognizers Without Accuracy Loss

DevFeed: [Decode Early, Verify Late: Streaming Offline Speech Recognizers Without Accuracy Loss](<https://devfeed.tech/articles/decode-early-verify-late-streaming-offline-speech-recognizers-without-accuracy-loss-138984.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20064>)

Author: Doyeob Kim

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [asr](<https://devfeed.tech/topics/asr.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [1-5](<https://devfeed.tech/tags/1-5.md>), [22](<https://devfeed.tech/tags/22.md>), [a100](<https://devfeed.tech/tags/a100.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [after](<https://devfeed.tech/tags/after.md>), [agents](<https://devfeed.tech/tags/agents.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [asr](<https://devfeed.tech/tags/asr.md>), [audio](<https://devfeed.tech/tags/audio.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [complete](<https://devfeed.tech/tags/complete.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [equals](<https://devfeed.tech/tags/equals.md>), [falls](<https://devfeed.tech/tags/falls.md>), [offline](<https://devfeed.tech/tags/offline.md>), [speech](<https://devfeed.tech/tags/speech.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [verify](<https://devfeed.tech/tags/verify.md>)

### AI overview

Decode Early, Verify Late (DEVL) streams offline ASR by showing partial transcripts as drafts, then verifying the latest draft against the complete audio in one forward pass when the turn ends. Under stated input and decoding conditions, its final output equals greedy offline decoding in exact arithmetic. Across five recognizers, 95-100% of finals were byte-identical to offline transcripts, with word error rate within 0.13 points; single-stream time to final fell 1.5-7.3x. For Qwen3-ASR-1.7B on one A100, median final latency was 22-26 ms after turn end at up to 64 streams, 2.7-3.9x sooner than its official streaming mode. The tested system met a 100 ms p99 target at up to 80 streams on Pipecat-STT and 64 on WildASR.

### Source excerpt

arXiv:2611.20064v1 Announce Type: new Abstract: Voice agents need accurate transcripts soon after a user stops speaking. We propose Decode Early, Verify Late (DEVL), a training-free framework for streaming inference with offline ASR models. The server keeps partial transcripts as drafts and verifies the latest against the complete audio in one forward pass at turn end. Under stated input and decoding conditions, the final equals the model's greedy offline output in exact arithmetic. We adapt DEVL to four model families. In our five-recognizer comparison, 95-100% of final transcripts are byte-identical to the offline transcripts, WER stays within 0.13 points of offline, and single-stream time to final falls 1.5-7.3x. Serving Qwen3-ASR-1.7B on one A100, DEVL returns finals a median 22-26 ms after turn end at up to 64 streams, 2.7-3.9x sooner than the official streaming mode. It is the only system we tested that meets a 100 ms p99 target, up to 80 streams on Pipecat-STT and 64 on WildASR.

## LynnReal-Hand: Unlocking World Interaction with Hand Control

DevFeed: [LynnReal-Hand: Unlocking World Interaction with Hand Control](<https://devfeed.tech/articles/lynnreal-hand-unlocking-world-interaction-with-hand-control-137705.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21479>)

Author: Shaohao Rui, Chaoyang Zhang, Xiaofeng Mao, Weijie Ma

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [World models](<https://devfeed.tech/topics/world-models.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [appearance](<https://devfeed.tech/tags/appearance.md>), [camera](<https://devfeed.tech/tags/camera.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [code](<https://devfeed.tech/tags/code.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [fast](<https://devfeed.tech/tags/fast.md>), [flow](<https://devfeed.tech/tags/flow.md>), [fps](<https://devfeed.tech/tags/fps.md>), [generation](<https://devfeed.tech/tags/generation.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-memory](<https://devfeed.tech/tags/gpu-memory.md>), [inference](<https://devfeed.tech/tags/inference.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [interface](<https://devfeed.tech/tags/interface.md>), [latency](<https://devfeed.tech/tags/latency.md>), [model](<https://devfeed.tech/tags/model.md>), [world](<https://devfeed.tech/tags/world.md>)

### AI overview

LynnReal-Hand is a hand-controlled video world model designed to pair fine-grained object interaction with streaming generation. It uses 100K real and synthetic interaction clips with hand-aware supervision, a causal flow map for few-step generation, region-aware score distillation combining video and image priors, and a compact Ultra-Fast-VAE decoder. On HandBench, the full-sequence model reports FVD 181.67 and MANO Dice 93.02; its three-step causal version reports MANO Dice 92.49. At 540p on one H200, streaming reaches 18.03 FPS at 741 ms latency, with optimized peak GPU memory of 34.89 GB. The abstract reports better hand-following accuracy and visual quality than existing approaches, but does not identify the specific comparison systems.

### Source excerpt

arXiv:2611.21479v1 Announce Type: cross Abstract: Interactive video world models enable exploration through text and camera controls, yet direct object interaction requires finer control. Hands provide a natural interface, but translating continuous hand movements into coherent visual responses remains challenging because of complex hand-object dynamics and demanding real-time computation. To address these challenges, we introduce LynnReal-Hand, a hand-controlled video world model that unifies precise interaction modeling with efficient streaming generation. We curate 100K interaction clips spanning diverse real and synthetic environments and develop hand-aware supervision to improve fine-grained hand motions and object interactions. Building on this foundation, we learn a causal flow map for few-step generation and introduce region-aware score distillation that combines complementary video and image priors to refine interaction dynamics and visual appearance. To further reduce inference latency, we propose Ultra-Fast-VAE, a compact video decoder that preserves high reconstruction fidelity with substantially lower computational cost. Extensive evaluations on HandBench demonstrate superior hand-following accuracy and visual quality compared with existing approaches. Our full-sequence model achieves an FVD of 181.67 and a MANO Dice score of 93.02, while the three-step causal model maintains a MANO Dice score of 92.49. For real-time interaction, 540p streaming on a single H200 reaches 18.03 FPS with 741ms latency and an optimized peak GPU memory footprint of only 34.89GB. We will release the datasets, model weights, and the full training and inference code to support research on hand-controlled video world models.

## Recognizing Events Is Not Enough: Why Streaming VLMs Struggle in Compositional Instruction Triggering

DevFeed: [Recognizing Events Is Not Enough: Why Streaming VLMs Struggle in Compositional Instruction Triggering](<https://devfeed.tech/articles/recognizing-events-is-not-enough-why-streaming-vlms-struggle-in-compositional-instruction-triggering-137337.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18743>)

Author: Yehua Huang, Penglei Sun, Xiang Li, Jinhao Zhou, Jiawen Wen, Qingqing Xiao, Zihan Yang, Wenqi Wang, Haolan Chen, Yu Xu, Xiaowen Chu

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [GPU optimization](<https://devfeed.tech/topics/gpu-optimization.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [after](<https://devfeed.tech/tags/after.md>), [agent](<https://devfeed.tech/tags/agent.md>), [atomic](<https://devfeed.tech/tags/atomic.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [device](<https://devfeed.tech/tags/device.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [event](<https://devfeed.tech/tags/event.md>), [events](<https://devfeed.tech/tags/events.md>), [example](<https://devfeed.tech/tags/example.md>), [execution](<https://devfeed.tech/tags/execution.md>), [existing](<https://devfeed.tech/tags/existing.md>), [fine](<https://devfeed.tech/tags/fine.md>), [flash](<https://devfeed.tech/tags/flash.md>), [frame](<https://devfeed.tech/tags/frame.md>), [gap](<https://devfeed.tech/tags/gap.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [rtx-5060](<https://devfeed.tech/tags/rtx-5060.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [vlms](<https://devfeed.tech/tags/vlms.md>)

### AI overview

Recognizing separate events is not enough for streaming video assistants: they must track when temporal conditions are met to notify at the right moment or remain silent. The paper introduces CoTriggerBench, which tests this using temporally constrained instructions, executable event plans, and replay of annotated source events. Its proposed real-time VLM agent, TEMPI, breaks instructions into event queries and temporal constraints, then uses Scenario Memory to track progress across observations beyond the detector's visual window; relevant events update that state even when no notification is due. On CoTriggerBench, TEMPI reports 44.4% instruction-level success, 15.6 percentage points above Qwen3.8-Omni-Flash-Realtime and 18.1 above MOSS-VL-Realtime. The authors also report a 4.90x speedup for an optimized detector on an 8 GB RTX 5060, using overlapping visual encoding and LLM prefix prefill with incoming frames.

### Source excerpt

arXiv:2611.18743v1 Announce Type: cross Abstract: Recognizing individual events does not guarantee correct execution of compositional instructions in streaming video. For example, given the request "Notify me when the pot boils after someone turns on the stove and leaves the kitchen", existing models may recognize the individual events yet notify too early or miss the intended moment, failing to track how their temporal conditions are satisfied over time. To evaluate this gap, we introduce CoTriggerBench, which composes annotated video events into temporally constrained instructions and verifies notification timing and silence through executable event plans and source-event replay. To address this challenge, we propose a real-time VLM agent, TEMPI. TEMPI decomposes instructions into atomic event queries and temporal constraints, and uses Scenario Memory to continuously maintain task progress as event evidence arrives, allowing earlier observations to support subsequent decisions beyond the detector's visual window. Even when no notification is required, relevant events continue to update the scenario state. On CoTriggerBench, TEMPI achieves 44.4% instruction-level success, outperforming the strongest closed-source and fine-tuned open-source VLM baselines by 15.6 (Qwen3.8-Omni-Flash-Realtime) and 18.1 (MOSS-VL-Realtime) percentage points, respectively. To support efficient on-device streaming inference, we further optimize the streaming inference pipeline by overlapping partial visual encoding and LLM prefix prefill with frame arrival. On an 8 GB RTX 5060, our optimized detector implementation achieves a 4.90$\times$ speedup.

## Acting by Intent, Reacting by Sight: Visual Conditioned Reflexes for Real-Time VLM Agents

DevFeed: [Acting by Intent, Reacting by Sight: Visual Conditioned Reflexes for Real-Time VLM Agents](<https://devfeed.tech/articles/acting-by-intent-reacting-by-sight-visual-conditioned-reflexes-for-real-time-vlm-agents-137332.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18723>)

Author: Qiancheng Zhou, Wenhua Zhang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [real-time 3D](<https://devfeed.tech/topics/real-time-3d.md>)

Tags: [acting](<https://devfeed.tech/tags/acting.md>), [agents](<https://devfeed.tech/tags/agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [games](<https://devfeed.tech/tags/games.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [latency](<https://devfeed.tech/tags/latency.md>), [motion](<https://devfeed.tech/tags/motion.md>), [planner](<https://devfeed.tech/tags/planner.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [reflex](<https://devfeed.tech/tags/reflex.md>), [scripts](<https://devfeed.tech/tags/scripts.md>), [token](<https://devfeed.tech/tags/token.md>), [token-consumption](<https://devfeed.tech/tags/token-consumption.md>), [token-cost](<https://devfeed.tech/tags/token-cost.md>), [video-games](<https://devfeed.tech/tags/video-games.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vlm](<https://devfeed.tech/tags/vlm.md>)

### AI overview

ViCoR pairs deliberate VLM gameplay control with a fast visual reflex branch: the main model sets and updates lightweight guidance for recurring situations, while a no-thinking VLM checks live local views and motion cues to choose sub-second actions. The authors report that the reflex branch handled 67.8% of high-frequency reactions, with 680 ms per check; average decision latency fell from 14.0 to 5.4 seconds and token use by 36.9%. Across 17 2D and 3D GameWorld games, task success rose from 52.9% to 64.7% and normalized progress from 66.5% to 80.4%, compared with the strongest deliberative VLM and script baselines. In fast-paced games, refining reflexes across episodes increased progress by 1.67x. The abstract also reports that a low-effort planner exceeded high-effort deliberative baselines by 8.1% at one-third the token cost.

### Source excerpt

arXiv:2611.18723v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have shown strong capabilities as generalist agents that perceive, reason, and act in interactive environments. Yet real-time environments such as video games demand both deliberate control for exploration and split-second reactions to fast-moving hazards. Existing agents struggle to handle both: relying solely on direct VLM actions is too slow for sudden events, whereas delegating execution to fixed scripts leaves the agent blind at the moment of action, leading to frequent misfires whenever the scene shifts. Humans overcome this by pairing normal conscious control with conditioned reflexes-while the mind handles deliberate actions and shapes reflex rules, the reflex itself still relies on live vision to make the final instant decision. Emulating this mechanism, we propose ViCoR (Visual Conditioned Reflex). Alongside normal gameplay actions, the main VLM defines and updates lightweight situational guidance for recurring events, while a fast visual reflex branch (invoking the VLM in no-thinking mode) inspects live local views with motion cues to make the final sub-second action decisions. By offloading 67.8% of high-frequency reactions to the reflex branch (680 ms per check), ViCoR reduces average per-action decision latency by 2.6x (from 14.0 s to 5.4 s) and cuts token consumption by 36.9%. Across 17 diverse 2D and 3D games in the GameWorld benchmark, ViCoR achieves state-of-the-art performance, improving task success rate by +11.8% (from 52.9% to 64.7%) and average normalized progress by +13.9% (from 66.5% to 80.4%) over the strongest deliberative VLM and script baselines. In fast-paced real-time games, ViCoR refines its reflexes across episodes to boost progress by 1.67x, and even with a low-effort main planner, surpasses high-effort deliberative baselines by +8.1% at one-third of the token cost.

## AURA 2.0: From User-Driven Responses to Model-Initiated Conversations in Streaming Video

DevFeed: [AURA 2.0: From User-Driven Responses to Model-Initiated Conversations in Streaming Video](<https://devfeed.tech/articles/aura-2-0-from-user-driven-responses-to-model-initiated-conversations-in-streaming-video-132306.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17048>)

Author: Shuhan Li, Huankang Guan, Xintong Guo, Yang Bo, Yuyang He, Dunyuan Xu, Jingyan Chen, Fang Liu, Tsz Kin Wan, Jinpeng Chen, Zhenqi He, Yuxuan Hu, Wenbo Li, Han Gao, Qijun Li, Long Chen, Hongsheng Li, Ru

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [Conversational AI](<https://devfeed.tech/topics/conversational-ai.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [function calling](<https://devfeed.tech/topics/function-calling.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [ai-assistants](<https://devfeed.tech/tags/ai-assistants.md>), [aura](<https://devfeed.tech/tags/aura.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [information](<https://devfeed.tech/tags/information.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model-weights](<https://devfeed.tech/tags/model-weights.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [performance](<https://devfeed.tech/tags/performance.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-video](<https://devfeed.tech/tags/streaming-video.md>), [tool-calling](<https://devfeed.tech/tags/tool-calling.md>), [tools](<https://devfeed.tech/tags/tools.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

AURA 2.0 extends streaming video assistants beyond user-prompted replies: Model-Initiated Conversation (MIC) lets it start discussions about events in an evolving video, while Stream-Aware Tool Calling (SATC) uses visual and conversational context to call external tools and incorporate their results. Interactive Visual Grounding (IVG) handles user-marked objects and returns frame-aligned bounding boxes linking responses to regions. The system is trained on a dedicated streaming interaction dataset using a three-stage scheme, and Long-Horizon Interaction Memory (LHIM) supports continuity across extended streams. The authors report leading performance among open-source streaming models on public streaming benchmarks and positive targeted evaluations of the new capabilities; the abstract provides no comparative scores or further evaluation details. Weights and inference code are released.

### Source excerpt

arXiv:2611.17048v1 Announce Type: cross Abstract: Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) enables the model to invoke external tools using the current visual and conversational context and incorporate returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities. We open-source the model weights and inference code.

## Lakehouse//RT: Scale real-time analytics, not your bill

DevFeed: [Lakehouse//RT: Scale real-time analytics, not your bill](<https://devfeed.tech/articles/lakehouse-rt-scale-real-time-analytics-not-your-bill-130981.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/lakehousert-scale-real-time-analytics-not-your-bill>)

Author: JooHo Yeo; Kaitlin Baumgardner

Published: 2026-11-12T16:53:43Z

Content type: article

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [bill](<https://devfeed.tech/tags/bill.md>), [compute](<https://devfeed.tech/tags/compute.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [dashboards](<https://devfeed.tech/tags/dashboards.md>), [data-warehousing](<https://devfeed.tech/tags/data-warehousing.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [platform](<https://devfeed.tech/tags/platform.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-analytics](<https://devfeed.tech/tags/real-time-analytics.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

Lakehouse//RT aims to handle high, bursty real-time query volumes without scaling costs proportionally. Its approach combines higher query concurrency and faster end-to-end execution with capacity scaling in small increments, including one node at a time. It queries governed lakehouse data directly, avoiding a separate serving stack and duplicated data. The text reports Aurasell sustained over 250 QPS on one small warehouse, with queries 4-5x faster and 10x lower infrastructure costs; VisionHeight reported over 90% lower costs and nearly 3x faster responses than Apache Pinot on its heaviest query; nOps reported infrastructure costs cut by more than 6x and eliminated 40TB of duplicated data. These are vendor-reported customer outcomes. Lakehouse//RT is described as available in beta.

### Source excerpt

AI agents are pushing the limits of data warehouses. A human analyst might run a...

## Scaling real-time bidding and publisher monetization with the AWS Video Streaming Advertising Lens

DevFeed: [Scaling real-time bidding and publisher monetization with the AWS Video Streaming Advertising Lens](<https://devfeed.tech/articles/scaling-real-time-bidding-and-publisher-monetization-with-the-aws-video-streaming-advertising-lens-127784.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/media/scaling-real-time-bidding-and-publisher-monetization-with-the-aws-video-streaming-advertising-lens/>)

Author: Abraham Lobo

Published: 2026-11-10T18:51:51Z

Content type: release

Language: en

Sources: [AWS for M&E Blog](<https://devfeed.tech/sources/aws-for-m-e-blog.md>)

Topics: [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [advertising](<https://devfeed.tech/tags/advertising.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-web-services](<https://devfeed.tech/tags/amazon-web-services.md>), [amazon-web-services-aws](<https://devfeed.tech/tags/amazon-web-services-aws.md>), [announce](<https://devfeed.tech/tags/announce.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-well-architected](<https://devfeed.tech/tags/aws-well-architected.md>), [aws-well-architected-framework](<https://devfeed.tech/tags/aws-well-architected-framework.md>), [best](<https://devfeed.tech/tags/best.md>), [bidding](<https://devfeed.tech/tags/bidding.md>), [industries](<https://devfeed.tech/tags/industries.md>), [lens](<https://devfeed.tech/tags/lens.md>), [marketing-advertising](<https://devfeed.tech/tags/marketing-advertising.md>), [media-entertainment](<https://devfeed.tech/tags/media-entertainment.md>), [monetization](<https://devfeed.tech/tags/monetization.md>), [news](<https://devfeed.tech/tags/news.md>), [publisher](<https://devfeed.tech/tags/publisher.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [security](<https://devfeed.tech/tags/security.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [time](<https://devfeed.tech/tags/time.md>), [video](<https://devfeed.tech/tags/video.md>), [video-streaming](<https://devfeed.tech/tags/video-streaming.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The AWS Well-Architected Video Streaming Advertising Lens V3 extends its guidance beyond real-time bidding infrastructure to publisher monetization, FAST and SGAI playout, privacy-enhanced data collaboration, agentic AI, and unified linear and digital ad operations. Its design patterns include parallel prebid requests with strict server-side timeouts and stateless horizontal scaling; SCTE-35 ad markers in cacheable content manifests; and shared data foundations for cross-platform attribution. It also discusses using AdCP and ARTF for AI advertising workflows, with human oversight, bias detection, and explainability. The article describes the lens as guidance for sub-100 ms auctions and tenfold traffic spikes, while noting privacy regulation, third-party signal loss, and infrastructure cost as architectural concerns. It is available as a custom lens and whitepaper; the suggested first steps are to import it into AWS Well-Architected Tool, assess an advertising workload, and consult the whitepaper.

### Source excerpt

At Amazon Web Services (AWS), we're excited to announce our next evolution of the AWS Well-Architected Video Streaming Advertising (VSA) Lens. A Well-Architected Lens extends the AWS Well-Architected Framework with industry-specific or technology-specific guidance. The VSA Lens provides architectural best practices specifically for video streaming advertising workloads. The lens consists of a VSA whitepaper and a custom lens, available to the public on the AWS Well-Architected custom lens GitHub repository

## GARP: Real-Time Generative Agents on Local Hardware in a Game Engine

DevFeed: [GARP: Real-Time Generative Agents on Local Hardware in a Game Engine](<https://devfeed.tech/articles/garp-real-time-generative-agents-on-local-hardware-in-a-game-engine-126068.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11064>)

Author: Piero Molino, Patrick John Chia, Federico Bianchi, Jonathan P. Chen, Paul Szerlip

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Local AI](<https://devfeed.tech/topics/local-ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Godot](<https://devfeed.tech/topics/godot.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [conversations](<https://devfeed.tech/tags/conversations.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-hc](<https://devfeed.tech/tags/cs-hc.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [decision](<https://devfeed.tech/tags/decision.md>), [emotion](<https://devfeed.tech/tags/emotion.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [game](<https://devfeed.tech/tags/game.md>), [game-engine](<https://devfeed.tech/tags/game-engine.md>), [generative](<https://devfeed.tech/tags/generative.md>), [godot](<https://devfeed.tech/tags/godot.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [json](<https://devfeed.tech/tags/json.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [local](<https://devfeed.tech/tags/local.md>), [memory](<https://devfeed.tech/tags/memory.md>), [real-time](<https://devfeed.tech/tags/real-time.md>)

### AI overview

GARP runs generative agents in Godot using a local 3B-parameter open-weight model, with no cloud service. Its Character Behavior System combines planning, memory and reflection with LLM-appraised emotions, four graded personality axes that affect both simulation parameters and prompts, six time-pressured needs, and relationships updated after conversations. To manage decisions, it narrows more than 50 candidate actions to typically 3-8, constrains outputs with JSON schemas, and surfaces important memories in prompts. An adjustable game clock, advance decisions, tiered caching, prompt compression and asynchronous execution help keep the game responsive while the model thinks. In nine scenarios, 25 of 27 runs passed log-based checks; both failures were in the birthday-party scenario, where a guest left the event out of their daily plan.

### Source excerpt

arXiv:2611.11064v1 Announce Type: new Abstract: Generative agents are game characters whose plans, conversations, and memories are written on the fly by a large language model (LLM). The original Generative Agents demonstration used cloud inference in a custom simulation. We present GARP, a platform for real-time generative agents inside Godot, and describe the Character Behavior System used in GARP 2. The complete loop runs on local hardware using a 3B-parameter open-weight model and no cloud service. Alongside the original planning, memory, and reflection, GARP gives each agent an emotion system driven by LLM appraisal, four graded personality axes drawn from the Big Five that set simulation parameters as well as prompt text, six needs whose urgency grows over time, and pairwise relationships updated after every conversation. The system narrows 50+ candidate actions to typically 3-8, constrains model responses with JSON schemas, and surfaces important memories in decision prompts. A layered optimization strategy (an adjustable game clock, decisions taken ahead of need, multi-tier caching, prompt compression, and fully asynchronous execution) keeps the game responsive while the model thinks. In a nine-scenario evaluation, 25 of 27 runs passed log-based checks. Both failures occurred in the birthday party scenario, where one guest omitted the event from their daily plan.

## Announcing streamed responses for Appwrite Sites and Functions

DevFeed: [Announcing streamed responses for Appwrite Sites and Functions](<https://devfeed.tech/articles/announcing-streamed-responses-for-appwrite-sites-and-functions-127498.md>)

Original publisher: [Read original article](<https://appwrite.io/blog/post/announcing-streaming-responses>)

Author: Harsh Mahajan

Published: 2026-11-10T00:00:00Z

Content type: release

Language: en

Sources: [Appwrite Blog](<https://devfeed.tech/sources/appwrite-blog.md>)

Topics: [Appwrite](<https://devfeed.tech/topics/appwrite.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-features](<https://devfeed.tech/tags/ai-features.md>), [ai-responses](<https://devfeed.tech/tags/ai-responses.md>), [ai-sdk](<https://devfeed.tech/tags/ai-sdk.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [api](<https://devfeed.tech/tags/api.md>), [astro](<https://devfeed.tech/tags/astro.md>), [async](<https://devfeed.tech/tags/async.md>), [batch](<https://devfeed.tech/tags/batch.md>), [brotli](<https://devfeed.tech/tags/brotli.md>), [browser](<https://devfeed.tech/tags/browser.md>), [functions](<https://devfeed.tech/tags/functions.md>), [model](<https://devfeed.tech/tags/model.md>), [pages](<https://devfeed.tech/tags/pages.md>), [parts](<https://devfeed.tech/tags/parts.md>), [render](<https://devfeed.tech/tags/render.md>), [responses](<https://devfeed.tech/tags/responses.md>), [server](<https://devfeed.tech/tags/server.md>), [sites](<https://devfeed.tech/tags/sites.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

Appwrite Cloud now streams server-rendered pages and function responses, delivering chunks as they are written. Frameworks that already render HTML in parts need no extra configuration; functions can return `res.sendStream()` with runtime-native chunk sources, and Node.js, Bun, and Deno can also return a Fetch API `Response` with a `ReadableStream`. Function streams reach clients calling through the function's domain; the Create execution endpoint and SDK calls return the complete body after streaming ends. Site streams run up to the site timeout (maximum 30 seconds); function streams through a domain run up to the shorter of the function timeout and 60 seconds, and count toward execution time. Use Appwrite Realtime for longer-running updates. Preview URLs displaying the Appwrite banner do not stream HTML. Static files and responses with a known size remain sent in one piece. Support excludes Rust and several deprecated runtimes; Go requires `types-for-go/v4` v4.0.11 or later. Self-hosted installations enable streaming by default in releases that include it, with `_APP_COMPUTE_STREAMING=disabled` available to turn it off.

### Source excerpt

Appwrite Sites and Functions now stream their responses, so server-rendered pages show up in parts as they render and AI output reaches your users while the model is still writing.

## TurnWise Outperforms Streaming Video RL Baselines Across Three Benchmarks

DevFeed: [TurnWise Outperforms Streaming Video RL Baselines Across Three Benchmarks](<https://devfeed.tech/articles/turnwise-outperforms-streaming-video-rl-baselines-across-three-benchmarks-123552.md>)

Original publisher: [Read original article](<https://hackernoon.com/turnwise-outperforms-streaming-video-rl-baselines-across-three-benchmarks?source=rss>)

Author: aimodels44

Published: 2026-11-09T07:15:02Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [credit-assignment](<https://devfeed.tech/tags/credit-assignment.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [ml-research](<https://devfeed.tech/tags/ml-research.md>), [multimodal-ai](<https://devfeed.tech/tags/multimodal-ai.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rl](<https://devfeed.tech/tags/rl.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [streaming-video](<https://devfeed.tech/tags/streaming-video.md>), [streaming-video-rl](<https://devfeed.tech/tags/streaming-video-rl.md>), [turnwise](<https://devfeed.tech/tags/turnwise.md>), [video](<https://devfeed.tech/tags/video.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

TurnWise addresses shared trajectory credit in streaming video RL by assigning each answer-or-wait decision a group-relative advantage based on its remaining return. It scores a prefix with later actions replaced by silence, subtracts accrued reward, and uses the existing trajectory evaluations without a critic, extra rollouts, or judge calls. Separately, its loss averages token losses within each turn and weights turns equally, preventing long answers from dominating short waits. The authors report the best aggregate scores among the evaluated methods on ProactiveVQA, OVO-Timing, and StreamingBench with Qwen2.5-VL-3B and Qwen3-VL-4B; they also report improved timing and less repetition from turn-mean aggregation versus credit-only optimization. The excerpt gives no numerical scores or effect sizes. The credit signal depends on the fixed trajectory scorer and its penalties, and the reported evidence is limited to the listed benchmarks and configurations.

### Source excerpt

Learn how TurnWise improves streaming video reinforcement learning through turn-level credit assignment, remaining-return advantages, and GRPO optimization.

## Real-Time Execution with Autoregressive Policies

DevFeed: [Real-Time Execution with Autoregressive Policies](<https://devfeed.tech/articles/real-time-execution-with-autoregressive-policies-124441.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.13355>)

Author: Sangkyu Lee, Seohyeon Park, Tackgeun You, Avi Caciularu, Idan Szpektor, Hwasup Lim, Youngjae Yu

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [real-time](<https://devfeed.tech/topics/real-time.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [environments](<https://devfeed.tech/tags/environments.md>), [flow-matching](<https://devfeed.tech/tags/flow-matching.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [performance](<https://devfeed.tech/tags/performance.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [vision-language-action](<https://devfeed.tech/tags/vision-language-action.md>)

### AI overview

The paper reports that autoregressive Vision-Language-Action policies can support real-time execution by adjusting the tokenization horizon and using constrained decoding to enforce latency bounds, enabling multi-trajectory decoding. In simulated and real-world environments, the policy reportedly outperformed an equivalent-level flow-matching counterpart and improved task completion speeds compared with synchronous inference. The abstract also cites faster convergence and better instruction-following generalizability as advantages. These findings are based on the supplied abstract; no implementation details or quantitative results are provided.

### Source excerpt

arXiv:2606.13355v2 Announce Type: replace-cross Abstract: Real-time execution, enabled by asynchronous inference that ensures both smooth action trajectories and fast reactivity, is critical for realistic deployments of large-scale Vision-Language-Action models. However, recent work on real-time execution primarily focuses on variants of diffusion policies, even though it is more critical for autoregressive policies given their slower rollout speed in synchronous inference. In contrast, we demonstrate that autoregressive policies can achieve real-time execution by adjusting the tokenization horizon and applying constrained decoding, thereby guaranteeing strict latency bounds that enable multi-trajectory decoding to maximize performance. Across simulated and real-world environments, we find that the autoregressive policy consistently outperforms its equivalent-level flow-matching policy counterpart while achieving significantly improved task completion speeds from synchronous inference. Coupled with the inherent advantages of autoregressive policies, such as faster convergence and better generalizability in instruction-following, these results confirm that autoregressive policies can remain a competitive policy type supporting real-time execution.

## Microsoft patent filing describes AI that rewrites what people say in livestreams in real time

DevFeed: [Microsoft patent filing describes AI that rewrites what people say in livestreams in real time](<https://devfeed.tech/articles/microsoft-patent-filing-describes-ai-that-rewrites-what-people-say-in-livestreams-in-real-time-122131.md>)

Original publisher: [Read original article](<https://www.techspot.com/news/114477-microsoft-patent-filing-describes-ai-rewrites-what-people.html>)

Author: Daniel Sims

Published: 2026-11-07T13:51:00Z

Content type: news

Language: en

Sources: [TechSpot](<https://devfeed.tech/sources/techspot.md>)

Topics: [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [ai-editing](<https://devfeed.tech/tags/ai-editing.md>), [audio](<https://devfeed.tech/tags/audio.md>), [broadcast](<https://devfeed.tech/tags/broadcast.md>), [deepfake](<https://devfeed.tech/tags/deepfake.md>), [fact-checking](<https://devfeed.tech/tags/fact-checking.md>), [fraud](<https://devfeed.tech/tags/fraud.md>), [game](<https://devfeed.tech/tags/game.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [language](<https://devfeed.tech/tags/language.md>), [latency](<https://devfeed.tech/tags/latency.md>), [livestream](<https://devfeed.tech/tags/livestream.md>), [livestreams](<https://devfeed.tech/tags/livestreams.md>), [matching](<https://devfeed.tech/tags/matching.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [patent](<https://devfeed.tech/tags/patent.md>), [patents](<https://devfeed.tech/tags/patents.md>), [performance](<https://devfeed.tech/tags/performance.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

Microsoft's patent filing describes two uses for real-time AI editing of livestreams. One claim would detect words and replace them based on viewer profile information, potentially cleaning up pauses and filler, censoring profanity, or changing wording based on internet searches. The text says the claim does not specify whether viewers would be told what changed or why, and describes lip-movement adjustments that could conceal edits. Microsoft says the filing is intended to clean up and translate broadcasts; the article notes accuracy and misinformation concerns, while cautioning that patents do not necessarily become products. A separate claim would translate livestreams with matching lip sync and retain the original recording as a "ground truth."

### Source excerpt

AI-edited and AI-generated videos have been among the most popular generative AI applications since the technology's initial explosion in popularity. However, a Microsoft patent filing from late last month, recently uncovered by Patentlyze, aims to enhance how AI can edit real-time streaming. Read Entire Article

## StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos

DevFeed: [StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos](<https://devfeed.tech/articles/streamgaze-gaze-guided-temporal-reasoning-and-proactive-understanding-in-streaming-videos-119986.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2512.01707>)

Author: Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [ar](<https://devfeed.tech/tags/ar.md>), [ar-glasses](<https://devfeed.tech/tags/ar-glasses.md>), [augmented-reality](<https://devfeed.tech/tags/augmented-reality.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [develop](<https://devfeed.tech/tags/develop.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [proactive](<https://devfeed.tech/tags/proactive.md>), [process](<https://devfeed.tech/tags/process.md>), [prompting](<https://devfeed.tech/tags/prompting.md>), [qa](<https://devfeed.tech/tags/qa.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [temporal-reasoning](<https://devfeed.tech/tags/temporal-reasoning.md>), [videos](<https://devfeed.tech/tags/videos.md>)

### AI overview

StreamGaze benchmarks whether multimodal large language models can use human gaze to reason about streaming video and anticipate user intent. It separates past and present tasks, answered from observations available at query time, from proactive tasks that require continuous frame processing and a response at the appropriate future moment. Its QA pipeline aligns egocentric video with raw gaze trajectories using fixation extraction, region-specific visual prompting, and scanpath construction to create spatiotemporally grounded questions. The authors report substantial performance gaps between state-of-the-art models and humans across tasks, pointing to limitations in gaze-based temporal reasoning, intent modeling, and proactive prediction. The abstract says the work also analyzes gaze prompting strategies, model reasoning, and task-specific failure modes; it provides no detailed comparative results.

### Source excerpt

arXiv:2512.01707v4 Announce Type: replace-cross Abstract: Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intent for realistic applications such as Augmented Reality (AR) glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether Multimodal Large Language Models (MLLMs) can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs utilize gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively assess streaming video understanding. Past and present tasks require immediate responses based on observations available at query time, whereas proactive tasks require models to continuously process incoming frames and respond at the appropriate future moment. To build StreamGaze, we develop a gaze-video Question Answering (QA) generation pipeline that aligns egocentric videos with raw gaze trajectories through fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, highlighting key limitations in gaze-based temporal reasoning, intent modeling, and proactive prediction. We further provide detailed analyses of gaze prompting strategies, reasoning behaviors, and task-specific failure modes, offering insights into current limitations and directions for future research. All data and code are publicly available to support research in gaze-guided streaming video understanding.

## Google Cloud API Gateway Introduces Streaming Support for Real-Time and AI-Powered Applications

DevFeed: [Google Cloud API Gateway Introduces Streaming Support for Real-Time and AI-Powered Applications](<https://devfeed.tech/articles/google-cloud-api-gateway-introduces-streaming-support-for-real-time-and-ai-powered-applications-119504.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/gcp-api-gateway-streaming/>)

Author: Sergio De Simone

Published: 2026-11-05T16:00:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [Streaming](<https://devfeed.tech/topics/streaming.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-workflows](<https://devfeed.tech/tags/agent-workflows.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [ai-powered-applications](<https://devfeed.tech/tags/ai-powered-applications.md>), [api](<https://devfeed.tech/tags/api.md>), [api-gateway](<https://devfeed.tech/tags/api-gateway.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [communication](<https://devfeed.tech/tags/communication.md>), [development](<https://devfeed.tech/tags/development.md>), [gcp-api-gateway-streaming](<https://devfeed.tech/tags/gcp-api-gateway-streaming.md>), [google](<https://devfeed.tech/tags/google.md>), [google-cloud](<https://devfeed.tech/tags/google-cloud.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [model](<https://devfeed.tech/tags/model.md>), [news](<https://devfeed.tech/tags/news.md>), [preview](<https://devfeed.tech/tags/preview.md>), [public](<https://devfeed.tech/tags/public.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [real-time-communication](<https://devfeed.tech/tags/real-time-communication.md>), [responses](<https://devfeed.tech/tags/responses.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

### AI overview

Google Cloud API Gateway's public preview adds streaming while retaining existing gateway policies and security controls. It supports SSE, chunked transfer and HTTP response streaming, WebSockets, and gRPC, with connections open for up to one hour. Enabling it requires the `--enable-streaming` flag when creating a gateway; existing gateways cannot be switched in place. The excerpt says existing API specifications need no changes and lists API key enforcement, JWT validation, backend service account authentication, logging, monitoring, routing, and path templating as continuing to work. Predictable hostnames can be determined before gateway creation. For AI applications, the feature can stream progressive model responses and agent updates; model routing offers one OpenAI-compatible endpoint for Gemini, Claude, or OpenAI gpt-oss based on the request's `model` field. A monitoring caveat: request-latency metrics measure the whole stream, not time to first byte, so the latency chart may not reflect when users first receive a response.

### Source excerpt

Google has announced the public preview of streaming support for API Gateway, extending its traditional request-response model to support applications requiring real-time communication. According to Google, this is particularly relevant for AI applications that provide streaming model responses, multi-step agent workflows, and interactive sessions. By Sergio De Simone

## Runway Characters: Serving a 57B Video Diffusion Transformer for Real-Time Conversation

DevFeed: [Runway Characters: Serving a 57B Video Diffusion Transformer for Real-Time Conversation](<https://devfeed.tech/articles/runway-characters-serving-a-57b-video-diffusion-transformer-for-real-time-conversation-119097.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05642>)

Author: Rohan Agarwal, Robin Andeer, Alex Armbruster, Bennett Arthur, Michail Christos Doukas, Piero Esposito, Qiming Fang, Aron Filkey, Bryan Fox, Anastasis Germanidis, Jonathan Granskog, Corina Gurau, Robin

Published: 2026-11-05T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [real-time](<https://devfeed.tech/topics/real-time.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [denoising](<https://devfeed.tech/tags/denoising.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [fp8](<https://devfeed.tech/tags/fp8.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [h100](<https://devfeed.tech/tags/h100.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [latency](<https://devfeed.tech/tags/latency.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [optimizations](<https://devfeed.tech/tags/optimizations.md>), [production](<https://devfeed.tech/tags/production.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [runway](<https://devfeed.tech/tags/runway.md>), [serving](<https://devfeed.tech/tags/serving.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

Runway Characters serves an approximately 57B-parameter autoregressive video diffusion model at 704x1088 and 24 fps on four H100 GPUs, with a voice agent in the loop. To reduce repeated work across 240 attention calls per four-frame chunk, the system moves KV-cache processing to write time using a ring cache, one-time RoPE and FP8 quantization, and a delta rotation when positions reset. It also fuses tensor-parallel all-reduce into the FP8 GEMM using in-switch reduction and assigns video-frame decoding its own GPU. The diffusion step fell from 240.9 ms to 159.3 ms without the fused kernel, with no detected lip-sync-score degradation; the fused version reaches 147.4 ms, after a production rollback and re-enablement. Over seven production weeks, 99.5% of sessions streaming at least one second averaged above 23.5 fps. The abstract says two open-source systems tested on the same GPUs at four denoising passes missed real time or added over a second of latency, at one-third to one-half the resolution; one latency figure is an estimate.

### Source excerpt

arXiv:2611.05642v1 Announce Type: new Abstract: Conversational video characters must generate video as fast as it plays, for one user at a time: at 24 fps, a sustained 166.7 ms per four-frame chunk at batch size one, with only a few chunks of buffering. We describe the system behind Runway Characters, which serves a $\approx$57B-parameter autoregressive video diffusion transformer (four denoising passes per chunk) at 704$\times$1088 and 24 fps on four H100 GPUs with a voice agent in the loop. Each chunk takes 240 attention calls, each reading a 5,788-slot KV cache in which only 757 tokens are new, and 29% of the original step was whole-cache work repeated in every call. We move that work to write time (a ring cache, RoPE and FP8 quantization applied once per write, and a delta rotation when positions reset), fuse the tensor-parallel all-reduce into the FP8 GEMM with in-switch reduction, and give the decoder that turns latents into video frames its own GPU. Without the fused kernel, our optimizations cut the diffusion step from 240.9 to 159.3 ms (1.51$\times$, 24.5 fps sustained) with no detected degradation in lip-sync scores; the fused kernel, rolled back once in production and since re-enabled, reaches 147.4 ms (1.63$\times$, 43% of FP8 peak). In seven weeks of production the worker averaged above 23.5 fps in 99.5% of sessions that streamed for at least a second, and our pipeline takes 0.49 s of a 2.1 s median conversational turn. On the same GPUs, the two open-source real-time video systems we benchmarked, at four denoising passes, miss real time or add over a second of latency (one by estimate), at a third to half of our resolution.

## Building ultra-low-latency voice AI applications on AWS

DevFeed: [Building ultra-low-latency voice AI applications on AWS](<https://devfeed.tech/articles/building-ultra-low-latency-voice-ai-applications-on-aws-115498.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/building-ultra-low-latency-voice-ai-applications-on-aws/>)

Author: Nolan Chen

Published: 2026-11-03T16:40:54Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Conversational AI](<https://devfeed.tech/topics/conversational-ai.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Amazon EC2](<https://devfeed.tech/topics/amazon-ec2.md>), [Amazon Connect](<https://devfeed.tech/topics/amazon-connect.md>), [Amazon SageMaker JumpStart](<https://devfeed.tech/topics/amazon-sagemaker-jumpstart.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-model](<https://devfeed.tech/tags/ai-model.md>), [amazon-connect](<https://devfeed.tech/tags/amazon-connect.md>), [amazon-ec2](<https://devfeed.tech/tags/amazon-ec2.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [aws](<https://devfeed.tech/tags/aws.md>), [deploy](<https://devfeed.tech/tags/deploy.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [has](<https://devfeed.tech/tags/has.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [low-latency-voice-ai](<https://devfeed.tech/tags/low-latency-voice-ai.md>), [sonic](<https://devfeed.tech/tags/sonic.md>), [speech](<https://devfeed.tech/tags/speech.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [text-to-speech](<https://devfeed.tech/tags/text-to-speech.md>), [voice](<https://devfeed.tech/tags/voice.md>), [voice-ai](<https://devfeed.tech/tags/voice-ai.md>)

### AI overview

Cartesia's Sonic and Ink use State Space Models for streaming voice workloads, with the article reporting approximately 90 ms first audio byte for Sonic and sub-100 ms first-byte latency for both models. Sonic generates speech before receiving all the text; Ink streams partial transcripts and provides semantic turn detection without a separate voice activity detection model. Sonic can run as an AWS Marketplace model package on a managed, in-VPC SageMaker AI endpoint, which the article presents as the faster path with less operational overhead. Alternatively, teams can run Cartesia's container on EC2 GPU infrastructure for more control over runtime, networking, and scaling, while taking responsibility for provisioning, updates, monitoring, and capacity tuning. The deployment instructions cover Sonic; Ink is available through the Cartesia API.

### Source excerpt

Learn how Cartesia, an AWS Partner, uses purpose-built State Space Models and AWS infrastructure to deliver sub-100ms text-to-speech and speech-to-text at global scale, and how to deploy its Sonic model on Amazon SageMaker AI or self-managed Amazon EC2 GPU instances.

## SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding

DevFeed: [SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding](<https://devfeed.tech/articles/surgstream-a-workflow-anchored-proactive-streaming-framework-and-benchmark-for-real-time-surgical-video-understanding-115250.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01982>)

Author: Ruipeng Ma, Qing Xu, Jie Liang, Zhen Chen

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Video Analytics](<https://devfeed.tech/topics/video-analytics.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [real-time](<https://devfeed.tech/topics/real-time.md>)

Tags: [alerts](<https://devfeed.tech/tags/alerts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [framework](<https://devfeed.tech/tags/framework.md>), [learning](<https://devfeed.tech/tags/learning.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [proactive](<https://devfeed.tech/tags/proactive.md>), [qa](<https://devfeed.tech/tags/qa.md>), [queries](<https://devfeed.tech/tags/queries.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [responses](<https://devfeed.tech/tags/responses.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [video](<https://devfeed.tech/tags/video.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

SurgStream targets real-time surgical video understanding by combining scene perception, memory of past events, and timely responses. Workflow-Anchored Memory records structured instrument-action-target observations and consolidates them into phase-level summaries for compact, queryable recall. Deadline-aware Transition Learning supervises response states within clinical windows to help the system respond before procedural deadlines. The accompanying SurgStream-Bench contains 11.7K streaming interactions spanning real-time perception, backward memory queries, forward proactive QA, proactive alerts, and caption memory. The paper reports gains over state-of-the-art methods, especially on memory and forward proactive QA; the supplied abstract gives no quantitative results or evaluation details.

### Source excerpt

arXiv:2611.01982v1 Announce Type: cross Abstract: Surgical video understanding is essential for intraoperative assistance such as workflow monitoring, instrument preparation and procedural documentation. Delivering such assistance in real time requires a model to jointly perceive the ongoing scene, remember past surgical events, and respond before clinically relevant moments pass. Existing approaches provide limited support for jointly retaining procedural evidence and deciding when to respond in streaming surgical video. In this work, we propose SurgStream, a workflow-anchored proactive streaming framework for real-time surgical video understanding. We first introduce Workflow-Anchored Memory (WAM) to record structured instrument-action-target observations and fold them into phase-level summaries, providing compact and queryable surgical memory. Deadline-aware Transition Learning (DTL) is then devised to supervise response states within clinical windows, enabling timely responses before procedural deadlines. We further introduce SurgStream-Bench, comprising 11.7K streaming interactions across various tasks covering real-time perception, backward memory queries, forward proactive QA, proactive alerts and caption memory. Experiments on SurgStream-Bench show gains over state-of-the-art methods, especially in memory and forward proactive QA. The source code will be available.

## SCRAMJet: An Ultra Low Latency JetClass-Scale Transformer for the HL-LHC Level-1 Trigger

DevFeed: [SCRAMJet: An Ultra Low Latency JetClass-Scale Transformer for the HL-LHC Level-1 Trigger](<https://devfeed.tech/articles/scramjet-an-ultra-low-latency-jetclass-scale-transformer-for-the-hl-lhc-level-1-trigger-114338.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.00086>)

Author: Aadeesh Sharma, Chang Sun, Lauri Laatu, Arianna Cox, Benedikt Maier, Alexander Tapper, Maria Spiropulu, Wayne Luk, Zhiqiang Que

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [quantization](<https://devfeed.tech/topics/quantization.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>), [data type](<https://devfeed.tech/topics/data-type.md>)

Tags: [area](<https://devfeed.tech/tags/area.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [attention](<https://devfeed.tech/tags/attention.md>), [budget](<https://devfeed.tech/tags/budget.md>), [circuit](<https://devfeed.tech/tags/circuit.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [fpgas](<https://devfeed.tech/tags/fpgas.md>), [hep-ex](<https://devfeed.tech/tags/hep-ex.md>), [latency](<https://devfeed.tech/tags/latency.md>), [logic](<https://devfeed.tech/tags/logic.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [model](<https://devfeed.tech/tags/model.md>), [operations](<https://devfeed.tech/tags/operations.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [physics-ins-det](<https://devfeed.tech/tags/physics-ins-det.md>), [projection](<https://devfeed.tech/tags/projection.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [resource](<https://devfeed.tech/tags/resource.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [trigger](<https://devfeed.tech/tags/trigger.md>)

### AI overview

SCRAMJet reports 71.4 ± 0.6% accuracy on ten-class JetClass using 0.157M multiply-accumulates, 6.1 accuracy points above the strongest published resource-constrained model at one-eighth its operations. For the LHC FPGA trigger setting, it replaces multi-head attention with a low-rank projection, uses learned per-parameter fixed-point quantization, and compiles to Verilog with distributed arithmetic optimization. It meets timing on a Virtex UltraScale+ VU13P at 147 ns using 25.7% of logic. The authors also identify "virtual channels": attention channels that still carry signal during training but receive zero bit-width and disappear from the compiled circuit, without the loss reflecting their removal. At a matched budget, flooring attention datalane width prevents this and gains 3.1 accuracy points.

### Source excerpt

arXiv:2611.00086v1 Announce Type: cross Abstract: Level-1 Triggers at the LHC classify collisions within a few microseconds on FPGAs, where a network is a fully unrolled circuit whose cost is area rather than time. We present SCRAMJet, a transformer encoder that replaces multi-head attention with a low-rank projection, trains under per-parameter learned fixed-point quantization, and compiles to Verilog with distributed arithmetic optimization. On the ten-class JetClass dataset it reaches $71.4 \pm 0.6\%$ accuracy with 0.157M multiply-accumulates, 6.1 accuracy points above the strongest published resource-constrained model at an eighth of the operations, and meets timing on a Virtex UltraScale+ VU13P at 147 ns using 25.7% of the logic. We also show that resource-aware training silently removes attention channels: one whose learned bit-width reaches zero still carries signal during training but is absent from the compiled circuit, so the loss never sees it go. We call these virtual channels, and a floor on the attention datalane that prevents them is worth 3.1 accuracy points at matched budget. Code and trained checkpoints are available at https://github.com/Aaddy-1/hgq-jetformer.

## Local or Escalate? Post-Training Expert Routing for Real-Time Duplex Speech Models

DevFeed: [Local or Escalate? Post-Training Expert Routing for Real-Time Duplex Speech Models](<https://devfeed.tech/articles/local-or-escalate-post-training-expert-routing-for-real-time-duplex-speech-models-115196.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01639>)

Author: Ruoyu He, Jisen Li, Anthony Ji, Changyi Yang, Zetian Li, Runduo Ma, Zhen Zhang, Zhongzhu Zhou

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Real-time AI interactions](<https://devfeed.tech/topics/real-time-ai-interactions.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [eess-as](<https://devfeed.tech/tags/eess-as.md>), [local](<https://devfeed.tech/tags/local.md>), [models](<https://devfeed.tech/tags/models.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [routing](<https://devfeed.tech/tags/routing.md>), [speech](<https://devfeed.tech/tags/speech.md>), [speech-models](<https://devfeed.tech/tags/speech-models.md>)

### AI overview

The paper presents a post-training gate that decides whether a duplex speech model should answer immediately or hand a question to a stronger expert. It freezes MiniCPM-o 4.5 and fits a logistic failure readout to hidden states at the model's listen-to-speak transition; the gate adds no model generation, and its threshold can be retuned without changing duplex policy. A probe trained on text-input states also works on spoken questions, including unseen human recordings. On the internal benchmark, selective handoff raises answer accuracy by 27.1 points and cuts mean server time to first answer audio by 31.5% versus always using the expert. The abstract also reports better routing than matched random policies on external speech QA pools and a zero-shot GPT-5.4 mini router rating transcribed questions, plus transfer to a second duplex model family. These results are reported without further duplex training.

### Source excerpt

arXiv:2611.01639v1 Announce Type: new Abstract: A full-duplex model should answer easy questions immediately, but hard questions often need stronger reasoning, which may require waiting. The challenge is to choose between these paths at the moment the model begins to speak. Recent systems learn related mode decisions through special-token generation inside the dialogue model, coupling routing to turn-taking and temporally annotated full-duplex training. We study a post-training alternative. We freeze MiniCPM-o 4.5, using only its speech input and output, and fit a logistic failure readout on hidden states already available at its native listen-to-speak transition. The resulting gate requires no additional model generation and exposes thresholds that can be retuned without changing the duplex policy. A probe fitted on text-input states also reads the states of spoken questions, including unseen human recordings. On the internal benchmark, selective handoff improves answer accuracy by 27.1 points. Compared with always using the expert, it reduces mean server time to first answer audio by 31.5 percent. It also outperforms matched random routing across external speech question-answering pools, as well as a zero-shot GPT-5.4 mini router that rates the transcribed question. The training recipe also transfers to a second duplex model family. These results show that an existing duplex model can be equipped to answer easy turns immediately and wait for stronger reasoning or fresher knowledge on the turns that need it, without further duplex training.

[Next page](<https://devfeed.tech/topics/real-time-ai-interactions.md?cursor=WyIyMDI2LTExLTAzVDA1OjAwOjAwKzAwOjAwIiwgIjU5NmIyMTZmLTk1ZjYtNGZhZi1hZjc4LWZhZmM4M2E5NjFlMSJd>)