# Language models

A machine learning model that estimates the probability of a token or token sequence in a longer sequence.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Necessary, Not Sufficient, and No Longer Available: Grader Non-Determinism After the Withdrawal of Sampling Controls

DevFeed: [Necessary, Not Sufficient, and No Longer Available: Grader Non-Determinism After the Withdrawal of Sampling Controls](<https://devfeed.tech/articles/necessary-not-sufficient-and-no-longer-available-grader-non-determinism-after-the-withdrawal-of-sampling-controls-144911.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.26185>)

Author: Hiroki Tamba

Published: 2026-11-20T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Temperature settings in AI](<https://devfeed.tech/topics/temperature-settings-in-ai.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Error Propagation](<https://devfeed.tech/topics/error-propagation.md>)

Tags: [anthropic](<https://devfeed.tech/tags/anthropic.md>), [api](<https://devfeed.tech/tags/api.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [error](<https://devfeed.tech/tags/error.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [fail](<https://devfeed.tech/tags/fail.md>), [gate](<https://devfeed.tech/tags/gate.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gpt-5-6](<https://devfeed.tech/tags/gpt-5-6.md>), [gpt-5-6-sol](<https://devfeed.tech/tags/gpt-5-6-sol.md>), [gpt-6-1-sol](<https://devfeed.tech/tags/gpt-6-1-sol.md>), [harness](<https://devfeed.tech/tags/harness.md>), [index](<https://devfeed.tech/tags/index.md>), [information](<https://devfeed.tech/tags/information.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-as-judge](<https://devfeed.tech/tags/llm-as-judge.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [openai](<https://devfeed.tech/tags/openai.md>), [sufficient](<https://devfeed.tech/tags/sufficient.md>)

### AI overview

The study reports that temperature=0 is no longer a generally available control for LLM grader variability. In October-November 2026 measurements, Anthropic models rejected it; OpenAI chat completions models accepted only temperature 1, while Responses accepted temperature=0 for four of five current models with reasoning off, a configuration the author had not evaluated. The author proposes reporting per-item normalized entropy of grade distributions at the vendor-permitted setting. Across 22 condition runs and 4,760 calls, this measure ranked graders without estimating rates and identified items where the judge retained no information. Borderline items transferred only partly and unpredictably across model generations, so prior calibration sets could not simply be inherited. The author argues grader disagreement should be reported as an error bar on each verdict.

### Source excerpt

arXiv:2606.26185v3 Announce Type: replace Abstract: LLM-as-judge ("grader") components are now standard in evaluation harnesses, including safety evaluations where a pass/fail verdict may gate downstream deployment decisions. Earlier work, my own included, treated temperature=0 as the remedy for grader non-determinism and asked whether it was sufficient. It was necessary and not sufficient. It is now also unavailable. I report a two-vendor measurement taken in October and November 2026. Anthropic's current models reject the setting outright, and under the API version dated 2026-10-21 all fifteen models listed for my key reject it, the older generations included. On the chat completions endpoint OpenAI's current models accept the parameter and refuse every value but one, so a harness grading through that endpoint is obliged to sample its judge at temperature 1. On the Responses endpoint a request carrying temperature=0 is accepted by four of five current models once reasoning is switched off, a configuration I have not graded under, and by gpt-6.1-sol under no setting I tried. I propose the measurement that replaces the earlier protocol: the per-item normalized entropy of the grade distribution, reported at whatever setting the vendor permits. The direct alternative is closed too: gpt-4o returns token log-probabilities, while gpt-5.6-sol refuses the parameter and the Anthropic Messages API does not accept it at all. Across twenty-two condition runs and 4,760 calls the index orders graders without requiring a rate estimate, and identifies items at which a judge retains no information at all. I further show that a borderline item set carries across model generations only in part and not predictably, so a harness cannot inherit its calibration set. I argue that grader disagreement must be reported as an error bar on every verdict, because it is no longer something an evaluator can configure away.

## Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge

DevFeed: [Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge](<https://devfeed.tech/articles/safeguarding-llms-via-model-agnostic-latent-safety-signals-from-dark-knowledge-145395.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.07532>)

Author: Wonjun Lee, Kyungsik Yang, Gaeun Ji, Vaidehi Patil, Haon Park, Bumsub Ham, Mohit Bansal, Suhyun Kim

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [attack](<https://devfeed.tech/tags/attack.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [defense](<https://devfeed.tech/tags/defense.md>), [discrimination](<https://devfeed.tech/tags/discrimination.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [hidden-states](<https://devfeed.tech/tags/hidden-states.md>), [llms](<https://devfeed.tech/tags/llms.md>), [model-agnostic](<https://devfeed.tech/tags/model-agnostic.md>), [robust](<https://devfeed.tech/tags/robust.md>), [safety](<https://devfeed.tech/tags/safety.md>), [tokenizers](<https://devfeed.tech/tags/tokenizers.md>), [utility](<https://devfeed.tech/tags/utility.md>)

### AI overview

LADE detects potentially harmful queries using safety signals in an LLM's first-token probability distribution beyond its most likely token. It selects tokens whose probabilities differ sharply between harmful and benign queries, maps them across tokenizers, then applies k-nearest-neighbor classification. The authors report that these signals align across diverse LLMs, enabling model-agnostic use without relying on architecture-specific hidden states. Across the evaluated models and benchmarks, LADE reportedly lowers jailbreak attack success rates while maintaining a competitive safety-utility trade-off. The supplied text gives no specific metrics or benchmark names.

### Source excerpt

arXiv:2610.07532v2 Announce Type: replace-cross Abstract: LLMs have advanced rapidly, raising growing concerns about their safety. Recent work has proposed approaches to detect and defend against attacks including defenses at decoding stage that leverage models' hidden states. However, existing decoding-stage defenses suffer from two limitations. First, they introduce a trade-off between safety and over-refusal, where strengthening safety degrades the model's helpfulness on benign queries. Second, many of these methods rely on internal hidden states and are thus restricted to specific architectures, incurring substantial overhead and limited generalization across models. To address these limitations, we introduce LADE (Latent Safety Signals for Defense), which leverages latent safety signals extracted by contrasting harmful and benign queries from dark knowledge (i.e., information carried by the output probability distribution beyond its argmax) in the first-token output probability distribution. Our key insight is that, beyond surface-level refusal tokens, the dark knowledge in the first-token distribution contains latent safety signals, defined as tokens whose probabilities differ sharply between harmful and benign queries. We show that these signals consistently align across LLMs, forming a model-agnostic direction that emerges from safety alignment. LADE consists of three components: (1) Extracting Latent Safety Signals from Dark Knowledge, which selects top-k safety-discriminative tokens from the first-token probability distribution; (2) Tokenizer Mapping, which maps these tokens across different tokenizers to enable model-agnostic application; and (3) kNN-based Discrimination, which classifies queries via a k-Nearest Neighbors search over the mapped tokens. Across diverse LLMs and benchmarks, LADE is robust against a wide range of jailbreak attacks and lowers attack success rates while maintaining a competitive safety-utility trade-off.

## Supportive Token Revealing for Fast Diffusion Language Model Decoding

DevFeed: [Supportive Token Revealing for Fast Diffusion Language Model Decoding](<https://devfeed.tech/articles/supportive-token-revealing-for-fast-diffusion-language-model-decoding-144901.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.04236>)

Author: Giries Abu Ayoub, Mario Barbara, Llu\\'is Pastor-P\\'erez, Tanja Bien, Aneesh Barthakur, Alaa Maalouf, Loay Mualem

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Decoding](<https://devfeed.tech/topics/decoding.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [axon](<https://devfeed.tech/tags/axon.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code](<https://devfeed.tech/tags/code.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decode](<https://devfeed.tech/tags/decode.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [denoising](<https://devfeed.tech/tags/denoising.md>), [dependency](<https://devfeed.tech/tags/dependency.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [existing](<https://devfeed.tech/tags/existing.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fast](<https://devfeed.tech/tags/fast.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

AXON is a training-free add-on for parallel diffusion language model decoders that aims to make remaining uncertain tokens easier to denoise. It monitors those tokens and, when they appear to need more context, uses attention, uncertainty, and confidence signals to select confident masked tokens as anchors for later denoising. This shifts the reveal criterion from which token is safest to reveal to which reveal would best support other tokens, while leaving the base decoder in place. On reasoning and code-generation benchmarks across multiple diffusion models, the abstract reports improved quality-latency trade-offs, often with fewer function evaluations while maintaining or improving accuracy.

### Source excerpt

arXiv:2606.04236v2 Announce Type: replace-cross Abstract: Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which tokens are safe to reveal using confidence or dependency criteria. However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain tokens may depend on masked tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked tokens and intervenes only when their current state suggests that additional context is needed. It then shifts the criterion from which tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked tokens that uncertain positions attend to, using attention, uncertainty, and confidence signals. Experiments on reasoning and code-generation benchmarks across multiple diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.

## Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations

DevFeed: [Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations](<https://devfeed.tech/articles/persistent-sparse-autoencoders-learning-feature-specific-timescales-in-language-model-representations-144914.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.17117>)

Author: Haoyan Luo, Mateo Espinosa Zarlenga, Mateja Jamnik

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [LLM observability](<https://devfeed.tech/topics/llm-observability.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [autoencoders](<https://devfeed.tech/tags/autoencoders.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [extension](<https://devfeed.tech/tags/extension.md>), [fast](<https://devfeed.tech/tags/fast.md>), [feature](<https://devfeed.tech/tags/feature.md>), [features](<https://devfeed.tech/tags/features.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [hidden-states](<https://devfeed.tech/tags/hidden-states.md>), [information](<https://devfeed.tech/tags/information.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [monitor](<https://devfeed.tech/tags/monitor.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [prompt-injection](<https://devfeed.tech/tags/prompt-injection.md>), [quality](<https://devfeed.tech/tags/quality.md>), [reduce](<https://devfeed.tech/tags/reduce.md>)

### AI overview

Persistent Sparse Autoencoders (Persistent SAEs) learn a persistence coefficient for each sparse feature, giving features different timescales based on reconstruction. The authors report that standard SAE features can already persist: after activation, hidden states remain aligned with a feature direction, and past activations help reconstruct later states. Persistent SAEs retain competitive reconstruction quality while producing fast features that remain locally interpretable and slow features that accumulate higher-level context. In a prompt-injection monitoring case study, slow features retain injection signals over long contexts and causally affect monitor judgements. The abstract presents these results as opportunities for model interpretation and monitoring.

### Source excerpt

arXiv:2607.17117v3 Announce Type: replace Abstract: Sparse autoencoders (SAEs) decompose language model activations into sparse features, yet these models traditionally encode each token independently, failing to expose information that persists across a sequence. We first show that temporal persistence can naturally emerge in standard SAE features: after a feature activates, the hidden state remains aligned with its direction, and past activations reduce the reconstruction error of later hidden states. How long this lasts varies widely across features. We therefore introduce Persistent Sparse Autoencoders (Persistent SAEs), an extension of standard SAEs that learns a persistence coefficient for each feature, allowing feature-specific timescales to emerge from reconstruction. Our experiments show that Persistent SAEs retain competitive reconstruction quality while learning a spectrum of timescales: short-timescale (fast) features remain locally interpretable, whereas long-timescale (slow) features accumulate higher-level contextual information. Moreover, we show in a prompt-injection monitoring case study that slow features retain injection signals over long contexts and causally affect monitor judgements. These results suggest that Persistent SAEs offer new opportunities for interpreting and monitoring language models via persistent features.

## NVIDIA's Dory Shows Why Thinking Longer Doesn't Make AI Smarter

DevFeed: [NVIDIA's Dory Shows Why Thinking Longer Doesn't Make AI Smarter](<https://devfeed.tech/articles/nvidia-s-dory-shows-why-thinking-longer-doesn-t-make-ai-smarter-142901.md>)

Original publisher: [Read original article](<https://hackernoon.com/nvidias-dory-shows-why-thinking-longer-doesnt-make-ai-smarter?source=rss>)

Author: Michał Piszczek

Published: 2026-11-19T08:30:28Z

Content type: opinion

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-thinking](<https://devfeed.tech/tags/ai-thinking.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [code](<https://devfeed.tech/tags/code.md>), [compute](<https://devfeed.tech/tags/compute.md>), [confidence-interval](<https://devfeed.tech/tags/confidence-interval.md>), [cost](<https://devfeed.tech/tags/cost.md>), [costs](<https://devfeed.tech/tags/costs.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [effort](<https://devfeed.tech/tags/effort.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [failure](<https://devfeed.tech/tags/failure.md>), [fetch](<https://devfeed.tech/tags/fetch.md>), [future-of-ai](<https://devfeed.tech/tags/future-of-ai.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [llms](<https://devfeed.tech/tags/llms.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [paying-ai-to-think-harder](<https://devfeed.tech/tags/paying-ai-to-think-harder.md>), [thinking](<https://devfeed.tech/tags/thinking.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

NVIDIA's 4B Dory transformer makes extra reasoning a measurable choice: it repeats a 20-layer core while keeping the same weights, increasing computation per token without adding information. The excerpt says more passes can help assemble facts already in the input, but cannot supply missing facts; use an external lookup for those and code or a calculator for exact arithmetic or strings. A cited Fastino 27B result reports a small held-out gain (+0.56 points; 95% CI 0.27-0.89) alongside about 35% higher median latency (83 to 112 ms), and a third pass worsened one causal-reasoning benchmark. The suggested checks are whether the answer is present, where gains flatten, and the cost per verified answer including retries. The excerpt distinguishes transformer loops, generated chain of thought, tool-using agent loops, and across-version recursive self-improvement by what they consume or leave behind; only tool use brings in outside information.

### Source excerpt

Looped transformers turn AI thinking into a dial. The Alice test shows when turning it up helps and when you pay for effort that can't succeed.

## DV-Lens: Revealing the Functional Organization of Language Model Parameters

DevFeed: [DV-Lens: Revealing the Functional Organization of Language Model Parameters](<https://devfeed.tech/articles/dv-lens-revealing-the-functional-organization-of-language-model-parameters-143323.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.04489>)

Author: Chenhang Cui, Jian Yu, Jiani Luo, Shuyi Miao, Dingrui Liu, Rui Huang, Xiaohao Liu, Fei Shen, An Zhang, Tat-Seng Chua

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [attention](<https://devfeed.tech/tags/attention.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [capability](<https://devfeed.tech/tags/capability.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [effects](<https://devfeed.tech/tags/effects.md>), [framework](<https://devfeed.tech/tags/framework.md>), [interpretability](<https://devfeed.tech/tags/interpretability.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llms](<https://devfeed.tech/tags/llms.md>), [mappings](<https://devfeed.tech/tags/mappings.md>), [models](<https://devfeed.tech/tags/models.md>), [modules](<https://devfeed.tech/tags/modules.md>), [native](<https://devfeed.tech/tags/native.md>), [networks](<https://devfeed.tech/tags/networks.md>), [orientation](<https://devfeed.tech/tags/orientation.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [project](<https://devfeed.tech/tags/project.md>)

### AI overview

DV-Lens maps parameter directions in attention Q/K/V/O projections and feed-forward networks to downstream vocabulary responses, then groups parameters by those readouts. Its DV-Complexity measure uses normalized reconstruction residuals to quantify structural variation within groups. Randomized controls and finite-difference tests report 98.0% coordinate-orientation agreement for predicted local logit changes across 720 cases in nine models; ablation, steering, and swapping also shifted target-token probabilities in predicted directions under controlled conditions across 21 models. Across 48 models, a joint DV-Complexity score had a Spearman correlation of 0.904 with benchmark-based capability rankings. These findings support the readouts as intervention-linked interpretations and show an association with capability, not evidence that complexity causes capability.

### Source excerpt

arXiv:2610.04489v2 Announce Type: replace-cross Abstract: Understanding parameter functions helps elucidate the internal mechanisms of large language models (LLMs). However, how to connect parameters from different modules to verifiable output effects and further characterize the relationship between their functional organization and model capability remains to be explored. To this end, we introduce the downstream vocabulary lens (DV-Lens), a parameter-level interpretability framework that links native parameter directions to their downstream vocabulary responses. Specifically, we first estimate module-specific downstream Jacobians over a reference prompt set for attention query, key, value, and output (Q/K/V/O) projections and feed-forward networks (FFNs). Second, we use these mappings to project native parameter columns into the final vocabulary space, obtaining signed readouts that characterize their average local output responses. Third, we group parameter columns by their vocabulary readouts and introduce downstream vocabulary complexity (DV-Complexity), which quantifies within-group structural variation using normalized reconstruction residuals of the original weights. At the parameter level, randomized controls and finite-difference tests show that DV-Lens readouts capture non-random vocabulary structure and predict local logit changes with 98.0% coordinate-orientation agreement across 720 cases from nine models. These readouts further guide parameter ablation, steering, and swapping across 21 models, shifting target-token probabilities in the predicted directions under controlled conditions. At the model level, the joint-parameter score of DV-Complexity achieves a Spearman correlation of 0.904 with benchmark-based capability rankings across 48 language models. Together, these results provide intervention-based evidence for DV-Lens interpretations and reveal an association between DV-Complexity and model capability.

## NullTok: empirical-null token selection for training-free long video understanding

DevFeed: [NullTok: empirical-null token selection for training-free long video understanding](<https://devfeed.tech/articles/nulltok-empirical-null-token-selection-for-training-free-long-video-understanding-142716.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23881>)

Author: Zhang Ziyue, Juwei Jian, Baiyang Song, Yunshan Zhong, Jun Peng, Yiyi Zhou, Rongrong Ji

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [audio](<https://devfeed.tech/tags/audio.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [budget](<https://devfeed.tech/tags/budget.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [deduplication](<https://devfeed.tech/tags/deduplication.md>), [experimental](<https://devfeed.tech/tags/experimental.md>), [llava](<https://devfeed.tech/tags/llava.md>), [monte-carlo](<https://devfeed.tech/tags/monte-carlo.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [null](<https://devfeed.tech/tags/null.md>), [patch](<https://devfeed.tech/tags/patch.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [selection](<https://devfeed.tech/tags/selection.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [token](<https://devfeed.tech/tags/token.md>), [training](<https://devfeed.tech/tags/training.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

NullTok selects video tokens by calibrating each patch's query affinity against same-position scores from 64 group-matched irrelevant control queries. Under a conditional exchangeability assumption, these references yield discrete super-uniform right-tail Monte Carlo p-values. Calibrated screening and a high-recall semantic guard identify candidates, followed by visual deduplication and joint spatiotemporal budget allocation. The training-free method is evaluated with frozen LLaVA-Video-7B and Qwen3-VL-8B backbones. At a cap of 64 full-frame equivalents, without subtitles or audio, it scores 65.81% and 68.85% on Video-MME, respectively, outperforming the evaluated baselines at the same budget for both backbones.

### Source excerpt

arXiv:2611.23881v1 Announce Type: cross Abstract: Long video understanding remains a daunting challenge for existing multimodal large language models (MLLMs), mainly due to the large number of visual tokens accumulated in long videos. Currently, frame selection and token selection are two main paradigms for addressing this issue, both aiming to select relevant visual information to form a compact input for MLLMs. A natural approach combines these two paradigms by selecting frames before pruning tokens, but noisy frame-level relevance estimates can cause irreversible evidence loss. In this paper, we study this problem from the perspective of assessing the spatiotemporal distribution of query-specific evidence and propose a novel and training-free approach termed NULLTOK. For every patch across all frames, we construct an empirical-null distribution from 64 group-matched irrelevant control queries, capturing the scores expected in the absence of target-specific evidence. Target affinities are calibrated against these same-position references into right-tail Monte Carlo p-values, which are discrete super-uniform under a conditional exchangeability assumption. Calibrated screening and a high-recall semantic guard produce candidate tokens for deterministic visual deduplication and joint spatiotemporal budget allocation. To validate NullTok, we apply it to two representative MLLMs and conduct extensive experiments on a set of video benchmarks. The experimental results demonstrate its effectiveness under a bounded visual token budget. For example, it achieves 65.81% and 68.85% on Video-MME using frozen LLaVA-Video-7B and Qwen3-VL-8B backbones, respectively, at a cap of 64 full-frame equivalents (FFE), without subtitles or audio. These results outperform all evaluated baselines on Video-MME under the same budget for both backbones. Code is available at https://github.com/potato-kitty/NullTok.

## Learning through Discernment: when a small language model learns from user feedback on the job, and when it decays

DevFeed: [Learning through Discernment: when a small language model learns from user feedback on the job, and when it decays](<https://devfeed.tech/articles/learning-through-discernment-when-a-small-language-model-learns-from-user-feedback-on-the-job-and-when-it-decays-142695.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23716>)

Author: Sirisilp Kongsilp

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [lora](<https://devfeed.tech/topics/lora.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>), [On-device AI](<https://devfeed.tech/topics/on-device-ai.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [job](<https://devfeed.tech/tags/job.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [learning](<https://devfeed.tech/tags/learning.md>), [small-language-model](<https://devfeed.tech/tags/small-language-model.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

In experiments with Qwen3 0.6B and 1.7B plus LoRA adapters, nightly learning from user feedback depended on whether the judge filtered out failures and whether users could see outcomes. A feedback sense that treated polite give-ups as success collapsed in 3 of 5 lives. A version that first checked for a clear user verdict stayed near oracle performance in 4 of 5 lives for two weeks, then faded to self-judging levels by day 60 as clean lessons became scarce. On τ²-bench, it interpreted reactions correctly, but 80% of lessons were failed attempts users had thanked; across 97 lives, the share of failed lessons strongly predicted outcomes (Spearman ρ = −0.81). The abstract concludes that user-guided learning held up only when failures were excluded and users could observe outcomes, and warns that skewed lessons may narrow the model.

### Source excerpt

arXiv:2611.23716v1 Announce Type: cross Abstract: Small language models on devices could keep improving by learning, overnight, from their own daily work. Without a true outcome, however, something must decide which attempts are worth learning from, and both the model's own judgment and the user's reaction can be wrong. We study a small model (Qwen3 0.6B and 1.7B) with a LoRA adapter, fine-tuned each night on the attempts a judge accepts, over 127 lives of up to a year on WebShop and $\tau^2$-bench. We compare the true outcome, the user's stated verdict and the model's own judgment with a separate, frozen "feedback sense" that reads the user's reaction, built in three versions. Judging itself, the model improved and then decayed, over a year and whatever its adapter. A sense that took polite give-ups for success collapsed 3 of 5 lives; one built for discernment, which first asks whether the user gave a clear verdict, kept 4 of 5 near the oracle for two weeks, then faded on clean but scarce lessons, to the level of self-judging by day 60. When the user cannot see the outcome ($\tau^2$-bench), the sense read reactions correctly, yet 80% of its lessons were failures the customer had thanked. Across 97 lives, the share of failed attempts among the lessons predicted the outcome (Spearman $\rho$ = -0.81). In our experiments, learning from users held up only where the judge kept failures out and the user could see the outcome; lopsided lessons may also narrow the model.

## Reading Human Preference from Looped Transformer Internal States

DevFeed: [Reading Human Preference from Looped Transformer Internal States](<https://devfeed.tech/articles/reading-human-preference-from-looped-transformer-internal-states-142516.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.09870>)

Author: Jan Kirin

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [rlhf](<https://devfeed.tech/topics/rlhf.md>)

Tags: [anthropic](<https://devfeed.tech/tags/anthropic.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [controls](<https://devfeed.tech/tags/controls.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [edge](<https://devfeed.tech/tags/edge.md>), [hh](<https://devfeed.tech/tags/hh.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [layers](<https://devfeed.tech/tags/layers.md>), [linear](<https://devfeed.tech/tags/linear.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [network](<https://devfeed.tech/tags/network.md>), [pair](<https://devfeed.tech/tags/pair.md>), [property](<https://devfeed.tech/tags/property.md>), [rlhf](<https://devfeed.tech/tags/rlhf.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

On Ouro-2.6B-Thinking, a frozen model that runs its layers four times per token, probes of hidden states identify the human-preferred reply in Anthropic's HH-RLHF data only modestly better than simple rules: 61.5% for a linear probe, 64.9% for a nonlinear network scoring replies separately, and 64.7% for the version 1 evaluator on the test split. A network scoring both replies together has an estimated 0.7-point advantage over separate scoring (95% CI 0.0-1.4), leaving open whether preference is relational or a property of each reply. Results account for both pair orders, pair-level splits, and-- for probes--conversation endings where replies differ. Earlier version 1 headline scores (95.2% and 84.5%) fail the first two controls; version 2's relational-probe accuracy was four to five points too low because it omitted the third.

### Source excerpt

arXiv:2604.09870v3 Announce Type: replace-cross Abstract: Show a frozen language model two replies to the same request, one of which a person preferred. Can a small probe on the model's hidden states tell which? On Ouro-2.6B-Thinking, a model that runs its layers four times per token, and Anthropic's HH-RLHF data, the answer is yes, but only faintly. A linear probe picks the preferred reply 61.5% of the time, a small nonlinear network that scores each reply separately 64.9%, and the evaluator trained for version 1 reaches 64.7% on the test split: clearly better than simple rules such as picking the longer reply, but weak. Whether preference is stored as a comparison between the two replies or as a property of each is still open. A linear probe cannot tell, and a network that reads both replies together has an estimated edge of only 0.7 points (95% CI 0.0 to 1.4) over one that scores each reply separately. These numbers hold under three controls: scoring every pair in both orders, splitting the data by pair, and, for the probes, keeping the end of each conversation, where the two replies differ. Version 1's 95.2% and 84.5% were obtained without the first two and do not hold; version 2's corrected relational-probe accuracy, obtained without the third, was four to five points too low. This version replaces both.

## SANTA++: Sampling Attention through Representative Keys

DevFeed: [SANTA++: Sampling Attention through Representative Keys](<https://devfeed.tech/articles/santa-sampling-attention-through-representative-keys-143316.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.35629>)

Author: Kyle Lee, Christian Z. Pratt, Ruoyu Fang, Heekyung Lee, Avinash Lohitsa, Ryan Modafe, Kerem Y. Camsari

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [infini-attention](<https://devfeed.tech/topics/infini-attention.md>), [kernels](<https://devfeed.tech/topics/kernels.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [budget](<https://devfeed.tech/tags/budget.md>), [cache](<https://devfeed.tech/tags/cache.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [flash](<https://devfeed.tech/tags/flash.md>), [fp16](<https://devfeed.tech/tags/fp16.md>), [full](<https://devfeed.tech/tags/full.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [helmet](<https://devfeed.tech/tags/helmet.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [inclusion](<https://devfeed.tech/tags/inclusion.md>), [int4](<https://devfeed.tech/tags/int4.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [keys](<https://devfeed.tech/tags/keys.md>), [kv-cache](<https://devfeed.tech/tags/kv-cache.md>), [lets](<https://devfeed.tech/tags/lets.md>), [logical](<https://devfeed.tech/tags/logical.md>), [memory](<https://devfeed.tech/tags/memory.md>), [sampling](<https://devfeed.tech/tags/sampling.md>)

### AI overview

SANTA++ reduces KV-cache reads by grouping cached keys into teams, scoring one representative key per team, and sampling teams for exact within-team attention. It reweights sampled contributions by inverse inclusion probability to estimate full-cache attention; the sampling budget trades memory reads for accuracy. In the reported Qwen2.5-7B-Instruct tests at 32K context, contiguous S=1024 retained 95-98% of dense aggregate scores on LongBench v2, HELMET RAG, and RULER with 28-31% logical KV access. On RULER it beat Quest across the tested accuracy-access sweep, using 48.7% fewer reads at about 79% accuracy, and beat MagicPIG at low access. The reported Triton kernel was 1.38x faster than PyTorch Flash SDPA's attention operator in FP16; batch-eight decoding was 1.23x faster on an RTX 5090 Laptop GPU with matched INT4 weights and FP16 KV cache.

### Source excerpt

arXiv:2609.35629v2 Announce Type: replace Abstract: Attention often concentrates on a small subset of tokens in the context, but which subset matters changes from one query to the next. To exploit this changing structure, we introduce SANTA++, a training-free stochastic attention method that uses representative keys for memory-efficient selection without scanning the entire key-value (KV) cache. Cached keys are organized into teams, and the query scores one representative from each team to decide which teams to sample. We compute exact attention scores within the sampled teams and reweight each team's contribution by the inverse of its inclusion probability. This importance sampling correction estimates attention over the full cache, with a sampling budget that lets us trade memory reads for accuracy. With Qwen2.5-7B-Instruct at 32K context, contiguous SANTA++ at S=1024 retains 95-98% of dense aggregate scores on LongBench v2, HELMET RAG, and RULER using 28-31% logical KV access. On RULER, SANTA++ outperforms Quest across the tested accuracy-access sweep, using 48.7% fewer KV reads at approximately 79% accuracy. It also outperforms MagicPIG at low access. At S=1024, our Triton implementation gives a 1.38x attention-operator speedup over the Flash backend of PyTorch scaled dot-product attention (Flash SDPA) in FP16. At batch eight, full-model decoding is 1.23x faster than dense Flash SDPA with matched INT4 model weights and FP16 KV-cache precision on an RTX 5090 Laptop GPU. Our kernels are available at https://github.com/OPUSLab/santapp-kernel-demo.

## Weight Space Programming for Vision Language Model Repair

DevFeed: [Weight Space Programming for Vision Language Model Repair](<https://devfeed.tech/articles/weight-space-programming-for-vision-language-model-repair-142681.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23615>)

Author: Avinash Goutham Aluguvelly

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [32-bit](<https://devfeed.tech/tags/32-bit.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [bits](<https://devfeed.tech/tags/bits.md>), [coding-agent](<https://devfeed.tech/tags/coding-agent.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [encoder](<https://devfeed.tech/tags/encoder.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [noise](<https://devfeed.tech/tags/noise.md>), [precision](<https://devfeed.tech/tags/precision.md>), [programming](<https://devfeed.tech/tags/programming.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [repair](<https://devfeed.tech/tags/repair.md>), [smolvlm](<https://devfeed.tech/tags/smolvlm.md>), [token-embeddings](<https://devfeed.tech/tags/token-embeddings.md>), [vision-language-model](<https://devfeed.tech/tags/vision-language-model.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>), [weight-space-programming](<https://devfeed.tech/tags/weight-space-programming.md>)

### AI overview

In tests on SmolVLM-256M and LFM2-VL-450M, a coding agent repaired rescaled projector rows and vision-encoder noise by editing weights directly, often restoring exact original values and near-original accuracy. The repairs depended on low-bit traces left when faults were computed in 32-bit arithmetic against 16-bit weights: rounding those bits away preserved test answers but stopped unaided exact recovery. A prompt naming statistical clues such as row norms then guided repairs of rescaled rows, while swapped rows or columns were not found unaided within session budgets. The authors recommend reporting weight number formats and including a rounded control in repair studies.

### Source excerpt

arXiv:2611.23615v1 Announce Type: cross Abstract: In weight-space programming, a language model edits a network's weights directly, without gradients. We ask what a successful repair of this kind relies on. We plant faults in the weights of two vision-language models, SmolVLM-256M and LFM2-VL-450M, and check each edit made by a coding agent against the original weights. Without being told where to look, the agent finds rescaled rows in the projector and noise in the vision encoder of both models, infers the unknown scale factors, restores most or all of the rescaled weights to their exact original values, and returns accuracy to near its original level; programs that it writes from such weights repair unseen checkpoints of the same model without feedback. These repairs depend on a trace of how the faults were computed. The original weights have 16-bit precision, and faults computed in 32-bit arithmetic leave low bits that only the damaged weights carry. Rounding those bits away leaves each model's answers to our test questions unchanged, and the agent, unaided, then restores no weight exactly: on SmolVLM it no longer finds the damage, and in a single session on LFM2-VL its only gain comes from rescaling answer-token embeddings, which changes how often the model answers yes. Marking the damaged weights again in their low bits brings the edits back to them. Rounded weights still hold statistical evidence of the rescaling, in the distribution of stored values and in row norms, and one sentence in the prompt that names such a statistic directs the agent's edits to the damaged rows on both models. Swapped rows or columns, which leave the set of stored values unchanged, are not found unaided on either model within our session budgets. We recommend that studies of weight repair report the number format of the weights and include a rounded control.

## Qwen4-Max vs. Claude Opus 5.5: The cheaper model costs you somewhere else.

DevFeed: [Qwen4-Max vs. Claude Opus 5.5: The cheaper model costs you somewhere else.](<https://devfeed.tech/articles/qwen4-max-vs-claude-opus-5-5-the-cheaper-model-costs-you-somewhere-else-141057.md>)

Original publisher: [Read original article](<https://thenewstack.io/qwen4-max-vs-claude-opus-5-5/>)

Author: Jessica Wachtel

Published: 2026-11-18T19:10:39Z

Content type: comparison

Language: en

Sources: [The New Stack](<https://devfeed.tech/sources/the-new-stack.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Multi-step AI processes](<https://devfeed.tech/topics/multi-step-ai-processes.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>)

Tags: [accounting](<https://devfeed.tech/tags/accounting.md>), [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [alibaba](<https://devfeed.tech/tags/alibaba.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [billing](<https://devfeed.tech/tags/billing.md>), [bug-fix](<https://devfeed.tech/tags/bug-fix.md>), [claude-opus-5-5](<https://devfeed.tech/tags/claude-opus-5-5.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [costs](<https://devfeed.tech/tags/costs.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [qwen4-max](<https://devfeed.tech/tags/qwen4-max.md>), [software-testing](<https://devfeed.tech/tags/software-testing.md>)

### AI overview

In three single-run tasks, Qwen4-Max cost about a third as much as Claude Opus 5.5 overall, but took about 3.14 times longer and scored 170/183 versus Opus's 179/183. Results varied by task: Qwen scored 71/76 on a multi-app workflow, missing points after misreading an ownership rule; it edged Opus 59/60 to 58/60 on incident investigation at about 31% of the reported cost, while taking longer; and it scored 40/47 on a coding task versus Opus's 45/47, missing a payment-rounding bug and leaving a scratch file. Opus fixed all planted bugs but missed a flawed test the task required reporting. The author sees Qwen as a lower-cost option for investigation when extra time is acceptable, and favors Opus for coding or consequential work. Each model ran once, so small score differences may shift; the tests do not establish general performance rankings. The excerpt also notes different APIs, cost-estimation methods, and benchmark setups.

### Source excerpt

Qwen4-Max, made available in preview to Token Plan subscribers on November 11 and generally released on November 16, is quite The post Qwen4-Max vs. Claude Opus 5.5: The cheaper model costs you somewhere else. appeared first on The New Stack.

## A language model for DOS that fits on a floppy disk

DevFeed: [A language model for DOS that fits on a floppy disk](<https://devfeed.tech/articles/a-language-model-for-dos-that-fits-on-a-floppy-disk-140901.md>)

Original publisher: [Read original article](<https://blog.adafruit.com/2026/11/18/a-language-model-for-dos-that-fits-on-a-floppy-disk/>)

Author: Anne Barela

Published: 2026-11-18T17:18:37Z

Content type: news

Language: en

Sources: [Adafruit Industries](<https://devfeed.tech/sources/adafruit-industries-makers-hackers-artists-designers-and-engineers.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [Tensorflow](<https://devfeed.tech/topics/tensorflow.md>)

Tags: [486dx](<https://devfeed.tech/tags/486dx.md>), [c](<https://devfeed.tech/tags/c.md>), [code](<https://devfeed.tech/tags/code.md>), [dos](<https://devfeed.tech/tags/dos.md>), [fable-5](<https://devfeed.tech/tags/fable-5.md>), [fable-5-2](<https://devfeed.tech/tags/fable-5-2.md>), [floppy](<https://devfeed.tech/tags/floppy.md>), [floppy-disk](<https://devfeed.tech/tags/floppy-disk.md>), [github](<https://devfeed.tech/tags/github.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [model](<https://devfeed.tech/tags/model.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [opus](<https://devfeed.tech/tags/opus.md>), [opus-5](<https://devfeed.tech/tags/opus-5.md>), [opus-5-5](<https://devfeed.tech/tags/opus-5-5.md>), [pentium](<https://devfeed.tech/tags/pentium.md>), [programming](<https://devfeed.tech/tags/programming.md>), [published](<https://devfeed.tech/tags/published.md>), [question](<https://devfeed.tech/tags/question.md>), [repository](<https://devfeed.tech/tags/repository.md>), [small](<https://devfeed.tech/tags/small.md>), [tiny](<https://devfeed.tech/tags/tiny.md>), [turbo-c](<https://devfeed.tech/tags/turbo-c.md>), [vintage-computing](<https://devfeed.tech/tags/vintage-computing.md>)

### AI overview

Tiny-2.1 is a question-answering transformer for plain DOS on a Pentium or 486DX, written in Turbo C 2.0 and small enough, with its files, to fit on a floppy disk. To run it, copy the DOS folder files to the machine and keep TINY.EXE and TINY.BIN together; the folder's README lists keys and options. Rebuilding uses one Turbo C command line. The training kit includes facts, gen2.py for producing question-and-answer pairs in varied wording, and train2.py, which uses Python 3, NumPy and PyTorch; a normal CPU is sufficient. The training notes describe how to add facts, and exporting unchanged produces the released TINY.BIN. The code is MIT licensed, while three files derived from public datasets retain ODbL and CC BY-SA licenses.

### Source excerpt

WollyMonjac has published Tiny-2.1 on GitHub, a small question-answering transformer written in Turbo C 2.0 that runs on a Pentium or 486DX under plain DOS. The whole thing fits on a floppy disk. The repository says the code was written by Fable 5.2 and the model trained by Opus 5.5. Running it means copying the [...]

## Large-scale Neuron Community Discovery Reveals Neuron-Sparse Behaviors in Language Models

DevFeed: [Large-scale Neuron Community Discovery Reveals Neuron-Sparse Behaviors in Language Models](<https://devfeed.tech/articles/large-scale-neuron-community-discovery-reveals-neuron-sparse-behaviors-in-language-models-141844.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23112>)

Author: Igor Shilov, Thomas Fel, Tal Haklay, Lucius Bushnaq, Owen Lewis

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [llama](<https://devfeed.tech/topics/llama.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [addition](<https://devfeed.tech/tags/addition.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [based](<https://devfeed.tech/tags/based.md>), [basis](<https://devfeed.tech/tags/basis.md>), [behaviors](<https://devfeed.tech/tags/behaviors.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [coherent](<https://devfeed.tech/tags/coherent.md>), [community](<https://devfeed.tech/tags/community.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large](<https://devfeed.tech/tags/large.md>), [models](<https://devfeed.tech/tags/models.md>), [neuron](<https://devfeed.tech/tags/neuron.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

Neuron community discovery identifies reusable groups of existing neurons that contribute to common computations across prompts. The unsupervised method combines gradient-based attribution, a graph of neurons repeatedly attributed together, and community detection. Applied to Llama-3.1-8B on ten million prompts, it found thousands of candidate communities, then tested them on held-out prompts for community necessity, sufficiency within their layer subset, and layer necessity. Examples include grammatical agreement, token-probability calibration, fixed phrases, in-context copying, addition, and multiplication. With communities limited to 128 neurons across all layers, at least one layer's contribution was explained for 10.3% of held-out prompts; coverage was 8.7% when requiring four or more layers and 3.3% for twelve or more. These findings indicate neuron-sparse explanations for some behaviors, with coverage decreasing as the number of required layers increases.

### Source excerpt

arXiv:2611.23112v1 Announce Type: new Abstract: What reusable units of computation can we discover within a language model's neurons? Prior work has shown that individual forward passes can often be sparsified post-hoc to small groups of neurons, but it has been unclear whether these groups form coherent computational entities reused across inputs. We introduce neuron community discovery, an unsupervised method for finding neuron groups that recur across prompts and are causally necessary and sufficient, within the layers they span, for common computations. The method combines gradient-based attribution, a graph connecting neurons that are repeatedly attributed together, and community detection. Applied to Llama-3.1-8B using ten million prompts, it discovers thousands of candidate communities. We evaluate them on held-out prompts through three causal tests: community necessity, sufficiency within a layer subset, and layer necessity. These tests identify groups whose removal disrupts predictions and whose retention preserves predictions despite ablating the remaining MLP neurons in the tested layers. Discovered communities support functions like grammatical agreement, token-probability calibration, fixed phrases, in-context copying, addition, and multiplication. With communities capped at 128 neurons across all layers, 10.3% of held-out prompts have at least one layer's contribution explained on a neuron-sparse basis. Coverage remains 8.7% when requiring at least four layers and 3.3% when requiring at least twelve. These results show that small groups of existing neurons can provide reusable computational components across prompts, and that such groups can be discovered at scale.

## Latent Fact Composition under Next-Token Training

DevFeed: [Latent Fact Composition under Next-Token Training](<https://devfeed.tech/articles/latent-fact-composition-under-next-token-training-141697.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21921>)

Author: Tianyi Zhang, Feiying Chen

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [intermediate](<https://devfeed.tech/tags/intermediate.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [model](<https://devfeed.tech/tags/model.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [token](<https://devfeed.tech/tags/token.md>), [training](<https://devfeed.tech/tags/training.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

In a synthetic two-hop reasoning task, standard and looped transformers and DiscoLoop answered almost none of the questions where the intermediate entity was absent from two-hop training examples. The new Baton looped-transformer family passes that entity to its second loop as a token embedding; two of its three variants performed almost perfectly when trained to predict only the answer. With next-token training, Baton's accuracy on these questions rose and then fell. Removing two training-set-dependent terms from the loss gradient raised peak accuracy but did not prevent the decline, which occurred during the second hop even though the model still recalled both facts. The experiments support three proposed design conditions, tested by changing one at a time; the supplied abstract does not specify those conditions.

### Source excerpt

arXiv:2611.21921v1 Announce Type: cross Abstract: Controlled studies of latent two-hop reasoning train transformers to predict only the answer, whereas language models are trained to predict every token. We ask whether a model trained on every token learns to answer two-hop questions whose intermediate entity appears in no two-hop training example. We build a synthetic corpus with such questions. On this corpus every term of the loss can be computed exactly. A looped transformer, a standard transformer and DiscoLoop answer almost none of these questions. We therefore introduce the Baton family of looped transformers. They pass the intermediate entity to the second loop as a token embedding. When trained to predict only the answer, two of its three variants answer these questions almost perfectly. We then relate the success and failure of each model to three design conditions. Experiments that change one condition at a time agree with this hypothesis. When Baton is trained on every token, however, its accuracy on these questions rises and then declines. We further split the gradient of the next-token loss into three terms. Removing the two terms that depend on the particular training set raises the peak accuracy but does not remove the decline. The decline arises in the second hop while the model still recalls both facts.

## Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)

DevFeed: [Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)](<https://devfeed.tech/articles/long-context-generation-is-a-sampling-problem-when-you-turn-up-the-heat-139971.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22148>)

Author: Allen G. Roush, Minh Nhat Nguyen, Ravid Shwartz-Ziv, Judah Goldfeder, Sanjay Basu

Published: 2026-11-18T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [Temperature settings in AI](<https://devfeed.tech/topics/temperature-settings-in-ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [chain-of-thought](<https://devfeed.tech/tags/chain-of-thought.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [data](<https://devfeed.tech/tags/data.md>), [generation](<https://devfeed.tech/tags/generation.md>), [github](<https://devfeed.tech/tags/github.md>), [github-issues](<https://devfeed.tech/tags/github-issues.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llama-3-1](<https://devfeed.tech/tags/llama-3-1.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [reading](<https://devfeed.tech/tags/reading.md>), [sampling](<https://devfeed.tech/tags/sampling.md>)

### AI overview

The study reports that sampler behavior as temperature rises can strongly affect long-generation quality in open-weight models. Across 29 models and ten benchmarks plus retrieval up to one million tokens, top-nsigma retained a median 100% of its temperature-1 score at temperature 2, while top-p retained 26%. On three Qwen3.5 models, coding agents using top-p, top-k, or Top-H resolved none of 100 GitHub issues at temperature 2; top-nsigma resolved 46 to 62. The authors associate quality loss with growth in the candidate-token pool: for one Llama-3.1-8B example, top-p expanded from 4 tokens at temperature 1 to 51,845 at 2, while top-nsigma stayed at 3. They report that fixing the pool at temperature 1 also prevented Top-H, min-p, and p-less from collapsing. Most samplers performed similarly at temperature 1.

### Source excerpt

arXiv:2611.22148v1 Announce Type: new Abstract: Open-weight language models are widely observed to get worse the longer they generate. Often, attempts to fix this involve more training. At high temperature, The sampler, i.e. the rule that picks the next token out of the model's probabilities, decides whether long and agentic generation survives and/or thrives. We test 29 open-weight models from 1B to 122B parameters on ten established benchmarks plus needle retrieval out to one million tokens, covering SWE-bench Verified coding agents, long chain-of-thought reasoning, long-form writing, long structured output and long-context reading. Samplers that keep their pool of candidate tokens small and steady as temperature rises don't lose accuracy. Samplers that let in low-probability tokens fall apart. At temperature 2, coding agents running top-p, top-k or Top-H resolve 0 of 100 GitHub issues, while the same agents running top-nsigma resolve 46 to 62 and win every issue the two samplers disagree on, on three Qwen3.5 models. Over all benchmarks top-nsigma keeps a median 100% of its temperature-1 score where top-p keeps 26%. Counting how many tokens each sampler lets through explains why. In one Llama-3.1-8B story, going from temperature 1 to 2 grows top-p's pool from 4 tokens to 51,845 while top-nsigma stays at 3. Across 133 combinations of sampler setting and temperature, the typical pool size predicts judged quality (Spearman correlation of -0.77), and when we pick the pool at temperature 1 and only apply temperature afterwards, Top-H, min-p and p-less stop collapsing. At the default temperature of 1 most samplers perform about the same, We release all code and data to the public.

## Structure--Residual Inductive Biases for Language Model Pretraining

DevFeed: [Structure--Residual Inductive Biases for Language Model Pretraining](<https://devfeed.tech/articles/structure-residual-inductive-biases-for-language-model-pretraining-140130.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23197>)

Author: Hai Huang, Bartosz \\.Zuk, Rassul Magauin, Jiaxin Huang, Xue Liu

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [bbc](<https://devfeed.tech/tags/bbc.md>), [bias](<https://devfeed.tech/tags/bias.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [cc](<https://devfeed.tech/tags/cc.md>), [coding](<https://devfeed.tech/tags/coding.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [diagnostics](<https://devfeed.tech/tags/diagnostics.md>), [energy](<https://devfeed.tech/tags/energy.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [inductive](<https://devfeed.tech/tags/inductive.md>), [information](<https://devfeed.tech/tags/information.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [model](<https://devfeed.tech/tags/model.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [structure](<https://devfeed.tech/tags/structure.md>)

### AI overview

The paper tests whether language-model representations benefit from an explicit structure-residual bias alongside next-token prediction. Boundary Bottleneck Concept (BBC) routes segment information through boundary carriers while retaining local autoregressive computation; Continuous Concept (CC) regularizes representation geometry into low-frequency structure and high-frequency residual. BBC works best in early layers, while CC is most effective later. On GPT-2 Medium, combining them improves NLL by 1.44% at 300M training tokens, with gains reported through runs up to 20B tokens. Diagnostics associate BBC with greater predictive importance of boundary states and CC with reduced spectral residual energy.

### Source excerpt

arXiv:2611.23197v1 Announce Type: cross Abstract: Next-token prediction is relied upon to learn both predictive behavior and internal representations. We ask whether representation structure itself should be an explicit inductive bias. Motivated by the two-part coding view of Kolmogorov complexity, we study a structure--residual principle in which a representation separates structure from realization detail. We instantiate this principle in two complementary ways. Boundary Bottleneck Concept (BBC) routes information from token segments through boundary carriers while preserving local autoregressive computation. Continuous Concept (CC) regularizes representation geometry and induces a spectral decomposition into low-frequency structure and high-frequency residual. We derive coding bounds for both views and show that multiplicity-corrected boundary attention becomes exact as the within-segment path energy vanishes. Empirically, BBC and CC exhibit strikingly different depth preferences: BBC works best in early layers, whereas CC is most effective in later layers. At 300M training tokens, combining BBC and CC improves NLL by 1.44\% on GPT-2 Medium. These gains persist with training runs up to 20B tokens. Representation diagnostics show that BBC increases the relative predictive importance of boundary states, while CC reduces spectral residual energy. Together, these results suggest that structure--residual biases can improve language-model pretraining beyond next-token prediction. Code: https://github.com/haihuang-ml/structure-residual

## ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models

DevFeed: [ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models](<https://devfeed.tech/articles/remarl-learning-to-unmask-and-remask-in-frozen-diffusion-language-models-141788.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22576>)

Author: Egor Shibaev, Danila Biktimirov, Ivan Klimov, Artem Makoian, Viacheslav Meshchaninov, Dmitry Vetrov

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [grpo](<https://devfeed.tech/topics/grpo.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [code](<https://devfeed.tech/tags/code.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [group-relative-policy-optimization](<https://devfeed.tech/tags/group-relative-policy-optimization.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [learning](<https://devfeed.tech/tags/learning.md>), [math](<https://devfeed.tech/tags/math.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>)

### AI overview

ReMaRL trains a roughly 0.3M-parameter policy with GRPO and final-answer rewards to choose which tokens a frozen diffusion language model should unmask, retain, or remask. Unlike reward-trained unmasking policies that cannot revise earlier predictions, ReMaRL can remask tokens so they can be corrected using newly generated context. With LLaDA-8B-Instruct, it achieved the highest scores among the compared policies in the tested budgets: 4-32 model calls for math and 8-16 for code.

### Source excerpt

arXiv:2611.22576v1 Announce Type: new Abstract: Masked diffusion language models generate tokens in parallel, and the order in which tokens are unmasked affects final-answer accuracy. Several methods learn unmasking policies from task rewards, but these policies cannot remask earlier predictions. Remasking lets the model correct those predictions using newly generated context. Heuristic rules guide both unmasking and remasking, while supervised remaskers learn to detect token errors. Neither approach directly optimizes these decisions for final-answer correctness. To address this gap, we introduce ReMaRL (Remasking with Reinforcement Learning), a lightweight policy that learns which tokens to unmask, retain, or remask. We train this roughly 0.3M-parameter policy with Group Relative Policy Optimization (GRPO) from final-answer rewards while keeping the diffusion model frozen. With LLaDA-8B-Instruct, ReMaRL achieves the highest scores among the compared unmasking and remasking policies within the tested inference budgets of 4-32 model calls on math and 8-16 calls on code.

## What MoE Routing Controls: Causal Interventions on Expert Trajectories

DevFeed: [What MoE Routing Controls: Causal Interventions on Expert Trajectories](<https://devfeed.tech/articles/what-moe-routing-controls-causal-interventions-on-expert-trajectories-141735.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22120>)

Author: Mukund Athmaram, Manoj Rao, Liang Ma

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [causality](<https://devfeed.tech/topics/causality.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [bias](<https://devfeed.tech/tags/bias.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experts](<https://devfeed.tech/tags/experts.md>), [form](<https://devfeed.tech/tags/form.md>), [format](<https://devfeed.tech/tags/format.md>), [formats](<https://devfeed.tech/tags/formats.md>), [generation](<https://devfeed.tech/tags/generation.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layers](<https://devfeed.tech/tags/layers.md>), [mixture-of-experts](<https://devfeed.tech/tags/mixture-of-experts.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [moe](<https://devfeed.tech/tags/moe.md>), [money](<https://devfeed.tech/tags/money.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [religion](<https://devfeed.tech/tags/religion.md>), [router](<https://devfeed.tech/tags/router.md>), [routing](<https://devfeed.tech/tags/routing.md>)

### AI overview

The study tests how steering mixture-of-experts routing affects generation in Qwen3-30B-A3B. Researchers build expert paths from tokens associated with a format or concept, then bias the corresponding router logits during generation. Steering format paths changes the produced text; steering concept paths shifts next-token probabilities toward the concept, including for paths built in another language, but has little effect on generated text. Partial-generation interventions suggest that a format can reinforce itself once begun. Religion and money show the strongest concept steering, with effects concentrated in middle layers. The excerpt says the authors examine embedding distinctiveness and word frequency to explain variation across concepts, but does not report the results of that analysis.

### Source excerpt

arXiv:2611.22120v1 Announce Type: cross Abstract: In mixture-of-experts (MoE) language models, a token is routed to a sequence of experts across layers. We investigate how forcing tokens onto chosen expert paths influences model behavior. We build a target path from the experts that a concept's or role's tokens most often use, and we add a fixed bias to each target expert's router logit during generation to steer the token toward this expert path. We study two kinds of targets. Format targets concern the form of the output, and concept targets concern its topic. In Qwen3-30B-A3B, we show that steering a format path changes the text the model writes. We also show that steering a concept path shifts the model's next-token probabilities toward the concept, even when the path is built in another language, yet this shift has little effect on the generated text. We investigate why formats reach the text but concepts do not by steering only part of the generation, and find evidence that a format, once started, reinforces itself. We show that the concepts religion and money steer most strongly, and that the effect is concentrated in the middle layers. We attempt to explain why the strength of the effect varies by concept through an analysis of embedding distinctiveness and word frequency.

## Token sticker shock pushes enterprises from frontier labs toward open-weight AI factories

DevFeed: [Token sticker shock pushes enterprises from frontier labs toward open-weight AI factories](<https://devfeed.tech/articles/token-sticker-shock-pushes-enterprises-from-frontier-labs-toward-open-weight-ai-factories-138616.md>)

Original publisher: [Read original article](<https://siliconangle.com/2026/11/17/gpus-alone-won-t-carry-enterprise-ai-pilots-production-dellatsc26/>)

Author: Kelly Knight

Published: 2026-11-17T23:27:01Z

Content type: news

Language: en

Sources: [SiliconANGLE](<https://devfeed.tech/sources/siliconangle.md>)

Topics: [digital sovereignty](<https://devfeed.tech/topics/digital-sovereignty.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>)

Tags: [accelerated-computing](<https://devfeed.tech/tags/accelerated-computing.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-data-centers](<https://devfeed.tech/tags/ai-data-centers.md>), [ai-factories](<https://devfeed.tech/tags/ai-factories.md>), [ai-factory](<https://devfeed.tech/tags/ai-factory.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-supercomputers](<https://devfeed.tech/tags/ai-supercomputers.md>), [anthropic](<https://devfeed.tech/tags/anthropic.md>), [anthropic-pbc](<https://devfeed.tech/tags/anthropic-pbc.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [brad-maltz](<https://devfeed.tech/tags/brad-maltz.md>), [compute-express-link](<https://devfeed.tech/tags/compute-express-link.md>), [cube-event-coverage](<https://devfeed.tech/tags/cube-event-coverage.md>), [cxl](<https://devfeed.tech/tags/cxl.md>), [data](<https://devfeed.tech/tags/data.md>), [data-sovereignty](<https://devfeed.tech/tags/data-sovereignty.md>), [dell](<https://devfeed.tech/tags/dell.md>), [dell-ai-factory](<https://devfeed.tech/tags/dell-ai-factory.md>), [dell-technologies-inc](<https://devfeed.tech/tags/dell-technologies-inc.md>), [dellatsc26](<https://devfeed.tech/tags/dellatsc26.md>), [digital-twins](<https://devfeed.tech/tags/digital-twins.md>), [disaggregated-memory](<https://devfeed.tech/tags/disaggregated-memory.md>), [energy-efficiency](<https://devfeed.tech/tags/energy-efficiency.md>), [enterprises](<https://devfeed.tech/tags/enterprises.md>), [exascale-computing](<https://devfeed.tech/tags/exascale-computing.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [gemma-allen](<https://devfeed.tech/tags/gemma-allen.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [heterogeneous-computing](<https://devfeed.tech/tags/heterogeneous-computing.md>), [high-performance-computing](<https://devfeed.tech/tags/high-performance-computing.md>), [high-performance-storage](<https://devfeed.tech/tags/high-performance-storage.md>), [high-speed-networking](<https://devfeed.tech/tags/high-speed-networking.md>), [hpc](<https://devfeed.tech/tags/hpc.md>), [hpc-infrastructure](<https://devfeed.tech/tags/hpc-infrastructure.md>), [hpc-unites](<https://devfeed.tech/tags/hpc-unites.md>), [infra](<https://devfeed.tech/tags/infra.md>), [john-furrier](<https://devfeed.tech/tags/john-furrier.md>), [labs](<https://devfeed.tech/tags/labs.md>), [memory-pooling](<https://devfeed.tech/tags/memory-pooling.md>), [model](<https://devfeed.tech/tags/model.md>), [national-laboratories](<https://devfeed.tech/tags/national-laboratories.md>), [nemotron](<https://devfeed.tech/tags/nemotron.md>), [news](<https://devfeed.tech/tags/news.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [nyse-dell](<https://devfeed.tech/tags/nyse-dell.md>), [open-weight-ai](<https://devfeed.tech/tags/open-weight-ai.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [openai-group](<https://devfeed.tech/tags/openai-group.md>), [parallel-file-systems](<https://devfeed.tech/tags/parallel-file-systems.md>), [production](<https://devfeed.tech/tags/production.md>), [sc26](<https://devfeed.tech/tags/sc26.md>), [sc26-2026](<https://devfeed.tech/tags/sc26-2026.md>), [sc26eventpage](<https://devfeed.tech/tags/sc26eventpage.md>), [supercomputing](<https://devfeed.tech/tags/supercomputing.md>), [thecube](<https://devfeed.tech/tags/thecube.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

Dell's Brad Maltz says unexpectedly high frontier-model and hyperscaler token bills--cited as $10,000-$20,000 per person per month--are one reason enterprises reconsider pilots. Data sovereignty, IP leakage, security, and governance also shape production decisions. He describes interest in open-weight models that can be fine-tuned and run on-premises, including Nvidia Nemotron, while noting that frontier labs and open-weight models will coexist and that open-weight diversity must meet enterprise needs. Dell's AI Factory pitch extends beyond servers and storage to software, models, open-source tools, and operations; its ecosystem program validates software partners. For sovereign deployments, Maltz points to disconnected on-premises systems keeping models near enterprise-controlled GPUs and data. These are claims from a Dell-sponsored interview excerpt.

### Source excerpt

Enterprises have spent the AI boom racing to secure GPUs, but the harder decisions now sit above the hardware. Token bills, data sovereignty and model choice increasingly decide which AI pilots reach production, a shift on display at the SC26 supercomputing conference. Dell Technologies Inc. has spent the past year pushing its AI Factory beyond [...] The post Token sticker shock pushes enterprises from frontier labs toward open-weight AI factories appeared first on SiliconANGLE.

## LoopSkim: Lossless Multi-Depth Speculative Decoding for Looped Language Models

DevFeed: [LoopSkim: Lossless Multi-Depth Speculative Decoding for Looped Language Models](<https://devfeed.tech/articles/loopskim-lossless-multi-depth-speculative-decoding-for-looped-language-models-138956.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19896>)

Author: Muhan Xiao, Shijian Li

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [loops](<https://devfeed.tech/tags/loops.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [speculative-decoding](<https://devfeed.tech/tags/speculative-decoding.md>)

### AI overview

LoopSkim accelerates decoding in looped language models by using each loop's existing next-token readout to verify draft tokens, without an extra pass. Context-copied drafts enter after early loops; a later readout can replace a contradicted draft, and only full-depth readouts commit tokens. By fixing each position's random draws in advance, the method compares two tokens at each check and is lossless in exact arithmetic under sampling, for any number of checks and draft chains. The abstract reports an average speedup over autoregressive decoding 1.73x that of LoopSpec across greedy decoding and sampling, with checks between the first loops and full depth speeding every tested model. An optional draft head adds a second chain and raises the average to 1.99x. These are abstract-reported results; no benchmark details are provided here.

### Source excerpt

arXiv:2611.19896v1 Announce Type: new Abstract: Looped language models such as Ouro and Raven apply one weight-shared block several times per token, so autoregressive decoding (AR) runs every loop of every token in sequence. Pipelined self-speculative decoders overlap these loops by drafting tokens from shallow readouts and verifying them at full depth. Yet every loop of a looped LM ends in a readout of the next token, and draft tokens pass through every depth anyway, so each depth can check them without an extra pass. We introduce LoopSkim, in which the depth of the pipeline is the verifier. Draft tokens copied from the context, as in prompt lookup decoding, enter after the first loops, any later readout that contradicts one replaces it at that depth, and only full-depth readouts commit tokens. LoopSkim fixes the random draws of every position in advance, so each check compares two tokens and decoding stays lossless in exact arithmetic under sampling for any number of checks and draft chains. Without training, its speedup over AR averages 1.73x that of LoopSpec across greedy decoding and sampling, and its checks between the first loops and full depth speed it up on every model. An optional draft head adds a second chain and raises this average to 1.99x.

## Latent Lattice Language Models

DevFeed: [Latent Lattice Language Models](<https://devfeed.tech/articles/latent-lattice-language-models-137722.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21608>)

Author: Shivam Duggal, Zongze Wu, William T. Freeman, Antonio Torralba, Richard Zhang, Eli Shechtman

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Mamba](<https://devfeed.tech/topics/mamba.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gated-deltanet](<https://devfeed.tech/tags/gated-deltanet.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [lattice](<https://devfeed.tech/tags/lattice.md>), [mamba](<https://devfeed.tech/tags/mamba.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mlp](<https://devfeed.tech/tags/mlp.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [recurrent](<https://devfeed.tech/tags/recurrent.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [state](<https://devfeed.tech/tags/state.md>), [stream](<https://devfeed.tech/tags/stream.md>), [time](<https://devfeed.tech/tags/time.md>), [token](<https://devfeed.tech/tags/token.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

The Lattice of Latents adds recurrent per-layer latent streams that persist across tokens alongside the Transformer's depth-wise residual stream. These states directly condition attention, KV-cache writes, MLP computation, and decoding. In reported tests on 20B FineWeb-Edu tokens, a parameter-matched 406M-parameter Lattice Transformer reduced validation perplexity from 15.41 to 14.40. Combined with hyper-connections and input residuals, it reached 13.61, versus 14.22 for a Transformer with those enhancements. Lattice variants also improved perplexity at every tested native state size on Gated DeltaNet, Mamba-2, Triadic GDN, and GDN-Mamba-2 hybrids; Lattice-GDN and Lattice-Triadic-GDN beat pure baselines with four times larger native states. The evidence is limited to the abstract excerpt.

### Source excerpt

arXiv:2611.21608v1 Announce Type: cross Abstract: Language models typically increase per-token computation through deeper or looped layers and latent reasoning steps, while richer residual connections improve information flow across layers. We instead increase computational capacity: the number of active computational states a model maintains. We introduce the Lattice of Latents, a set of per-layer horizontal latent streams that run recurrently along time, alongside the vertical residual stream that flows through depth. Like hyper-connections, mHC, and attention or value residuals, the Lattice enhances the residual stream, but rather than adding or rerouting residual pathways through depth, it provides companion computational states that persist across tokens. Unlike hybrid Transformer-SSM architectures, whose recurrent states primarily serve as internal memory for sequence mixing, these latents directly condition the vertical stream's attention, KV-cache writes, MLP computation, and decoding. On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 over a parameter-matched Transformer. These gains extend across architectural variants. The Lattice composes with prior residual-stream enhancements: with hyper-connections and input residuals, it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer with the same enhancements) to 13.61. On Gated DeltaNet, Mamba-2, Triadic GDN, and GDN-Mamba-2 hybrids, it improves perplexity at every tested native state size, with Lattice-GDN and Lattice-Triadic-GDN variants outperforming pure baselines with 4x larger native states. These results identify persistent per-layer state as a complementary axis for scaling language-model computation.

## Dual-Stage Channel-Posterior Integration for MLM-Aided Token Communications

DevFeed: [Dual-Stage Channel-Posterior Integration for MLM-Aided Token Communications](<https://devfeed.tech/articles/dual-stage-channel-posterior-integration-for-mlm-aided-token-communications-137367.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18958>)

Author: Hyelin Nam, Jihoon Lee, Seung-Woo Ko, Jinho Choi, Seong-Lyun Kim

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Language models](<https://devfeed.tech/topics/language-models.md>), [Natural language processing](<https://devfeed.tech/topics/nlp.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [channel](<https://devfeed.tech/tags/channel.md>), [communications](<https://devfeed.tech/tags/communications.md>), [cosine](<https://devfeed.tech/tags/cosine.md>), [cosine-similarity](<https://devfeed.tech/tags/cosine-similarity.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-it](<https://devfeed.tech/tags/cs-it.md>), [digital](<https://devfeed.tech/tags/digital.md>), [eess-sp](<https://devfeed.tech/tags/eess-sp.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [errors](<https://devfeed.tech/tags/errors.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [framework](<https://devfeed.tech/tags/framework.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [inference](<https://devfeed.tech/tags/inference.md>), [input](<https://devfeed.tech/tags/input.md>), [integration](<https://devfeed.tech/tags/integration.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [math-it](<https://devfeed.tech/tags/math-it.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

The framework uses channel reliability to identify unreliable token positions and build a vocabulary-level channel posterior. Posterior-Weighted Input Embedding (PWE) supplies that posterior to the masked language model, and Adaptive Output Logit Fusion (AOF) reuses it to adjust output logits. This lets a bidirectional MLM reconstruct multiple tokens in one inference, without retransmission, transmitter-side encoding, or additional MLM training. In noisy-channel experiments, the authors report up to 67.2% higher cosine similarity than context-only MLM reconstruction, with gains across tested MLM architectures and existing language-model-aided methods.

### Source excerpt

arXiv:2611.18958v1 Announce Type: new Abstract: Token transmission over conventional digital modulation is vulnerable to channel impairments, leading to bit errors that cause received tokens to become contextually inconsistent. This paper proposes a receiver-side dual-stage channel-posterior integration framework for masked language model (MLM)-aided token reconstruction. In the proposed framework, the receiver detects unreliable token positions from channel reliability and constructs a vocabulary-level channel-posterior. In particular, within the receiver, Posterior-Weighted Input Embedding (PWE) injects this posterior into the MLM input, while Adaptive Output Logit Fusion (AOF) reuses it at the output with a channel-adaptive weight. Consequently, multiple unreliable tokens can be reconstructed in a single bidirectional MLM inference without retransmission, transmitter-side encoding, or additional MLM training. Experiments over noisy channels show up to a $67.2$\% improvement in cosine similarity over context-only MLM reconstruction, with consistent gains across different MLM architectures and existing language-model-aided methods.

## How Much Determinism Can Fine-Tuning Buy? Margin Regularisation Against Batch-Induced Token Flips

DevFeed: [How Much Determinism Can Fine-Tuning Buy? Margin Regularisation Against Batch-Induced Token Flips](<https://devfeed.tech/articles/how-much-determinism-can-fine-tuning-buy-margin-regularisation-against-batch-induced-token-flips-137403.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19135>)

Author: Charan Teja Nagisettygari, Siva Raja Sindiramutty

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Language models](<https://devfeed.tech/topics/language-models.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [quantization](<https://devfeed.tech/topics/quantization.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [apply](<https://devfeed.tech/tags/apply.md>), [batch](<https://devfeed.tech/tags/batch.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [fewer](<https://devfeed.tech/tags/fewer.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gap](<https://devfeed.tech/tags/gap.md>), [greedy](<https://devfeed.tech/tags/greedy.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [inference](<https://devfeed.tech/tags/inference.md>), [invariance](<https://devfeed.tech/tags/invariance.md>), [margin](<https://devfeed.tech/tags/margin.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [noise](<https://devfeed.tech/tags/noise.md>), [nondeterminism](<https://devfeed.tech/tags/nondeterminism.md>), [precision](<https://devfeed.tech/tags/precision.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

Greedy decoding can produce different tokens for the same prompt when batch composition changes. Across six model families and three numerical precisions, the measured perturbation scaled with representable precision; some tensor shapes happened to be batch-invariant. On GSM8K, hinge penalization of the top-two logit gap reduced token flips by 35% and divergence from 53.8% to 38.1%, but cost 1.3-2.9 accuracy points, with the cost increasing with margin size. It did not change the gap threshold associated with surviving flips (median 0.125), because the soft penalty leaves the largest hardware perturbation intact. The abstract reports that a per-model viability diagnosis held for six models and was falsified for one.

### Source excerpt

arXiv:2611.19135v1 Announce Type: cross Abstract: Greedy decoding at zero temperature is widely assumed to be deterministic, although the exact same prompt may yield different tokens depending on the number of other prompts sharing the batch for inference. The currently available solutions operate during inference, trading determinism for throughput. Can we train a model in such a way that fewer decoding steps would be susceptible? We characterize the perturbation empirically across six model families and three numerical precisions and observe two empirical facts about the measurement: the perturbation depends exactly on the representable precision (8.00x for bfloat16 vs float16, 8192.0x for float16 vs float32, in line with predictions based on mantissa bits) and the batch invariance happens accidentally at certain tensor shapes, allowing a practitioner to achieve zero nondeterminism merely by setting the sequence length. Then we apply hinge penalization of the model's top-1 / top-2 logit gap, scaled to twice the measured noise quantile. On GSM8K dataset it leads to a 35% reduction in token flips and divergence from 53.8% to 38.1%, at the cost of 1.3 - 2.9 accuracy points, increasing monotonically with the margin size. It does not affect the gap threshold sufficient for determinism: the logit gaps of surviving flips are unchanged (median 0.125 for all arms), as the hinge is a soft penalty and the largest hardware perturbation remains the same. We reject seven hypotheses, including a per-model viability diagnosis, holding on six models and falsified on the seventh.

[Next page](<https://devfeed.tech/topics/language-models.md?cursor=WyIyMDI2LTExLTE3VDA1OjAwOjAwKzAwOjAwIiwgImM5YTE1OWUwLWZiZmMtNDdlOS05NmRiLTk4NTdlMzE2NTRiZiJd>)