# model architecture

Model architecture is the overall structural design of a model, including how its layers, components, and connections are organized.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Revealing Pathway Interactions in Network Computations with Complete Path Contribution

DevFeed: [Revealing Pathway Interactions in Network Computations with Complete Path Contribution](<https://devfeed.tech/articles/revealing-pathway-interactions-in-network-computations-with-complete-path-contribution-145142.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26101>)

Author: Shenghua Liu, Youssef Faragalla, Surya Ganguli, Stephen A. Baccus

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Tool](<https://devfeed.tech/topics/tool.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [circuit](<https://devfeed.tech/tags/circuit.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [mouse](<https://devfeed.tech/tags/mouse.md>), [network](<https://devfeed.tech/tags/network.md>), [q-bio-nc](<https://devfeed.tech/tags/q-bio-nc.md>)

### AI overview

Complete Path Contribution decomposes a network's output into mathematically complete contributions from individual input-to-output paths, enabling direct study of higher-order pathway interactions. Summing path contributions also defines complete contributions for circuits at different levels of granularity. An algorithm finds the k strongest paths without enumerating them all; its runtime was empirically linear in k rather than exponential in network depth. The authors apply the method to benchmark image classifiers to quantify excitatory and inhibitory interactions, and to an interpretable mouse-retina model, where it reveals pathway interaction motifs associated with dynamic receptive fields under time-varying natural stimuli.

### Source excerpt

arXiv:2611.26101v1 Announce Type: new Abstract: Networks pass signals through many layers of interconnected units, so explaining what they compute means tracing how information travels from input to output and how pathways interact. Most existing approaches analyze only pairwise interactions between components such as units, layers or attention heads. Even when these measurements are combined post hoc into a circuit, they do not capture higher-order interactions along full causal chains (paths) from input to output, and so cannot address how exponentially many paths interact to construct network computation. This leaves a large set of mechanistic questions unanswerable, for example whether inhibitory pathways veto excitatory ones, and how the two combine to yield network decisions. To address this gap, we introduce Complete Path Contribution, a method to decompose the network output into mathematically complete contributions of individual paths to study path-level interactions directly and at scale. Partial summation then induces definitions of contributions for any circuit at any granularity level, with completeness guaranteed at every level. We further present an efficient algorithm that identifies the $k$ strongest paths without examining all of them, with runtime empirically linear in $k$ rather than exponential in depth. By applying Path Contribution to benchmark image classification networks, we quantify excitatory-inhibitory interactions and show how they collectively construct network decisions at a level of detail previously unmeasurable. Furthermore, in an interpretable model of the mouse retina, it reveals how dynamic receptive fields emerge from excitatory-inhibitory interaction motifs among pathways between different cell types in response to time-varying natural stimuli. Overall, Path Contribution reveals how a network's causal pathways interact at scale to transform input to output.

## TabPFN: One Model to Rule Them All?

DevFeed: [TabPFN: One Model to Rule Them All?](<https://devfeed.tech/articles/tabpfn-one-model-to-rule-them-all-144854.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2505.20003>)

Author: Qiong Zhang, Yan Shuo Tan, Qinglong Tian, Pengfei Li

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [application](<https://devfeed.tech/tags/application.md>), [bayesian-inference](<https://devfeed.tech/tags/bayesian-inference.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [foundation-model](<https://devfeed.tech/tags/foundation-model.md>), [learning](<https://devfeed.tech/tags/learning.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [tabpfn](<https://devfeed.tech/tags/tabpfn.md>), [tabular-data](<https://devfeed.tech/tags/tabular-data.md>)

### AI overview

The paper interprets TabPFN, a transformer for tabular regression and classification, as approximate Bayesian inference and examines its potential relevance to statistics. The authors report that out-of-the-box TabPFN sometimes outperforms specialized state-of-the-art methods in semi-supervised parameter estimation, prediction under covariate shift, and heterogeneous treatment effect estimation. They suggest one explanation is its ability to adapt to both nonparametric and parametric structure; in some experiments, it outperforms LASSO even when LASSO's assumptions are correctly specified. The evidence provided is an abstract, so it does not detail experimental setups or results.

### Source excerpt

arXiv:2505.20003v3 Announce Type: replace Abstract: Hollmann et al. (Nature 637 (2025) 319-326) recently introduced TabPFN, a transformer-based deep learning model for regression and classification on tabular data, which they claim "outperforms all previous methods on datasets with up to 10,000 samples by a wide margin, using substantially less training time." Furthermore, they have called TabPFN a "foundation model" for tabular data, as it can support "data generation, density estimation, learning reusable embeddings and fine-tuning". In this paper, we provide a tailored explanation of how TabPFN works for a statistics audience, by emphasizing its interpretation as approximate Bayesian inference. We then explore the significance of TabPFN to the field of statistics: We show that an out-of-the-box application of TabPFN can sometimes outperform specialized state-of-the-art methods for semi-supervised parameter estimation, prediction under covariate shift, and heterogeneous treatment effect estimation. As a partial explanation for the predictive effectiveness of TabPFN, we show that it can simultaneously adapt to both nonparametric structure and parametric structure, for instance, sometimes outperforming LASSO even when assumptions are correctly specified. All experiments can be reproduced using the code provided at https://github.com/qinglong-tian/tabpfn_study (https://github.com/qinglong-tian/tabpfn_study).

## On the Informativeness of Structural Stability Bounds for GNNs: GINs versus GATs

DevFeed: [On the Informativeness of Structural Stability Bounds for GNNs: GINs versus GATs](<https://devfeed.tech/articles/on-the-informativeness-of-structural-stability-bounds-for-gnns-gins-versus-gats-145076.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25603>)

Author: Alan Gany, Paul Landrier, Bogdan Cautis, Laks V. S. Lakshmanan, Silviu Maniu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [graph theory](<https://devfeed.tech/topics/graph-theory.md>)

Tags: [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-lo](<https://devfeed.tech/tags/cs-lo.md>), [graph](<https://devfeed.tech/tags/graph.md>), [graph-isomorphism](<https://devfeed.tech/tags/graph-isomorphism.md>), [graph-neural-networks](<https://devfeed.tech/tags/graph-neural-networks.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [node](<https://devfeed.tech/tags/node.md>)

### AI overview

The structural stability bound is informative for GINs under a condition involving perturbation size, network depth, and maximum degree; it is never tighter than a graph-independent output-diameter bound for single-head GATv1. The authors derive bounds for deterministic edge perturbations that need not be small. Their condition holds with a wide margin on Cora but fails beyond a few layers on low-degree MUTAG graphs. For GATv1, the result motivates analyzing stability through attention masses.

### Source excerpt

arXiv:2611.25603v1 Announce Type: new Abstract: Stability bounds for graph neural networks limit how much node representations can drift when the graph changes, but they are rarely compared with the simplest alternative: a bound on the distance between any two outputs, whatever the graph. We derive structural stability bounds for Graph Isomorphism Networks (GINs) and single-head GATv1 networks under deterministic edge perturbations that need not be small, and compare each with such a perturbation-independent diameter bound. For GINs, we characterize exactly when the stability bound is tighter, through a condition linking perturbation size, depth and maximum degree; it holds with a wide margin on Cora, but fails beyond a few layers on the low-degree MUTAG graphs. For GATv1, the bound is never tighter, which calls for analyses based on attention masses.

## FEA solver integration of a composite World Model with a Spiking Neural Network memory Block for computationally efficient modelling of defects and variabilities

DevFeed: [FEA solver integration of a composite World Model with a Spiking Neural Network memory Block for computationally efficient modelling of defects and variabilities](<https://devfeed.tech/articles/fea-solver-integration-of-a-composite-world-model-with-a-spiking-neural-network-memory-block-for-computationally-efficient-modelling-of-defects-and-variabilities-145179.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26419>)

Author: Bassam El Said, Adam Sobey

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [defects](<https://devfeed.tech/tags/defects.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [encoder](<https://devfeed.tech/tags/encoder.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [inference](<https://devfeed.tech/tags/inference.md>), [integration](<https://devfeed.tech/tags/integration.md>), [material](<https://devfeed.tech/tags/material.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [modelling](<https://devfeed.tech/tags/modelling.md>), [models](<https://devfeed.tech/tags/models.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [snn](<https://devfeed.tech/tags/snn.md>), [state](<https://devfeed.tech/tags/state.md>), [structure](<https://devfeed.tech/tags/structure.md>), [world-model](<https://devfeed.tech/tags/world-model.md>)

### AI overview

The proposed Composites World Model aims to make structural-scale composite simulation more computationally practical by replacing costly material-scale models with a surrogate that can represent local architectures and defects, track irreversible state, and run recursively. It combines a material encoder, a Spiking Neural Network memory block for long-term material state, and transformer blocks for short-term applied strain. Training proceeds through architecture encoding, short-window base training, and recursive fine-tuning with scheduled sampling to limit cumulative error drift. At inference, the model is disassembled and coupled to a nonlinear FE solver. Benchmarks on IM7/8552 laminates under tension, compression, and transverse shear, including pristine and wrinkled cases, report prediction errors of 0.4%-4.8%; the abstract attributes successful tracking of material state to the SNN memory block.

### Source excerpt

arXiv:2611.26419v1 Announce Type: new Abstract: Each fibre-reinforced composite structure has a unique material architecture. The manufacturing methods used and the presence of defects lead to local variations which require material-scale models that can resolve the response in real time at every location in the structure. Concurrent multi-scale schemes provide this capability, but at a prohibitive computational cost for large structures as the number of integration points and time increments scales with size. Surrogates have been studied extensively as replacements for material-scale models to mitigate these computational limitations. A surrogate must represent an arbitrary local architecture including defects, track the irreversible material state over an unknown number of increments, remain stable when driven recursively by its own predictions, and fit within the memory available at every integration point. Here, a Composites World Model based on a Multi-Modal Spiking Transformer architecture is designed to meet these requirements. The proposed architecture combines a material encoder, a Spiking Neural Network memory block that tracks the long-term internal state of the material, and transformer blocks that learn the short-term applied strain state. The system is trained in three stages: material architecture encoding, base training on short time windows, and recursive fine-tuning with scheduled sampling to suppress cumulative error drift. The trained model is disassembled at inference and coupled to a non-linear FE solver, enabling structural scale simulation of composites containing embedded defects. Benchmarks against experimental results for IM7/8552 laminates under tension, compression and transverse shear, for pristine and wrinkled composites, show that the SNN memory block successfully tracks the material internal state and results in prediction errors between 0.4% and 4.8%.

## GLiDE: A Fast, Calibrated Decision Model Distilled from Reasoning Teachers

DevFeed: [GLiDE: A Fast, Calibrated Decision Model Distilled from Reasoning Teachers](<https://devfeed.tech/articles/glide-a-fast-calibrated-decision-model-distilled-from-reasoning-teachers-145678.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26509>)

Author: Sahibzada Allahyar, Julia White, Dhruv Atreja, George Hurn-Maloney, Urchade Zaratiana

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [calibrated-decision-model](<https://devfeed.tech/tags/calibrated-decision-model.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [decision](<https://devfeed.tech/tags/decision.md>), [decision-model](<https://devfeed.tech/tags/decision-model.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [di](<https://devfeed.tech/tags/di.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [fast](<https://devfeed.tech/tags/fast.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [model](<https://devfeed.tech/tags/model.md>), [performance](<https://devfeed.tech/tags/performance.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reasoning-model](<https://devfeed.tech/tags/reasoning-model.md>)

### AI overview

GLiDE transfers reasoning capability into single-pass decision models that score candidate answers without generating text or intermediate reasoning. The 0.4B-27B family uses Qwen hybrid language models with bidirectional interactions among context, question, and candidates, plus soft-target distillation from multiple teachers--including a reasoning model--and progressive post-training on varied decision tasks. Parameter-free recurrent depth adds inference-time computation without token generation. The abstract reports that each of five sizes outperforms evaluated alternatives of equal or smaller size on the Hugging Face benchmark. The 27B model scores 71.52 on a hidden set with 2.0% expected calibration error and 92.9 ms median latency on one GPU; checkpoint averaging and recurrent depth raise the score to 73.37, with 2.1% calibration error.

### Source excerpt

arXiv:2611.26509v1 Announce Type: new Abstract: Decision models enable fast, confidence-aware decisions by scoring candidate answers in a single forward pass, without generating text or intermediate reasoning. Yet reasoning can substantially improve decision quality, raising the question of whether its benefits can be retained without the cost of generating it. We investigate this question with \textbf{GLiDE}, a family of decision models spanning 0.4B to 27B parameters. Building on Qwen hybrid language models, GLiDE enables bidirectional interactions between the context, question, and candidate answers by removing causal masking from full-attention layers. To transfer reasoning capabilities into single-pass inference, we combine soft-target distillation from multiple teachers, including a reasoning model, with progressive post-training on diverse decision tasks. We further explore parameter-free recurrent depth to increase inference-time computation without generating tokens. Across five model sizes, GLiDE lies on the performance--parameter Pareto frontier of the Hugging Face \DI{}, with each model outperforming evaluated alternatives of equal or smaller size. \dmtwo{} (27B) achieves a hidden-set score of 71.52, with an expected calibration error of 2.0\% and a median latency of 92.9\,ms on a single GPU. Further checkpoint averaging and parameter-free recurrent depth yield GLiDE~27B, which reaches 73.37 with an expected calibration error of 2.1\%.

## Sirius-Tiamat: How Small Can a One-Pass Decision Encoder Get Before Its Decisions Change?

DevFeed: [Sirius-Tiamat: How Small Can a One-Pass Decision Encoder Get Before Its Decisions Change?](<https://devfeed.tech/articles/sirius-tiamat-how-small-can-a-one-pass-decision-encoder-get-before-its-decisions-change-145659.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26339>)

Author: Mohammed Nayeem Khan, Vrishank Tiwari

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [decision-making](<https://devfeed.tech/topics/decision-making.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [bf16](<https://devfeed.tech/tags/bf16.md>), [bits](<https://devfeed.tech/tags/bits.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [decision](<https://devfeed.tech/tags/decision.md>), [distilling](<https://devfeed.tech/tags/distilling.md>), [encoder](<https://devfeed.tech/tags/encoder.md>), [error](<https://devfeed.tech/tags/error.md>), [exit](<https://devfeed.tech/tags/exit.md>), [laya](<https://devfeed.tech/tags/laya.md>), [layers](<https://devfeed.tech/tags/layers.md>), [llm](<https://devfeed.tech/tags/llm.md>), [model](<https://devfeed.tech/tags/model.md>), [pass](<https://devfeed.tech/tags/pass.md>), [question](<https://devfeed.tech/tags/question.md>), [scale](<https://devfeed.tech/tags/scale.md>), [schema](<https://devfeed.tech/tags/schema.md>), [size](<https://devfeed.tech/tags/size.md>), [student](<https://devfeed.tech/tags/student.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [temperature](<https://devfeed.tech/tags/temperature.md>)

### AI overview

The study finds that compression can preserve Laya's decisions to varying degrees, with results depending on method and evaluation set. On in-family versus out-of-family items, bf16 changed 0.2% versus 2.2% of decisions, 6-bit grouped rounding 1.3% versus 5.9%, and deleting six layers 0.9% versus 9.1%. Pruning also raised expected calibration error from 5.9% to 39.2%; a fitted temperature requiring no labels reduced it to 3.6%. A single scale per tensor failed below six bits, while CPU dynamic int8 changed 30% of decisions, nearly all through one layer. At equal stored size, rounding changed fewer decisions than deletion. Distillation reduced a 16-layer student's in-family flips from 44% to 9%, though out-of-family flips were 45%. The evidence is limited to synthetic, LLM-labelled data and one architecture.

### Source excerpt

arXiv:2611.26339v1 Announce Type: new Abstract: A one-pass decision encoder reads a text, a question and several named options, and returns a probability for each option in a single forward pass. We ask how far Laya, a 421M-parameter encoder of this kind, can be compressed before its decisions change. We count decisions that differ from Laya's own full-precision (fp32) answer, and track calibration (how well confidence matches accuracy, measured by expected calibration error, ECE), on 1,221 synthetic items with the training schemas, 273 with an unseen schema, and 800 out-of-family questions from ARC, MMLU and HellaSwag. Layer pruning of a related model family is known; we add flips, calibration, held-out data and retraining. bf16 changes 0.2% of in-family decisions but 2.2% out of family; 6-bit grouped rounding 1.3% and 5.9%; deleting 6 layers 0.9% and 9.1%, while ECE rises from 5.9% to 39.2%. One fitted temperature, needing no labels, restores it to 3.6%. One scale per tensor fails below six bits, and CPU dynamic int8 changes 30% of decisions, nearly all through one layer. Total variation tracks flips for rounding but not for deletion, and an early exit saves layers over a static cut only if its confidence gate is first recalibrated. Distilling a 16-layer student cuts its in-family flips from 44% to 9% (45% out of family). At equal stored size, rounding flips fewer decisions than deletion. Data are synthetic and LLM-labelled, for one architecture.

## Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality

DevFeed: [Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality](<https://devfeed.tech/articles/physics-of-language-models-part-4-2-canon-layers-at-scale-where-synthetic-pretraining-resonates-in-reality-145447.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24963>)

Author: Zeyuan Allen-Zhu

Published: 2026-11-20T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [canon](<https://devfeed.tech/tags/canon.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [deltanet](<https://devfeed.tech/tags/deltanet.md>), [gated-linear-attention](<https://devfeed.tech/tags/gated-linear-attention.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [linear](<https://devfeed.tech/tags/linear.md>), [playgrounds](<https://devfeed.tech/tags/playgrounds.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [research](<https://devfeed.tech/tags/research.md>), [scale](<https://devfeed.tech/tags/scale.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

Across 64 pretraining runs at 1.3-8B parameters, architectural differences that nearly disappeared in carefully tuned 1.3B models trained on 100B tokens became measurable at 1-2T tokens. At 8B, adding Canon local token mixing to Transformers cut the tokens needed to reach the same MMLU accuracy by about 30%. With Canon, gated linear attention matched Gated DeltaNet and outperformed Mamba2 at 8B/1T; the authors suggest the latter models' additional machinery may mainly mix neighboring tokens. Yet benchmark scores obscured basic in-context weaknesses: linear models could not reliably retrieve information from a 500-token context, and no tested model reliably beat chance on a simple two-hop question. The abstract argues that small synthetic experiments can expose differences standard benchmarks miss and recommends including in-context reasoning examples early in pretraining.

### Source excerpt

arXiv:2611.24963v1 Announce Type: cross Abstract: Do standard benchmarks see the architectural differences that matter? We study it with 64 from-scratch pretraining runs of Transformers and three linear-model families at 1.3-8B parameters on 1-2T Nemotron-CC tokens, at a cost of roughly 100k USD per 8B run. In academic-scale pretraining (1.3B parameters, 100B tokens), architectural differences can all but disappear when the models are carefully tuned. Instead, at 1-2T tokens they become measurable: Canon layers, which add local token mixing, consistently improve Transformers, saving about 30% of pretraining tokens for the same MMLU accuracy at 8B. We also introduce GLA5 and GDN2, which improve gated linear attention and Gated DeltaNet with low-rank output gates. Surprisingly, with Canon, the simplest linear model, gated linear attention, matches Gated DeltaNet and outperforms Mamba2 even at 8B/1T. Therefore, the extra machinery of Gated DeltaNet and Mamba2 may do little more than mix neighboring tokens. Even at 8B/1T, however, high scores on standard benchmarks can hide weaknesses in basic in-context skills. In particular, linear models that match or beat Transformers on these benchmarks cannot, in fact, reliably retrieve information from a short 500-token context. Moreover, no model, Transformer or linear, reliably beats random guessing on arguably the simplest two-hop question, stated in a few short sentences, even at this scale. These findings $\textbf{challenge traditional architecture research}$ based primarily on standard benchmark scores. In contrast, small synthetic playgrounds, which predicted the architectural differences above at <100M parameters, can reveal what these benchmarks miss. For practitioners, one lesson is to include in-context reasoning examples in pretraining data from an early stage, rather than reserve them for later mid-training, let alone post-training.

## Live Migration of Gated DeltaNet Requests: Exact Catch-Up and the Cost of Log Backlog

DevFeed: [Live Migration of Gated DeltaNet Requests: Exact Catch-Up and the Cost of Log Backlog](<https://devfeed.tech/articles/live-migration-of-gated-deltanet-requests-exact-catch-up-and-the-cost-of-log-backlog-146130.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26153>)

Author: Hanqing Yu, Jiaqi Yang, Si Wu, Feng Li, Xiuzhen Cheng, Pengfei Hu, Yongbo Yu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [data migrations](<https://devfeed.tech/topics/data-migrations.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [backlog](<https://devfeed.tech/tags/backlog.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [gated-deltanet](<https://devfeed.tech/tags/gated-deltanet.md>), [live-migration](<https://devfeed.tech/tags/live-migration.md>), [log](<https://devfeed.tech/tags/log.md>)

### AI overview

Live migration of streaming Gated DeltaNet requests recovers recurrent state at the correct token boundary on a worker with the same model weights. The paper adapts snapshots, batched replay and token catch-up, and proposes a compact update log; exact catch-up retains native single-token decoding and matches the applicable bitwise reference on tested inputs. In 1,350 scheduled pure-GDN trials across two GPU deployments, compact logs had the shortest exact handoff gap in all 90 configurations, while exact catch-up released the source earliest in the 80 complete configurations. When arithmetic differences were allowed, 32-step replay achieved the highest output rate; at 10 MB/s it cut median handoff gaps from 445-464 ms for logs to about 81 ms. Retrospective selection across 40 historical KV-aligned Qwen configurations reduced mean gap by 13.9% versus the strongest scored simple rule under either numerical contract. Whole-request hybrid results are initial, and joint KV/GDN policy evaluation remains to be done.

### Source excerpt

arXiv:2611.26153v1 Announce Type: new Abstract: Live migration of a streaming Gated DeltaNet (GDN) request requires recovering recurrent state at the correct token boundary on a worker with the same model weights. We adapt snapshots, batched replay, and token catch-up, and design a compact update-log migration path. Exact catch-up preserves native single-token decode and matches the applicable bitwise reference on the tested inputs. We use both a prior-based and a probe-calibrated selector to choose a path for user interruption, source release, or output rate. In 1,350 scheduled pure-GDN trials on two GPU deployments, compact logs achieved the shortest exact handoff gap in all 90 configurations, including 10 winners determined using a replay timeout bound. In the 80 complete configurations, exact catch-up always released the source earliest, while 32-step replay provided the highest output rate when arithmetic differences were allowed. At 10 MB/s, 32-step replay also reduced the median handoff gap from 445--464 ms for logs to about 81 ms across the two deployments. On 40 historical KV-aligned Qwen configurations, retrospective selection reduced the mean gap by 13.9% against the strongest scored simple rule under either numerical contract. We report initial whole-request hybrid results and specify how the joint KV and GDN policy will be evaluated.

## The Window Lever: Longer Observation Windows Outweigh the Observed Effects of Data, Capacity, and Architecture in Customer Behavior Forecasting

DevFeed: [The Window Lever: Longer Observation Windows Outweigh the Observed Effects of Data, Capacity, and Architecture in Customer Behavior Forecasting](<https://devfeed.tech/articles/the-window-lever-longer-observation-windows-outweigh-the-observed-effects-of-data-capacity-and-architecture-in-customer-behavior-forecasting-145191.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26521>)

Author: Tara Sowrirajan, Yanqiao Wang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Time Series](<https://devfeed.tech/topics/time-series.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [customer](<https://devfeed.tech/tags/customer.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [model](<https://devfeed.tech/tags/model.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [weekly](<https://devfeed.tech/tags/weekly.md>), [window](<https://devfeed.tech/tags/window.md>), [windows](<https://devfeed.tech/tags/windows.md>)

### AI overview

In the reported long-horizon customer-behavior forecasting experiments, extending history from 8 to 16 weeks increased log-space R² by 0.070-0.107 across model families at the 1M tier; matched test cohorts suggest the raw comparison understates the gain. The direction also held on five public datasets. The authors report smaller, unresolved observed effects for data, model capacity, and architecture, and recommend trying longer windows for this class of data. However, longer windows also admit more and colder training customers, and the design does not separate history length from calendar coverage. The authors therefore report a robust positive association, not a measured calendar-independent history effect. The only fully blind registered prediction failed.

### Source excerpt

arXiv:2611.26521v1 Announce Type: new Abstract: In our experiments on long-horizon customer-behavior forecasting, a longer observation window outweighed the observed effects of more data, more capacity and newer architectures. The window is the weeks of history a model sees before the week it forecasts from. Going from an 8- to a 16-week window adds a raw +0.070 to 0.107 log-space $R^2$ in every family at 1M. Matched test cohorts, which score the same customer-weeks at every window, show this raw comparison understates the gain in every family they cover. The direction holds on five public datasets, four spanning over a year (ridge and our transformer on all five). On our internal data, longer windows admit up to 10% more and colder training customers, which matching does not fix. Our design does not separate window length from calendar coverage, because every test anchor (the week a forecast is made from) falls in one eight-week band. On training-period anchors, the weekly ridge probe's gain persists outside the holiday season, keeping about 72% of it. We therefore report a robust positive association with longer windows and do not identify the magnitude of a calendar-independent history effect. Of four internally registered predictions, the only fully blind one failed. An activity-mixture model predicted +0.014 from 24 to 32 weeks; the weekly-aggregate probe returned -0.0004. For the sequence families, the other three levers are about an order of magnitude smaller, bounded and unresolved. Their observed effects are at most 0.006 between the nominal 1M and 10M tiers and 0.0052 among five sequence families at 16 weeks with clocks as built, both single-seed on the full test split, and 0.006 for capacity on three-seed means. These are observed intervals, not equivalence bounds. Our claim is the order-of-magnitude contrast between the window and those bounds. A team building on this class of data should start by lengthening the window.

## GPU-Accelerated Secure Inference for Drug--Target Interaction Prediction

DevFeed: [GPU-Accelerated Secure Inference for Drug--Target Interaction Prediction](<https://devfeed.tech/articles/gpu-accelerated-secure-inference-for-drug-target-interaction-prediction-145088.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25664>)

Author: Sikha Pentyala, Steven Golob, Daniil Filienko, Ming-Chun Kao, Martine De Cock

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [AI Architecture](<https://devfeed.tech/topics/ai-architecture.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [accelerated](<https://devfeed.tech/tags/accelerated.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [co-design](<https://devfeed.tech/tags/co-design.md>), [communication](<https://devfeed.tech/tags/communication.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [competition](<https://devfeed.tech/tags/competition.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [custom](<https://devfeed.tech/tags/custom.md>), [deep](<https://devfeed.tech/tags/deep.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [deep-learning-models](<https://devfeed.tech/tags/deep-learning-models.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-accelerated](<https://devfeed.tech/tags/gpu-accelerated.md>), [inference](<https://devfeed.tech/tags/inference.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [target](<https://devfeed.tech/tags/target.md>)

### AI overview

The paper's first-place iDASH 2026 Track III solution combines an MPC-friendly DTI model with GPU protocols for two-party secure inference. SlimDTA reduces operations on the secret drug, especially nonlinear operations that require communication between servers. The GPU framework evaluates linear layers using additive secret sharing and nonlinear layers using function secret sharing, with a custom CUDA comparison kernel. The authors report under 10 ms per prediction on average, with secure predictions matching the plaintext model to within 0.02%.

### Source excerpt

arXiv:2611.25664v1 Announce Type: cross Abstract: Deep learning models that predict drug--target interactions (DTI) can speed up drug discovery, but the candidate drugs they score are valuable intellectual property that their owners cannot reveal. Secure multiparty computation (MPC) lets two servers compute a prediction on a secret-shared drug, without either server learning the drug. However, secure inference of deep models is slow. Non-linear operations require communication between the servers, and MPC arithmetic does not map naturally to GPUs. In this paper, we present our solution to Track~III of the iDASH 2026 privacy and security competition, which asked for fast two-party secure DTI inference on GPUs. Our solution is built on the co-design of MPC-friendly models and MPC protocols. On the model side, we design SlimDTA, a DTI model that reduces the operations on the secret drug, and in particular the non-linear operations that require communication. On the protocol side, we build a GPU framework that evaluates the linear layers with additive secret sharing and the non-linear layers with function secret sharing, including a custom CUDA kernel for the comparisons. Our solution won first place in the competition, at under 10,ms per prediction on average, and its secure predictions match the plaintext model to within 0.02%.

## Model Compression Beyond Quantization

DevFeed: [Model Compression Beyond Quantization](<https://devfeed.tech/articles/model-compression-beyond-quantization-143867.md>)

Original publisher: [Read original article](<https://machinelearningmastery.com/model-compression-beyond-quantization/>)

Author: Adrian Tam

Published: 2026-11-19T14:06:50Z

Content type: article

Language: en

Sources: [MachineLearningMastery.com](<https://devfeed.tech/sources/machinelearningmastery-com.md>)

Topics: [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>)

Tags: [compression](<https://devfeed.tech/tags/compression.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [inference-from-transformer-models](<https://devfeed.tech/tags/inference-from-transformer-models.md>), [knowledge-distillation](<https://devfeed.tech/tags/knowledge-distillation.md>), [model-compression](<https://devfeed.tech/tags/model-compression.md>), [pruning](<https://devfeed.tech/tags/pruning.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>), [structured](<https://devfeed.tech/tags/structured.md>)

### AI overview

Compression methods can reduce model size by removing weights or structures, or by training a smaller model to imitate a larger one. Unstructured magnitude pruning zeros low-magnitude weights, but speed or storage gains depend on sparse representation, matrix shape, sparsity, dtype, hardware, and kernel support. Structured pruning removes coordinated units such as MLP channels; matching dimensions must be edited together, and hardware-friendly sizes matter. Knowledge distillation trains a student from teacher signals such as output distributions or intermediate representations. It requires teacher targets, compatible logits and token positions, and representative training data; generating targets has a cost. The text stresses evaluating held-out quality, memory, latency, and throughput, since reduced parameter counts do not guarantee preserved behavior or faster serving.

### Source excerpt

Overview This post is divided into three parts; they are: - Pruning and Sparsity - Structured Pruning - Knowledge Distillation Pruning and Sparsity Pruning sets selected model parameters to zero or removes larger structures.

## Sparse Layers are Critical to Scaling Looped Language Models

DevFeed: [Sparse Layers are Critical to Scaling Looped Language Models](<https://devfeed.tech/articles/sparse-layers-are-critical-to-scaling-looped-language-models-143256.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.09165>)

Author: Ryan Lee, Jacob Biloki, Edward J. Hu, Jonathan May

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mixture of Experts (MoE)](<https://devfeed.tech/topics/mixture-of-experts-moe.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layers](<https://devfeed.tech/tags/layers.md>), [loops](<https://devfeed.tech/tags/loops.md>), [memory](<https://devfeed.tech/tags/memory.md>), [mixture-of-experts-moe](<https://devfeed.tech/tags/mixture-of-experts-moe.md>), [quality](<https://devfeed.tech/tags/quality.md>), [routing](<https://devfeed.tech/tags/routing.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [standard](<https://devfeed.tech/tags/standard.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

The abstract reports that Looped-MoE models scale better than the standard transformer baseline, while dense looped models do not. It attributes the difference to routing divergence: shared layers activate different experts on successive passes, adding expressivity without additional parameters. It also finds that loop boundaries are better early-exit points than those in standard models because outputs converge earlier there. The authors report improved compute-quality trade-offs and potential memory and inference savings with minimal quality degradation; the supplied excerpt gives no quantitative results or experimental details.

### Source excerpt

arXiv:2605.09165v3 Announce Type: replace Abstract: Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.

## Stability Does Not Mirror Plasticity: A Module-Level View of Declarative and Procedural Learning in Transformers

DevFeed: [Stability Does Not Mirror Plasticity: A Module-Level View of Declarative and Procedural Learning in Transformers](<https://devfeed.tech/articles/stability-does-not-mirror-plasticity-a-module-level-view-of-declarative-and-procedural-learning-in-transformers-143391.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23427>)

Author: Jinjun Peng, Wailiang Zhao, Baishakhi Ray, Zhou Yu, Jufeng Yang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformers](<https://devfeed.tech/topics/transformers.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [best](<https://devfeed.tech/tags/best.md>), [capability](<https://devfeed.tech/tags/capability.md>), [continual-learning](<https://devfeed.tech/tags/continual-learning.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [declarative](<https://devfeed.tech/tags/declarative.md>), [echo](<https://devfeed.tech/tags/echo.md>), [environment](<https://devfeed.tech/tags/environment.md>), [existing](<https://devfeed.tech/tags/existing.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [explain](<https://devfeed.tech/tags/explain.md>), [functional](<https://devfeed.tech/tags/functional.md>), [learning](<https://devfeed.tech/tags/learning.md>), [mirror](<https://devfeed.tech/tags/mirror.md>), [module](<https://devfeed.tech/tags/module.md>), [plasticity](<https://devfeed.tech/tags/plasticity.md>), [procedural](<https://devfeed.tech/tags/procedural.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [view](<https://devfeed.tech/tags/view.md>)

### AI overview

In controlled synthetic experiments, plasticity and stability in transformers depend on different aspects of module function. QK and VO projections were most plastic for procedural tasks, while MLPs were most plastic for declarative memorization. During learning of a new procedure, updating QK preserved over half of stored declarative memory; updating MLPs preserved much less, and updating VO lost roughly four-fifths. Reset-and-recovery experiments suggest plasticity reflects a module's fit for a task's main computation or storage, while stability also depends on how existing capabilities use that module, including to access information stored elsewhere. Experiments on a realistic model and tasks broadly echoed these patterns.

### Source excerpt

arXiv:2611.23427v1 Announce Type: new Abstract: Plasticity and stability in continual learning are typically studied at the entire-model level and in a task-agnostic manner, yet prior work has reported divergent conclusions across settings. Inspired by cognitive distinctions between declarative and procedural memory, we instead study how plasticity and stability vary across transformer modules and task types. In a controlled synthetic environment with clean measurement, we show a double dissociation: query-key (QK) and value-output (VO) projections are the most plastic on procedural tasks, whereas MLPs are the most plastic on declarative memorization. Stability, however, does not simply mirror plasticity: when learning a new procedure, updating QK retains over half of the stored declarative memory, while updating MLPs retains far less and updating VO loses roughly four-fifths. Targeted module reset-and-recovery experiments explain this asymmetry: plasticity tracks which module is best suited to a task's primary computation or storage, whereas stability depends on all ways the existing capability relies on that module, including access to information stored in other modules. Experiments on a realistic model and tasks broadly echo these asymmetric dynamics. Overall, our work shows that plasticity and stability depend on different aspects of a module's functional role, offering a task-dependent, module-level perspective on continual learning.

## JEPA as a Transferable Optimizer: Learning and Understanding Black-Box Optimization from Reward-Agnostic Pretraining

DevFeed: [JEPA as a Transferable Optimizer: Learning and Understanding Black-Box Optimization from Reward-Agnostic Pretraining](<https://devfeed.tech/articles/jepa-as-a-transferable-optimizer-learning-and-understanding-black-box-optimization-from-reward-agnostic-pretraining-142678.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23608>)

Author: Mayukh Debnath, Manan Roy Choudhury

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Self-supervised learning](<https://devfeed.tech/topics/self-supervised-learning.md>), [few-shot](<https://devfeed.tech/topics/few-shot.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [history](<https://devfeed.tech/tags/history.md>), [information](<https://devfeed.tech/tags/information.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [policy](<https://devfeed.tech/tags/policy.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [pretraining](<https://devfeed.tech/tags/pretraining.md>), [random](<https://devfeed.tech/tags/random.md>), [reward](<https://devfeed.tech/tags/reward.md>)

### AI overview

JEPA-PS learns to propose black-box optimization candidates from short evaluation histories, predicting the next point in a trajectory rather than rewards or gradients. Pretrained on random rollouts with parameter vectors and scalar returns, it proposes candidates in one forward pass without task descriptions, reward shaping, or test-time gradients. Across 15 control environments and 12 optimizers, it was competitive with tuned classical baselines, transferred zero-shot to environments excluded from pretraining, and beat an LLM optimizer by more than 2x on Swimmer at 71x lower per-step cost. Its useful history varied by landscape: smooth tasks benefited from the full trajectory, while on rugged tasks the best-seen point carried nearly all the signal. Evidence that this pattern relates to fitness-distance correlation is weak and was not corroborated by a separate reward-ordering probe. Decoder ablations also found every trained decoder variant worse than the default frozen representation.

### Source excerpt

arXiv:2611.23608v1 Announce Type: cross Abstract: Black-box optimization evaluates candidates through simulations, rollouts, or experiments, often without gradients. Learned optimizers are usually judged by whether they outperform existing methods. We ask a complementary question: what information does a learned optimizer use from its search history? We introduce JEPA-PS (JEPA Policy Search), based on a Joint-Embedding Predictive Architecture. It is pretrained on random rollouts using parameter vectors and scalar returns. Its latent-prediction objective predicts the next point in an optimization trajectory rather than its reward or gradient, allowing us to examine what was tried and how well it performed. At test time, JEPA-PS reads a short evaluation history and proposes a candidate in one forward pass, without task descriptions, reward shaping, or test-time gradients. We evaluate it on 15 control environments against 12 optimizers, including evolutionary methods, Bayesian optimization, CMA-ES, and an LLM optimizer. JEPA-PS is competitive with tuned classical baselines and outperforms the LLM optimizer by more than 2x on Swimmer at 71x lower per-step cost. It also transfers zero-shot to environments excluded from pretraining. Null-model comparisons and decoder-training ablations indicate that its latent representation carries information beyond simple history statistics; every trained decoder variant performs worse than the default frozen representation. Our main finding is that the history JEPA-PS uses depends on the task. On smooth landscapes it uses the full trajectory, whereas on rugged landscapes the best-seen point carries nearly all the signal. This memory requirement has a weak positive correlation with fitness-distance correlation (FDC; r=0.112, n=7), while an independent reward-ordering sensitivity probe does not corroborate the relationship (r=-0.024, n=7). Code: https://github.com/mdebnath22/jepa-props.

## Operational Proto-Introspection in Looped Language Models: Reading Correctness Before the Answer

DevFeed: [Operational Proto-Introspection in Looped Language Models: Reading Correctness Before the Answer](<https://devfeed.tech/articles/operational-proto-introspection-in-looped-language-models-reading-correctness-before-the-answer-142543.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.18553>)

Author: Jan Kirin

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [generate](<https://devfeed.tech/tags/generate.md>), [hidden-states](<https://devfeed.tech/tags/hidden-states.md>), [introspection](<https://devfeed.tech/tags/introspection.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [layers](<https://devfeed.tech/tags/layers.md>), [logic](<https://devfeed.tech/tags/logic.md>), [loops](<https://devfeed.tech/tags/loops.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [signal](<https://devfeed.tech/tags/signal.md>)

### AI overview

In Ouro-RLTT, probes of hidden states from the first 48 reasoning tokens predicted correctness on a synthetic logic task (AUROC 0.758 versus 0.515 for token confidence), including ranking the correct answer higher in 74% of same-question pairs. However, question features plus token confidence performed about as well on a larger set, and the within-question result was unresolved on GSM8K and a second checkpoint. For finished candidate answers, the signal strengthened across loops and was largely shared between them. It improved abstention and answer selection over chance, but surface features performed about as well; once sample agreement was known, the probe added no measurable value. Attempts to use the signal to improve generation did not work. The findings concern what probes can read; they do not show the model uses this information itself.

### Source excerpt

arXiv:2607.18553v4 Announce Type: replace-cross Abstract: Can one tell, while a language model reasons, whether its answer will be right? Looped models are a good place to ask: they run the same layers several times per token, leaving one hidden state per loop for a probe to read. I probe Ouro-RLTT, a 2.6B-parameter looped model, for what these states reveal about how its reasoning will end. I call such information operational proto-introspection: operational because it is defined by what a probe can read, and proto because nothing here shows that the model itself uses it. In part, yes. Reading the first 48 tokens of reasoning, from answers that state no answer yet, a probe on the hidden states reaches an AUROC of 0.758 on a synthetic logic task, against 0.515 for token confidence. Question difficulty is a rival explanation: on 1,164 further questions, token confidence plus simple features of the question predict about as well. It is not only difficulty: between two answers to the same question, the probe scores the correct one higher 74% of the time. A fine-tuned language model reading the same tokens does about as well. On GSM8K and on a second checkpoint this within-question test is unresolved. A probe that ranks finished candidate answers is weak at layer 8 on the first loop, much better on the second, and as accurate as the last layers by the third, on four Ouro checkpoints. The direction it reads is largely shared across loops: a probe trained on the last loop reads the first nearly as well as a probe trained there. The signal helps decide what to do with finished outputs. It improves abstention over surface features and picks correct answers among candidates better than chance, though surface features pick about as well. Once several samples are finished, a baseline that knows their agreement leaves the answer-free probe nothing measurable to add. None of my attempts to use it to make the model generate better answers worked.

## Beyond the Sink Token: Sink-Guided Key Editing in Vision-Language Models

DevFeed: [Beyond the Sink Token: Sink-Guided Key Editing in Vision-Language Models](<https://devfeed.tech/articles/beyond-the-sink-token-sink-guided-key-editing-in-vision-language-models-142667.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23516>)

Author: Hui Zhu, Saleh Zare Zade, Rafi Ibn Sultan, Xiangyu Zhou, Dongxiao Zhu

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [vlm](<https://devfeed.tech/topics/vlm.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [enhancing AI model accuracy](<https://devfeed.tech/topics/enhancing-ai-model-accuracy.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [attention-steering](<https://devfeed.tech/tags/attention-steering.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [boosting](<https://devfeed.tech/tags/boosting.md>), [components](<https://devfeed.tech/tags/components.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [gemma-3](<https://devfeed.tech/tags/gemma-3.md>), [generation](<https://devfeed.tech/tags/generation.md>), [image](<https://devfeed.tech/tags/image.md>), [keys](<https://devfeed.tech/tags/keys.md>), [layers](<https://devfeed.tech/tags/layers.md>), [llava](<https://devfeed.tech/tags/llava.md>), [magnitude](<https://devfeed.tech/tags/magnitude.md>), [operations](<https://devfeed.tech/tags/operations.md>), [points](<https://devfeed.tech/tags/points.md>), [position](<https://devfeed.tech/tags/position.md>), [question](<https://devfeed.tech/tags/question.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reference](<https://devfeed.tech/tags/reference.md>), [steering](<https://devfeed.tech/tags/steering.md>), [vision-language-models](<https://devfeed.tech/tags/vision-language-models.md>)

### AI overview

Across four probed vision-language models, increasing positive sink-aligned key components raised attention to image and question tokens on average, while increasing negative components lowered it; the effects persisted in later layers. Using this relationship, the authors propose SHRINK, a training-free method that partially reduces sink-aligned components across visible keys during generation. On nine benchmarks with sampled reasoning, it raised mean accuracy by 5.1 percentage points for LLaVA-OneVision-7B and 2.7 points for Qwen2.5-VL-3B, with smaller gains for Qwen2.5-VL-7B and Gemma-3-4B. Rescaling these key components also replaced attention operations in GIFT and ACG with comparable accuracy on hallucination benchmarks.

### Source excerpt

arXiv:2611.23516v1 Announce Type: cross Abstract: In vision-language models (VLMs), a few sink tokens attract a disproportionate share of attention. While previous studies have examined why these tokens attract attention, we investigate how the sink-key direction affects attention to image and question tokens. Across four probed VLMs, boosting the magnitude of positive sink-aligned components increases attention on average, whereas boosting negative components decreases it; these responses persist through later layers. This identifies the native sink key as a reference for controlling attention beyond its own position. Motivated by this finding, we propose SHRINK, a training-free method that partially reduces sink-aligned components across visible keys during generation. On nine benchmarks under sampled reasoning, it improves mean accuracy by 5.1 and 2.7 percentage points on LLaVA-OneVision-7B and Qwen2.5-VL-3B, with smaller gains on the other two main models, Qwen2.5-VL-7B and Gemma-3-4B. Beyond SHRINK, rescaling sink-aligned key components can replace the attention operations of two existing attention-steering methods, GIFT and ACG, with comparable accuracy on hallucination benchmarks.

## How Generative Models Generate: A Concise Guide to the Generative AI Landscape

DevFeed: [How Generative Models Generate: A Concise Guide to the Generative AI Landscape](<https://devfeed.tech/articles/how-generative-models-generate-a-concise-guide-to-the-generative-ai-landscape-142826.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24728>)

Author: Tianhua Chen

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Development](<https://devfeed.tech/topics/ai-development.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [generate](<https://devfeed.tech/tags/generate.md>), [generative](<https://devfeed.tech/tags/generative.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [generative-models](<https://devfeed.tech/tags/generative-models.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [models](<https://devfeed.tech/tags/models.md>)

### AI overview

The guide organizes generative models by how they produce observations: direct transformation or decoding, sequential conditional construction, and iterative updating or transport. It then connects these routes to the modeling and learning principles behind them, introducing representative approaches through their key components, objectives, and sampling procedures. The abstract also highlights overlaps and combinations among architectures and generation mechanisms. This is presented as a compact conceptual starting point for understanding unfamiliar methods and navigating the broader landscape; the supplied text does not provide detailed comparisons of particular models.

### Source excerpt

arXiv:2611.24728v1 Announce Type: cross Abstract: The rapidly expanding generative AI landscape brings together methods that differ in architecture, learning objective, mathematical representation and sampling procedure. A concise, selectively organised technical guide can provide an accessible starting point for developing a coherent understanding of this increasingly complex landscape. This article takes a generation-centred approach, connecting how models produce new observations with the modelling and learning principles that make generation possible. We develop a two-level perspective: first, distinguishing three recurring generation routes-direct transformation or decoding, sequential conditional construction, and iterative updating or transport; and second, examining the modelling and learning principles underlying these routes. Representative approaches are introduced through their key components, learning objectives and sampling procedures. We then examine their broader connections, including how different architectures and generation mechanisms can overlap or be combined. By bringing together essential technical explanations within a compact conceptual framework, the article aims to help readers develop a working understanding of unfamiliar approaches, recognise connections among models they have encountered, and navigate different approaches within the broader generative AI landscape.

## Does the Action Matter? Probing Action-Conditioned Semantic Forecasting as a World Model for Autonomous Driving

DevFeed: [Does the Action Matter? Probing Action-Conditioned Semantic Forecasting as a World Model for Autonomous Driving](<https://devfeed.tech/articles/does-the-action-matter-probing-action-conditioned-semantic-forecasting-as-a-world-model-for-autonomous-driving-142837.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24840>)

Author: Binay Kumar Singh, Niels Da Vitoria Lobo

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [autonomous vehicles](<https://devfeed.tech/topics/autonomous-vehicles.md>), [World models](<https://devfeed.tech/topics/world-models.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [autonomous-driving](<https://devfeed.tech/tags/autonomous-driving.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [driving](<https://devfeed.tech/tags/driving.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [planning](<https://devfeed.tech/tags/planning.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [predictions](<https://devfeed.tech/tags/predictions.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [world-model](<https://devfeed.tech/tags/world-model.md>)

### AI overview

The study tests whether autonomous-driving semantic forecasts actually depend on the ego action, using test-time action interventions scored with segmentation metrics. The executed action improves mid-term mIoU by at most 2.5 points on Cityscapes and 0.6 on BDD100K; supplying a wrong action lowers it by up to 23.2 and 7.0 points, respectively, below a model without action input. On real data, feature-wise action conditioning is 1.6 times more action-dependent than input-level conditioning at similar accuracy; in a synthetic setting where history does not determine action, their dependence is equal. The findings indicate that accuracy gains show action use but do not establish how strongly forecasts rely on actions.

### Source excerpt

arXiv:2611.24840v1 Announce Type: cross Abstract: World models are increasingly used in autonomous driving to predict how a scene evolves under the ego vehicle's actions, supporting counterfactual reasoning and model-based planning. However, existing evaluations focus on prediction fidelity and planning performance, and rarely verify that the predictions depend on the conditioning action. In this paper, we study action dependence in action-conditioned semantic forecasting, which predicts future semantic segmentation from past segmentations and the ego action. We compare a forecaster with feature-wise and input-level action conditioning against an identical model without action input, and measure action dependence with test-time action interventions scored by standard segmentation metrics. The evaluation is validated in a synthetic environment with known action-dependent dynamics and applied to Cityscapes and BDD100K, both within each dataset and across the two vehicle platforms. Experiments show that the executed action improves mid-term mIoU by at most 2.5 points on Cityscapes and 0.6 points on BDD100K, whereas a wrong action reduces it by up to 23.2 and 7.0 points, below the model without action input. On real data, feature-wise conditioning depends 1.6 times more on the action than input-level conditioning at similar accuracy, whereas both depend on it equally in the synthetic environment, where the history does not determine the action. These results show that an accuracy gain indicates that a driving world model uses its action, but not how strongly its forecasts depend on it. Code is available at https://github.com/binayksingh/does-action-matter.

## JEPA for astronomy: multi-survey spectral alignment puts stellar parameters on the leading principal axes

DevFeed: [JEPA for astronomy: multi-survey spectral alignment puts stellar parameters on the leading principal axes](<https://devfeed.tech/articles/jepa-for-astronomy-multi-survey-spectral-alignment-puts-stellar-parameters-on-the-leading-principal-axes-143400.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23492>)

Author: Xiaosheng Zhao, Rosemary F. G. Wyse, Yuan-Sen Ting, Alexander S. Szalay

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [alpha](<https://devfeed.tech/tags/alpha.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [astro-ph-ga](<https://devfeed.tech/tags/astro-ph-ga.md>), [astro-ph-im](<https://devfeed.tech/tags/astro-ph-im.md>), [astro-ph-sr](<https://devfeed.tech/tags/astro-ph-sr.md>), [astronomy](<https://devfeed.tech/tags/astronomy.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [gaia](<https://devfeed.tech/tags/gaia.md>), [legibility](<https://devfeed.tech/tags/legibility.md>), [linear](<https://devfeed.tech/tags/linear.md>), [performance](<https://devfeed.tech/tags/performance.md>), [physics](<https://devfeed.tech/tags/physics.md>), [stellar](<https://devfeed.tech/tags/stellar.md>), [survey](<https://devfeed.tech/tags/survey.md>)

### AI overview

The study finds that the alignment objective determines how easily stellar properties can be read from embeddings of spectra collected by different surveys. With the cross-survey architecture held fixed, JEPA--constraining each survey's embedding distribution without pushing individual stars apart--makes stellar labels more linearly recoverable than a contrastive loss. In the Gaia XP embedding, the first two principal components recover a Kiel-diagram-like plane, the third tracks [Fe/H], and [alpha/Fe] varies coherently across the plane, separating chemically defined populations. Adding LAMOST LRS, DESI, and APOGEE further improves performance; predicted abundances for 8 million XP giants reproduce the Milky Way's main [alpha/Fe]-[Fe/H] structure. These findings are reported in the supplied abstract.

### Source excerpt

arXiv:2611.23492v1 Announce Type: cross Abstract: Spectra from different surveys carry different systematics and are not calibrated against one another. Multi-survey foundation models for stellar spectra are then built by aligning embeddings of the same star observed across surveys. Most in astronomy follow the recipes established for language and vision, aligning with a contrastive loss that separates every star from the others. However, this does not ensure that the stellar physics is encoded in a form that a simple read-out can recover. Holding the cross-survey architecture fixed and varying only the alignment loss, we measure what an objective encodes from the principal-component geometry of the Gaia BP/RP (XP) spectra and from a linear probe of the stellar parameters. In a properly aligned embedding, the leading two principal components form a plane that recovers an analog of the Kiel diagram, the third component tracks [Fe/H], and [alpha/Fe] varies coherently across the plane, such that the chemically defined Galactic disks and accreted populations occupy distinct loci. We show that legibility, the ease of reading the information out, is highest for a training objective that never pushes stars apart, and instead constrains only the overall distribution of each survey's embeddings (JEPA). JEPA also exceeds the contrastive alternative at the same architecture, and the performance improves further as more surveys (Gaia XP+LAMOST LRS+DESI+APOGEE) are included. Predicted abundances for 8.0 million XP giants reproduce the main [alpha/Fe]-[Fe/H] structure of the Milky Way. These results indicate that the choice of alignment objective, not alignment alone, determines whether stellar labels are linearly recoverable.

## The Elephant in the Question: Unified Models Picture the Concepts Behind Their Answers

DevFeed: [The Elephant in the Question: Unified Models Picture the Concepts Behind Their Answers](<https://devfeed.tech/articles/the-elephant-in-the-question-unified-models-picture-the-concepts-behind-their-answers-143563.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24793>)

Author: Tobias Braun, Marcus Rohrbach, Lorenzo Cavallaro

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [ai-image-generator](<https://devfeed.tech/topics/ai-image-generator.md>), [Chain-of-thought](<https://devfeed.tech/topics/chain-of-thought.md>), [Image](<https://devfeed.tech/topics/image.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [concepts](<https://devfeed.tech/tags/concepts.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [generation](<https://devfeed.tech/tags/generation.md>), [generators](<https://devfeed.tech/tags/generators.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [hidden-states](<https://devfeed.tech/tags/hidden-states.md>), [image](<https://devfeed.tech/tags/image.md>), [images](<https://devfeed.tech/tags/images.md>), [inspect](<https://devfeed.tech/tags/inspect.md>), [intermediate](<https://devfeed.tech/tags/intermediate.md>), [intermediate-representations](<https://devfeed.tech/tags/intermediate-representations.md>), [lens](<https://devfeed.tech/tags/lens.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [picture](<https://devfeed.tech/tags/picture.md>), [question](<https://devfeed.tech/tags/question.md>), [unified-models](<https://devfeed.tech/tags/unified-models.md>)

### AI overview

V-lens visualizes a chosen token's hidden state in unified multimodal models by using their image generators, without training a decoder. Across four diffusion-based and two autoregressive models, the visualizations expose intermediate concepts linking questions to answers, including concepts never stated explicitly; controlled tests examine how they vary with context, token position, and depth. The paper also introduces lens tuning, which supervises selected hidden states with pictures through V-lens. For BAGEL, SenseNova-U1, and HunyuanImage-3.0, correct visualizations of pronoun referents rise from 2-11% to 86-94%. The abstract reports that this visual supervision also improves text answers without directly supervising answer tokens.

### Source excerpt

arXiv:2611.24793v1 Announce Type: cross Abstract: Visualizing intermediate representations could help us inspect the reasoning steps underlying a model output. Unified multimodal models offer this opportunity by combining understanding and generation in one architecture. Their image generators can access representations formed while processing text. We introduce the V-lens, which uses this connection to visualize a chosen token's hidden state without training a decoder. Across four diffusion-based and two purely autoregressive unified models, the resulting images reveal key intermediate concepts that connect a question to its answer even when the concepts are never named explicitly. Controlled tests show how this content changes with context, token position, and depth. We then introduce lens tuning, which supervises selected hidden states with pictures through the V-lens. Across BAGEL, SenseNova-U1, and HunyuanImage-3.0, the rate of correct visualizations of pronoun referents rises from 2-11% to 86-94%. Surprisingly, such visual supervision also improves text answers without supervising the answer tokens directly. Code is available at https://github.com/multimodal-ai-lab/V-lens.

## The multiplicative Lipschitz bound as an inductive bias

DevFeed: [The multiplicative Lipschitz bound as an inductive bias](<https://devfeed.tech/articles/the-multiplicative-lipschitz-bound-as-an-inductive-bias-143247.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.28113>)

Author: Simon Kuang, Yuezhu Xu, S. Sivaranjani, Xinfan Lin

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [activation-function](<https://devfeed.tech/tags/activation-function.md>), [autoencoders](<https://devfeed.tech/tags/autoencoders.md>), [classification](<https://devfeed.tech/tags/classification.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [inductive-bias](<https://devfeed.tech/tags/inductive-bias.md>), [learning](<https://devfeed.tech/tags/learning.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>)

### AI overview

The paper finds that multiplying layerwise Lipschitz constants--a bound often considered too loose--can work effectively as an inductive bias for Lipschitz-constrained learning. Combined with a quadrature nonlinearity built from concatenated cosine and sine activations, it achieves lower cross-entropy than Sandwich, Almost-Orthogonal Layers, Convex Potential Layers, and SDP-based Lipschitz Layers over most tested classification bounds. For autoencoders constrained to be nonexpansive in a learned norm, it reports the lowest pixel reconstruction error at every tested bottleneck width, improving on PCA by 12-24%. The authors provide a semi-rigorous analysis of why the bound may be loose.

### Source excerpt

arXiv:2603.28113v3 Announce Type: replace Abstract: We study the problem of training a neural network to minimize a loss function given a Lipschitz constraint. While the most obvious way to estimate the Lipschitz constant of a neural network is to multiply the Lipschitz constants of its layers, this bound, which we term the _trivial bound_, is often considered impractically loose. We show that, on the contrary, it can be more effective than state-of-the-art methods (Sandwich, Almost-Orthogonal Layers, Convex Potential Layers, and SDP-based Lipschitz Layers) for Lipschitz-constrained learning. Our method combines the trivial bound with a novel activation function (the quadrature nonlinearity) to achieve lower cross-entropy over most of the tested classification bounds. For autoencoders constrained to be nonexpansive in a learned norm, it achieves the lowest pixel reconstruction error at every tested bottleneck width, improving on PCA by 12--24%. Our architecture, consisting of concatenated cosine and sine activations, is supported by a semi-rigorous theoretical analysis of Lipschitz bound looseness that may prove valuable in its own right.

## Design Once, Deploy at Scale: Building a Template-Driven ML Ecosystem

DevFeed: [Design Once, Deploy at Scale: Building a Template-Driven ML Ecosystem](<https://devfeed.tech/articles/design-once-deploy-at-scale-building-a-template-driven-ml-ecosystem-141156.md>)

Original publisher: [Read original article](<https://engineering.fb.com/2026/11/18/ml-applications/design-once-deploy-at-scale-template-driven-ml-ecosystem/>)

Author: Anirudh Agrawal; Djordje Gligorijevic; Yao Xuan; Peggy Yao; Jiang Liu; John Martabano Landy; Rahul Suresh

Published: 2026-11-18T16:59:38Z

Content type: article

Language: en

Sources: [Engineering at Meta](<https://devfeed.tech/sources/engineering-at-meta.md>), [Meta ML Applications](<https://devfeed.tech/sources/meta-ml-applications.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Template](<https://devfeed.tech/topics/template.md>)

Tags: [all](<https://devfeed.tech/tags/all.md>), [approach](<https://devfeed.tech/tags/approach.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [building](<https://devfeed.tech/tags/building.md>), [but](<https://devfeed.tech/tags/but.md>), [deploy](<https://devfeed.tech/tags/deploy.md>), [design](<https://devfeed.tech/tags/design.md>), [devinfra](<https://devfeed.tech/tags/devinfra.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [ecosystems](<https://devfeed.tech/tags/ecosystems.md>), [efficiently](<https://devfeed.tech/tags/efficiently.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [inventing](<https://devfeed.tech/tags/inventing.md>), [large](<https://devfeed.tech/tags/large.md>), [meta](<https://devfeed.tech/tags/meta.md>), [ml](<https://devfeed.tech/tags/ml.md>), [ml-applications](<https://devfeed.tech/tags/ml-applications.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [more](<https://devfeed.tech/tags/more.md>), [scale](<https://devfeed.tech/tags/scale.md>), [template](<https://devfeed.tech/tags/template.md>)

### AI overview

Meta describes Standard Model Templates (SMTs) as a way to reduce fragmentation across its ads ranking and retrieval models: shared components define reusable model patterns, so teams can develop techniques at the template level and propagate them to groups of models. Multi-model optimization evaluates candidate techniques on representative models for aggregate gains and regression rates, then checks transfer on held-out models before integration. Meta reports that over two years the approach reduced per-model engineering time by 92%, shipped more than 5,000 technique-model combinations and 80 techniques, improved offline quality by 0.63% on average at neutral serving capacity, and delivered 0.86% cumulative online lift. The text says templates also provide a structured starting point for agents, while noting that coverage, reliability, explainability, and preserving useful flexibility remain ongoing challenges.

### Source excerpt

As ML ecosystems grow to a large number of production models, the bottleneck is no longer inventing new techniques to improve these models, but rather shipping them reliably and efficiently across a fragmented model space, creating a rising tide that lifts all the boats. At Meta, we built a template-driven approach that standardizes model design [...] Read More... The post Design Once, Deploy at Scale: Building a Template-Driven ML Ecosystem appeared first on Engineering at Meta.

## AI-Powered Sashiko Introduces Experimental Patch Series Design Assessment

DevFeed: [AI-Powered Sashiko Introduces Experimental Patch Series Design Assessment](<https://devfeed.tech/articles/ai-powered-sashiko-introduces-experimental-patch-series-design-assessment-140542.md>)

Original publisher: [Read original article](<https://www.phoronix.com/news/Sashiko-0.9.4-Released>)

Author: Michael Larabel

Published: 2026-11-18T11:13:06Z

Content type: release

Language: en

Sources: [Phoronix](<https://devfeed.tech/sources/phoronix.md>)

Topics: [ai code review](<https://devfeed.tech/topics/ai-code-review.md>), [Code review](<https://devfeed.tech/topics/code-review.md>), [Linux](<https://devfeed.tech/topics/linux.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [code-review](<https://devfeed.tech/tags/code-review.md>), [design](<https://devfeed.tech/tags/design.md>), [desktop-linux](<https://devfeed.tech/tags/desktop-linux.md>), [experimental](<https://devfeed.tech/tags/experimental.md>), [linux-benchmarking](<https://devfeed.tech/tags/linux-benchmarking.md>), [linux-hardware-benchmarks](<https://devfeed.tech/tags/linux-hardware-benchmarks.md>), [linux-hardware-reviews](<https://devfeed.tech/tags/linux-hardware-reviews.md>), [linux-how-to](<https://devfeed.tech/tags/linux-how-to.md>), [linux-performance](<https://devfeed.tech/tags/linux-performance.md>), [linux-server-benchmarks](<https://devfeed.tech/tags/linux-server-benchmarks.md>), [open-source-graphics](<https://devfeed.tech/tags/open-source-graphics.md>), [patch](<https://devfeed.tech/tags/patch.md>), [phoronix](<https://devfeed.tech/tags/phoronix.md>), [phoronix-test-suite](<https://devfeed.tech/tags/phoronix-test-suite.md>), [review](<https://devfeed.tech/tags/review.md>), [series](<https://devfeed.tech/tags/series.md>), [ubuntu-benchmarks](<https://devfeed.tech/tags/ubuntu-benchmarks.md>), [ubuntu-hardware](<https://devfeed.tech/tags/ubuntu-hardware.md>)

### AI overview

Sashiko 0.9.4 adds an experimental `sashiko review --design` option for local reviews. Its four-stage pipeline assesses a patch series as a whole, complementing Sashiko's existing reviews of individual patches for correctness and other code shortcomings. The release also adds first-class code review support for e2fsprogs, the user-space utilities for EXT4 and earlier filesystems. The article describes the series assessment as experimental and does not report how it performs in practice.

### Source excerpt

When it comes to leveraging AI for Linux kernel development, one of the most widely-used solutions at the moment is Sashiko that has evolved nicely over the past year for agentic AI code review. This AI-driven screening of generated code posted to the Linux kernel mailing list constantly identifies bugs and other code defects in a quick and straight-forward manner. With being developed and maintained by Google engineers, it uses Gemini by default but is LLM-agnostic overall. Released on Tuesday was Sashiko v0.9.4 as the next step forward for this Linux kernel helper...

## When the tail is earned by its companions: measuring label dependence in long-tailed multi-label classification

DevFeed: [When the tail is earned by its companions: measuring label dependence in long-tailed multi-label classification](<https://devfeed.tech/articles/when-the-tail-is-earned-by-its-companions-measuring-label-dependence-in-long-tailed-multi-label-classification-141814.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22859>)

Author: Christoffer L\\"offler

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Inference](<https://devfeed.tech/topics/inference.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [measuring](<https://devfeed.tech/tags/measuring.md>), [multi-label-classification](<https://devfeed.tech/tags/multi-label-classification.md>)

### AI overview

The study finds that long-tailed multi-label classifiers can appear to recover rare labels by relying on their frequent co-occurring labels. It proposes a within-label measure of this dependence, a rule that counts reduced dependence only when recall without companions improves, and a training-label statistic that bounds how much reported recall may exceed companion-free recall before training. Across 23 corpora, seven objectives, and five families of label-correlation mechanisms, companion-free recall was lower on all six corpora where the measure was defined; up to 18.4% of credited tail true positives would disappear at that rate, within the proposed bound on every corpus. No long-tail objective reduced dependence, but architecture affected it; models scoring tail labels from other labels' predictions were strongly dependent.

### Source excerpt

arXiv:2611.22859v1 Announce Type: new Abstract: A rare label seldom occurs alone, and a classifier can recover it from the frequent labels it co-occurs with, its companions, without having learned the label itself. Existing diagnostics of this need a trained model: none says before training whether a corpus admits the failure, or what the standard aggregate hides. Macro-F1, like any measure built from per-label confusion counts, is unchanged by how a label's recall divides between occurrences with and without a companion. We introduce a within-label estimand of that division, a reading rule under which a fall in dependence counts only if companion-free recall rises, and a statistic of training labels that bounds, before training, how far reported recall can exceed companion-free recall. Across 23 corpora, 7 training objectives and five families of label-correlation mechanisms, companion-free recall falls short on all six corpora where the estimand is defined. Up to 18.4% of the tail true positives macro-F1 credits would vanish at the companion-free rate, within the training-label bound on every corpus. No long-tail objective reduces the dependence, but architecture moves it. Models whose tail score is computed from other labels' predictions are strongly dependent.

[Next page](<https://devfeed.tech/topics/model-architecture.md?cursor=WyIyMDI2LTExLTE4VDA1OjAwOjAwKzAwOjAwIiwgImYzMjMxNzVmLWM5NjgtNDAwZC04MTg5LTk5ZWQ4NjkyNTA0ZSJd>)