# Transformer

A neural-network architecture based solely on attention mechanisms, dispensing with recurrence and convolutions.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Attention Head Specialization: Phase Transitions in Sample Complexity

DevFeed: [Attention Head Specialization: Phase Transitions in Sample Complexity](<https://devfeed.tech/articles/attention-head-specialization-phase-transitions-in-sample-complexity-145153.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26228>)

Author: L\\'eo Catteau, O. Duranthon, Lenka Zdeborov\\'a

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [bert](<https://devfeed.tech/tags/bert.md>), [bias](<https://devfeed.tech/tags/bias.md>), [cascade](<https://devfeed.tech/tags/cascade.md>), [changes](<https://devfeed.tech/tags/changes.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [cond-mat-dis-nn](<https://devfeed.tech/tags/cond-mat-dis-nn.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-structure](<https://devfeed.tech/tags/data-structure.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [emergence](<https://devfeed.tech/tags/emergence.md>), [error](<https://devfeed.tech/tags/error.md>), [features](<https://devfeed.tech/tags/features.md>), [function](<https://devfeed.tech/tags/function.md>), [learning](<https://devfeed.tech/tags/learning.md>), [model](<https://devfeed.tech/tags/model.md>), [multi-head](<https://devfeed.tech/tags/multi-head.md>), [online](<https://devfeed.tech/tags/online.md>), [online-learning](<https://devfeed.tech/tags/online-learning.md>), [sample](<https://devfeed.tech/tags/sample.md>), [specialization](<https://devfeed.tech/tags/specialization.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

The excerpt reports that BERT heads specialized to syntactic relations emerge as training dataset size increases. In a solvable high-dimensional multi-head attention model, specialization is a phase transition in sample complexity, marked by non-analytic changes in test error and other observables. Depending on data structure and regularization, heads may differentiate without conveying useful information, or specialize through successive transitions that recover progressively weaker latent features. The authors also relate these statistical transitions to stagewise specialization in online learning, separating statistical effects from training dynamics.

### Source excerpt

arXiv:2611.26228v1 Announce Type: new Abstract: Attention heads specialize into distinct functions during training, often emerging in stages together with sharp changes in model behavior. While this phenomenon has mostly been studied as a function of training time, we ask how head specialization depends on the amount of available training data. Empirically, we find that heads specialized to syntactic relations in BERT emerge as the dataset size increases. We then study this phenomenon in a solvable high-dimensional model of multi-head attention and establish that, in this model, head specialization is a genuine phase transition in sample complexity, corresponding to a non-analyticity in the test error and other observables. Depending on data structure and regularization, specialization can follow uninformative head differentiation or occur through a cascade of phase transitions that progressively recover weaker latent features, providing a statistical form of simplicity bias. Finally, we relate these statistical transitions to the stagewise emergence of specialized heads during online learning, disentangling statistical from dynamical origins of head specialization.

## FEA solver integration of a composite World Model with a Spiking Neural Network memory Block for computationally efficient modelling of defects and variabilities

DevFeed: [FEA solver integration of a composite World Model with a Spiking Neural Network memory Block for computationally efficient modelling of defects and variabilities](<https://devfeed.tech/articles/fea-solver-integration-of-a-composite-world-model-with-a-spiking-neural-network-memory-block-for-computationally-efficient-modelling-of-defects-and-variabilities-145179.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26419>)

Author: Bassam El Said, Adam Sobey

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [defects](<https://devfeed.tech/tags/defects.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [encoder](<https://devfeed.tech/tags/encoder.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [inference](<https://devfeed.tech/tags/inference.md>), [integration](<https://devfeed.tech/tags/integration.md>), [material](<https://devfeed.tech/tags/material.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [modelling](<https://devfeed.tech/tags/modelling.md>), [models](<https://devfeed.tech/tags/models.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [snn](<https://devfeed.tech/tags/snn.md>), [state](<https://devfeed.tech/tags/state.md>), [structure](<https://devfeed.tech/tags/structure.md>), [world-model](<https://devfeed.tech/tags/world-model.md>)

### AI overview

The proposed Composites World Model aims to make structural-scale composite simulation more computationally practical by replacing costly material-scale models with a surrogate that can represent local architectures and defects, track irreversible state, and run recursively. It combines a material encoder, a Spiking Neural Network memory block for long-term material state, and transformer blocks for short-term applied strain. Training proceeds through architecture encoding, short-window base training, and recursive fine-tuning with scheduled sampling to limit cumulative error drift. At inference, the model is disassembled and coupled to a nonlinear FE solver. Benchmarks on IM7/8552 laminates under tension, compression, and transverse shear, including pristine and wrinkled cases, report prediction errors of 0.4%-4.8%; the abstract attributes successful tracking of material state to the SNN memory block.

### Source excerpt

arXiv:2611.26419v1 Announce Type: new Abstract: Each fibre-reinforced composite structure has a unique material architecture. The manufacturing methods used and the presence of defects lead to local variations which require material-scale models that can resolve the response in real time at every location in the structure. Concurrent multi-scale schemes provide this capability, but at a prohibitive computational cost for large structures as the number of integration points and time increments scales with size. Surrogates have been studied extensively as replacements for material-scale models to mitigate these computational limitations. A surrogate must represent an arbitrary local architecture including defects, track the irreversible material state over an unknown number of increments, remain stable when driven recursively by its own predictions, and fit within the memory available at every integration point. Here, a Composites World Model based on a Multi-Modal Spiking Transformer architecture is designed to meet these requirements. The proposed architecture combines a material encoder, a Spiking Neural Network memory block that tracks the long-term internal state of the material, and transformer blocks that learn the short-term applied strain state. The system is trained in three stages: material architecture encoding, base training on short time windows, and recursive fine-tuning with scheduled sampling to suppress cumulative error drift. The trained model is disassembled at inference and coupled to a non-linear FE solver, enabling structural scale simulation of composites containing embedded defects. Benchmarks against experimental results for IM7/8552 laminates under tension, compression and transverse shear, for pristine and wrinkled composites, show that the SNN memory block successfully tracks the material internal state and results in prediction errors between 0.4% and 4.8%.

## DTS: Delta-Based Tile Scoring for Efficient Long-Context Sparse Attention

DevFeed: [DTS: Delta-Based Tile Scoring for Efficient Long-Context Sparse Attention](<https://devfeed.tech/articles/dts-delta-based-tile-scoring-for-efficient-long-context-sparse-attention-145120.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25938>)

Author: Emre U\\u{g}ur, Jiawen Qi, Qinyu Chen

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [context window](<https://devfeed.tech/topics/context-window.md>), [PyTorch](<https://devfeed.tech/topics/pytorch.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [anchor](<https://devfeed.tech/tags/anchor.md>), [attention](<https://devfeed.tech/tags/attention.md>), [bottleneck](<https://devfeed.tech/tags/bottleneck.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [concurrently](<https://devfeed.tech/tags/concurrently.md>), [content](<https://devfeed.tech/tags/content.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [delta](<https://devfeed.tech/tags/delta.md>), [flashattention](<https://devfeed.tech/tags/flashattention.md>), [free](<https://devfeed.tech/tags/free.md>), [full](<https://devfeed.tech/tags/full.md>), [gpus](<https://devfeed.tech/tags/gpus.md>), [h100](<https://devfeed.tech/tags/h100.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large](<https://devfeed.tech/tags/large.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [sparse-attention](<https://devfeed.tech/tags/sparse-attention.md>)

### AI overview

DTS reduces the cost of scoring blocks for sparse long-context attention by choosing query and key anchors based on local vector changes: it uses fewer anchors in redundant regions and more in heterogeneous ones. Parallel delta packing speeds selection by scanning tile-aligned chunks concurrently; selected tiles are computed with the original query, key, and value vectors. The method is training-free. On LLaMA-3.1-8B-Instruct, average RULER and LongBench scores were within 0.10 percentage points of full attention, with RULER tested up to 128K tokens. At 128K, scoring-pipeline latency was 38% lower than XAttention; at 512K, prefill was 3.3x faster than a PyTorch full-attention baseline using FlashAttention-2 on four NVIDIA H100 GPUs.

### Source excerpt

arXiv:2611.25938v1 Announce Type: new Abstract: Large language models (LLMs) increasingly need to process long documents and codebases, making long-context attention a major prefill bottleneck. Block sparse attention reduces computation by skipping unimportant blocks of the attention matrix, but scoring blocks accurately can itself be expensive. We propose Delta-Based Tile Scoring (DTS), a training-free, content-adaptive tile scoring method for sparse block attention. DTS selects query and key anchors according to local vector changes, using fewer representatives in redundant regions while retaining more detail in heterogeneous regions. To accelerate anchor selection, parallel delta packing partitions the query and key sequences into tile-aligned chunks and performs bounded local scans over these chunks concurrently. The selected tiles are then computed using the original query, key, and value vectors. On LLaMA-3.1-8B-Instruct, DTS keeps average RULER and LongBench scores within 0.10 percentage points of full attention, with RULER evaluated up to 128K tokens. At 128K tokens, DTS reduces measured scoring-pipeline latency by 38% compared with XAttention. On four NVIDIA H100 GPUs, it achieves a 3.3$\times$ prefill speedup over a PyTorch full-attention baseline using FlashAttention-2 at 512K tokens.

## HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

DevFeed: [HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs](<https://devfeed.tech/articles/hisa-sparsifying-the-hidden-dimension-of-rope-attention-in-llms-145170.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26363>)

Author: Ambroise Borbely, Mathieu Salzmann, Olivier Verscheure

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [kernels](<https://devfeed.tech/topics/kernels.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [flashattention](<https://devfeed.tech/tags/flashattention.md>), [inference](<https://devfeed.tech/tags/inference.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [memory](<https://devfeed.tech/tags/memory.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rope](<https://devfeed.tech/tags/rope.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>)

### AI overview

HiSA makes attention sparse in the hidden feature dimension: it keeps query and key projections intact but computes only part of their dot product, using a static, parameter-free rule based on token positions and RoPE structure. The authors report that models trained from scratch with HiSA outperform those using the full dot product while computing about one third as many query-key products. At inference, they report faster decoding and lower memory use, and describe FlashAttention-3 kernels for both training and inference. They also report that dense attention learns to suppress the score components HiSA omits. These findings are from the supplied abstract.

### Source excerpt

arXiv:2611.26363v1 Announce Type: new Abstract: Attention in large language models (LLMs) scores every query against every key with a full dot product. We investigate whether this dot product must be computed completely for attention to be effective. We propose Hidden Sparse Attention (HiSA), which keeps the query and key projections intact but only computes a partial dot product, following a static, parameter-free rule set by the token positions: attention becomes sparse in the hidden (feature) dimension, an axis much less explored than token sparsity, rather than over tokens as in almost all sparse attention methods. Based on the structure of rotary position embeddings (RoPE), we decompose keys and queries into parts which can be omitted when computing attention. By training LLMs from scratch with HiSA, we find that this partial dot product does not hurt but improves performance compared with the complete product while computing only about a third of the query-key products. During inference HiSA improves decoding speed and shrinks memory footprint. We also extend HiSA to FlashAttention-3 kernels for training and inference. Finally, we motivate our approach by showing that dense attention learns to suppress the attention score parts that HiSA drops. Code is available at https://github.com/ambroise-b/hisa.

## Fast Gauss Sums via Flash Attention

DevFeed: [Fast Gauss Sums via Flash Attention](<https://devfeed.tech/articles/fast-gauss-sums-via-flash-attention-144922.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.04910>)

Author: Nicolaj Rux, Sebastian Neumayer

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [beats](<https://devfeed.tech/tags/beats.md>), [code](<https://devfeed.tech/tags/code.md>), [core](<https://devfeed.tech/tags/core.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-na](<https://devfeed.tech/tags/cs-na.md>), [custom](<https://devfeed.tech/tags/custom.md>), [dimension](<https://devfeed.tech/tags/dimension.md>), [fast](<https://devfeed.tech/tags/fast.md>), [flash](<https://devfeed.tech/tags/flash.md>), [fp16](<https://devfeed.tech/tags/fp16.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [input](<https://devfeed.tech/tags/input.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [linear](<https://devfeed.tech/tags/linear.md>), [math-na](<https://devfeed.tech/tags/math-na.md>), [memory](<https://devfeed.tech/tags/memory.md>), [memory-scaling](<https://devfeed.tech/tags/memory-scaling.md>)

### AI overview

The method computes Gauss kernel sums with arbitrary signed weights using FLASH attention: two small input augmentations convert its normalized softmax reduction into the unnormalized sum, without custom GPU code. For feature dimensions above 8 in fp16, it is reported to outperform compiled PyTorch in speed and memory, and PyKeOps in speed and accuracy at comparable or lower memory. Memory scaling remains linear.

### Source excerpt

arXiv:2609.04910v2 Announce Type: replace Abstract: Gauss kernel sums are the computational core of maximum mean discrepancies (MMDs), kernel gradient flows, Stein variational gradient descent (SVGD), and many other kernel methods. At the same time, softmax attention has received an extraordinary amount of hardware-aware code engineering, culminating in FLASH attention. We show that Gauss kernel sums with arbitrary, signed weights can be evaluated via FLASH attention: two small input augmentations turn the normalized softmax reduction into the unnormalized Gauss sum, without writing a single line of custom GPU code. For feature dimension $D>8$ in fp16, this approach beats compiled PyTorch code in speed and memory, and PyKeOps kernels in speed and accuracy at comparable or lower memory. Its memory scaling remains linear.

## Linearized 2-Simplicial Attention

DevFeed: [Linearized 2-Simplicial Attention](<https://devfeed.tech/articles/linearized-2-simplicial-attention-142552.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.09307>)

Author: Aritra Das, Dhruman Gupta, Debayan Gupta

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [CUDA Tile](<https://devfeed.tech/topics/cuda-tile.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cubic](<https://devfeed.tech/tags/cubic.md>), [features](<https://devfeed.tech/tags/features.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [kimi](<https://devfeed.tech/tags/kimi.md>), [perplexity](<https://devfeed.tech/tags/perplexity.md>), [softmax](<https://devfeed.tech/tags/softmax.md>), [training](<https://devfeed.tech/tags/training.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

The method reduces 2-simplicial attention's sequence-length cost from cubic to quadratic by rewriting its trilinear score as a composite query-key inner product, approximating the key sum with positive random features, and keeping previous keys in a fixed-size state. The sum over second keys remains exact across all previous tokens, unlike windowed 2-simplicial attention. Custom Triton kernels combine the method with Kimi Delta Attention in a model without softmax attention. In a 15B-token FineWeb-Edu comparison against a KDA hybrid, validation loss fell from 2.35 to 2.32 and LAMBADA perplexity from 40 to 38.

### Source excerpt

arXiv:2608.09307v2 Announce Type: replace Abstract: We present a linearized form of 2-simplicial attention by rewriting the trilinear score as an inner product between a composite query and a key, so that the sum over keys takes the same form as ordinary softmax attention. We then approximate this sum with positive random features and store all previous keys in a fixed-size state, while the sum over second keys stays exact over all previous tokens. This reduces the cost from cubic to quadratic in sequence length, and both keys range over all previous tokens, which windowed 2-simplicial attention does not allow. We implement it with custom Triton kernels and combine it with Kimi Delta Attention to build a model with no softmax attention at all. With 15B equal training tokens on FineWeb-Edu, this model lowers validation loss from 2.35 to 2.32 over a KDA hybrid and lowers LAMBADA perplexity from 40 to 38.

## Every Query Routes Itself: Accelerating Video Diffusion Transformers with Per-Query Routed Sparse Attention

DevFeed: [Every Query Routes Itself: Accelerating Video Diffusion Transformers with Per-Query Routed Sparse Attention](<https://devfeed.tech/articles/every-query-routes-itself-accelerating-video-diffusion-transformers-with-per-query-routed-sparse-attention-143528.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24479>)

Author: Zhen Huang, Shuwei Li, Ruizhe Yao, Xinrui Chen, Danyi Liu, Haoyuan Liang, Zhiyu Ye, Qiuyan Sun, Bingwei Chen, Yiyuan Li, Xu Zhou, Haohuan Fu

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-transformers](<https://devfeed.tech/tags/diffusion-transformers.md>), [inference](<https://devfeed.tech/tags/inference.md>), [query](<https://devfeed.tech/tags/query.md>), [sparse-attention](<https://devfeed.tech/tags/sparse-attention.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

PQRS reduces video diffusion transformer attention cost by clustering keys only and allowing each query to select its own key clusters; centroids compensate for unselected clusters. To handle these distinct selections efficiently, it traverses key clusters, groups queries with identical selections into dense tasks, and merges partial results in one pass using a softmax offset carried from the previous denoising step. The training-free method was evaluated on four text-to-video and image-to-video models from the Wan and HunyuanVideo families. The authors report it was the fastest evaluated sparse attention method and closest to dense output, with 28.9 dB PSNR on Wan2.1-T2V and 35.4 dB on HunyuanVideo at 1.66-2.10x end-to-end speedup.

### Source excerpt

arXiv:2611.24479v1 Announce Type: cross Abstract: Self-attention over long token sequences dominates the inference cost of video diffusion transformers. Clustering-based sparse attention clusters both queries and keys and lets all queries in a cluster share one selection of key clusters. We show that this sharing limits accuracy: with key clusters and density fixed, attention grows more accurate as the query clusters are refined, and is most accurate when each query selects on its own, which needs no query clustering at all. We therefore propose Per-Query Routed Sparse Attention (PQRS), a training-free method that clusters only keys, lets each query select its own key clusters, and compensates for the unselected ones with their centroids. Per-query selections leave no query tile with a shared key list, so PQRS traverses key clusters instead of queries, buckets queries with identical selections into dense tasks, and merges their partial results in one pass under a fixed softmax offset carried from the previous denoising step. On four text-to-video and image-to-video models from the Wan and HunyuanVideo families, PQRS is the fastest of all the sparse attention methods we evaluate and the closest to the dense output, reaching $28.9$ dB PSNR on Wan2.1-T2V and $35.4$ dB on HunyuanVideo at $1.66$--$2.10\times$ end-to-end speedup.

## Attention Is Not Relevance: Challenging the Causality of Attention in Autonomous Driving

DevFeed: [Attention Is Not Relevance: Challenging the Causality of Attention in Autonomous Driving](<https://devfeed.tech/articles/attention-is-not-relevance-challenging-the-causality-of-attention-in-autonomous-driving-142746.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24051>)

Author: Luca Lusvarghi, Javier Gozalvez

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [autonomous vehicles](<https://devfeed.tech/topics/autonomous-vehicles.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [causality](<https://devfeed.tech/topics/causality.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [attention](<https://devfeed.tech/tags/attention.md>), [autonomous-driving](<https://devfeed.tech/tags/autonomous-driving.md>), [autonomous-vehicle](<https://devfeed.tech/tags/autonomous-vehicle.md>), [causality](<https://devfeed.tech/tags/causality.md>), [counterfactual-analysis](<https://devfeed.tech/tags/counterfactual-analysis.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [natural-language-processing](<https://devfeed.tech/tags/natural-language-processing.md>), [planning](<https://devfeed.tech/tags/planning.md>), [relevance](<https://devfeed.tech/tags/relevance.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

Across more than 80,000 synthetic driving scenes evaluated with a vehicle based on PlanT 2.0, attention scores were correlational rather than reliable measures of how much individual objects causally affect driving decisions. Compared with counterfactual estimates of object relevance, attention methods missed some relevant objects and often labeled objects with little or no impact as relevant. They performed effectively in simple, straight single-lane scenarios, but accuracy declined on multi-lane roads and at intersections. The findings caution against using attention alone as a proxy for causal object relevance; the abstract calls for methods that distinguish causal impact from correlation.

### Source excerpt

arXiv:2611.24051v1 Announce Type: new Abstract: Attention has become one of the most widely adopted solutions for estimating the relevance of surrounding objects and interpreting an autonomous vehicle's driving decisions. However, its ability to capture the causal impact of individual input elements on an AI model's output has long been discussed in the literature, particularly in the Natural Language Processing (NLP) domain. This work brings this discussion into the autonomous driving domain by analyzing the accuracy of attention in estimating the impact of an input object on an autonomous vehicle's driving, i.e., its relevance. To this end, we compare different attention-based relevance estimation methods against a causal reference of object relevance obtained through counterfactual analysis. We perform our analysis on more than 80,000 synthetic driving scenes using an autonomous vehicle based on PlanT 2.0, a state-of-the-art Transformer-based model for object-level planning. Results show that attention provides a correlational rather than causal indication of object relevance. Attention-based relevance estimation methods do not consistently identify all relevant objects and tend to incorrectly classify as relevant objects with little or no impact on the vehicle's driving. In particular, results show that attention can effectively identify relevant objects in simple single-lane, straight-road scenarios, but its accuracy degrades in more complex settings, such as multi-lane roads and intersections. These findings highlight the limitations of attention as a proxy for object relevance, calling for novel methods able to accurately capture the causal impact of individual objects on the vehicle's driving and distinguish it from correlational associations.

## DetectViT: Test-time Backdoor Detection for Vision Transformers via Inter-Head Attention Discrepancy

DevFeed: [DetectViT: Test-time Backdoor Detection for Vision Transformers via Inter-Head Attention Discrepancy](<https://devfeed.tech/articles/detectvit-test-time-backdoor-detection-for-vision-transformers-via-inter-head-attention-discrepancy-143135.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23770>)

Author: Siquan Huang, Yijiang Li, Xingfu Yan, Ningzhi Gao, Leyu Shi, Ying Gao

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [image recognition](<https://devfeed.tech/topics/image-recognition.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [backdoor](<https://devfeed.tech/tags/backdoor.md>), [classification](<https://devfeed.tech/tags/classification.md>), [clip](<https://devfeed.tech/tags/clip.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [deit](<https://devfeed.tech/tags/deit.md>), [detection](<https://devfeed.tech/tags/detection.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [false-positive](<https://devfeed.tech/tags/false-positive.md>), [false-positive-rate](<https://devfeed.tech/tags/false-positive-rate.md>), [hijacking](<https://devfeed.tech/tags/hijacking.md>), [image](<https://devfeed.tech/tags/image.md>), [image-classification](<https://devfeed.tech/tags/image-classification.md>), [inference](<https://devfeed.tech/tags/inference.md>), [llava](<https://devfeed.tech/tags/llava.md>), [model](<https://devfeed.tech/tags/model.md>), [test](<https://devfeed.tech/tags/test.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-transformers](<https://devfeed.tech/tags/vision-transformers.md>)

### AI overview

DetectViT detects test-time backdoors in Vision Transformers by measuring two attention anomalies: unusually high spatial overlap between heads and unusually variable attention concentration across heads. It combines these as inter-head consistency and entropy-variance scores, setting thresholds with a small OOD calibration set. The method needs no training data, model outputs, or prior trigger knowledge, and reuses attention weights already computed at inference. The abstract reports results across four attacks, DeiT, CLIP, and LLaVA, and classification and captioning tasks; with 64-128 calibration samples, it reports 100% true positive and 0.17% false positive rates on BadCLIP. These are abstract-reported results, not evidence of performance beyond the stated settings.

### Source excerpt

arXiv:2611.23770v1 Announce Type: cross Abstract: Vision Transformers (ViTs) now underpin the visual encoders of most multimodal models, but this success comes at a cost: relying on third-party pretrained checkpoints leaves these systems open to backdoor attacks. Existing test-time detection methods rely on either the predicted label or intermediate embeddings, limiting their applicability across diverse tasks and incurring substantial computational overhead. In this work, we study how backdoor triggers affect the multi-head attention mechanism of ViTs and identify two distinct anomalous patterns: $\textit{complete attention hijacking}$, causing the attention to exhibit abnormally high inter-head spatial overlap, and $\textit{partial attention hijacking}$, causing an abnormally large variance in the concentration of the attention distribution. Motivated by these findings, we propose $\textbf{DetectViT}$, a test-time backdoor detection method that requires neither training data, model outputs, nor any prior knowledge of the trigger. DetectViT quantifies the two hijacking phenomena via an inter-head consistency score and an inter-head entropy variance score, with thresholds estimated from a small out-of-distribution (OOD) calibration set drawn from an arbitrary source. Across four representative attacks, diverse architectures (DeiT, CLIP, and LLaVA), and downstream tasks (image classification and captioning), DetectViT substantially outperforms all baselines with as few as 64--128 OOD calibration samples, achieving a true positive rate of $100\%$ and a false positive rate of $0.17\%$ on BadCLIP. Notably, it introduces negligible additional overhead by relying solely on attention weights already computed during inference. Code is available at: https://github.com/siquanhuang/DetectViT.

## AdaS-ViT: Test-Time Adaptive Routing for Spiking Vision Transformers with Ultra-High Energy Efficiency

DevFeed: [AdaS-ViT: Test-Time Adaptive Routing for Spiking Vision Transformers with Ultra-High Energy Efficiency](<https://devfeed.tech/articles/adas-vit-test-time-adaptive-routing-for-spiking-vision-transformers-with-ultra-high-energy-efficiency-142696.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23728>)

Author: Zecheng Hao, Zihan Huang, Shenghao Xie, Jiyuan Zhang, Zhaofei Yu, Tiejun Huang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [image recognition](<https://devfeed.tech/topics/image-recognition.md>), [Green Software](<https://devfeed.tech/topics/green-software.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [all](<https://devfeed.tech/tags/all.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [building](<https://devfeed.tech/tags/building.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-ne](<https://devfeed.tech/tags/cs-ne.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [energy](<https://devfeed.tech/tags/energy.md>), [energy-efficiency](<https://devfeed.tech/tags/energy-efficiency.md>), [routing](<https://devfeed.tech/tags/routing.md>), [test](<https://devfeed.tech/tags/test.md>), [time](<https://devfeed.tech/tags/time.md>), [transformers](<https://devfeed.tech/tags/transformers.md>), [vision](<https://devfeed.tech/tags/vision.md>), [vision-transformers](<https://devfeed.tech/tags/vision-transformers.md>), [vit](<https://devfeed.tech/tags/vit.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

AdaS-ViT reduces redundant spiking computation by adapting firing precision at test time across temporal, spatial and channel dimensions. Its AdaE-IF module uses attention scores to route salient regions to fine-grained firing patterns and the rest to coarser, lower-energy patterns; S-GSA and S-LGU add selective spike emission at token and channel granularity. A progressive spatio-temporal training curriculum is used to stabilize the routing masks and firing precision. The abstract reports competitive results across benchmarks and 1.02-2.45 mJ per image on ImageNet-1k, with an accuracy-per-energy ratio of 32x-72x.

### Source excerpt

arXiv:2611.23728v1 Announce Type: cross Abstract: Spiking Neural Networks (SNNs) promise event-driven and energy-efficient vision, yet existing spiking vision transformers (S-ViTs) largely confine spiking tokens to fixed and uniform firing schemes along the temporal, spatial and channel dimensions, whereas suppressing the computational redundancy thus incurred calls for adaptive firing mechanisms that operate jointly across all three axes at test time. To this end, we propose the Adaptive Efficient Integrate-and-Fire (AdaE-IF) model, which consults the attention score of each spiking token and routes salient regions to fine-grained firing patterns while assigning energy-frugal coarse patterns to the remaining regions. Building on AdaE-IF, we further introduce Spiking Gated Self-Attention (S-GSA) and Spiking Local-Gated Unit (S-LGU), which enforce selective spike emission at token and channel granularity. We integrate the aforementioned mechanisms and modules into AdaS-ViT, a test-time adaptive-routing architecture trained with a progressive spatio-temporal curriculum that stabilizes the co-adaptation of firing precision and routing masks. Extensive experiments across benchmarks spanning diverse data scales and modalities demonstrate that AdaS-ViT consistently achieves highly competitive performance. Notably, on ImageNet-1k it consumes only 1.02 to 2.45 millijoules per image and attains an accuracy-per-energy ratio of 32$\times$ to 72$\times$, operating in an economical inference regime and thereby pushing the energy-efficiency boundary further.

## How to Turn an Attention Neural Network into an Interpretable Hard-Logic Predicate Circuit: Truth-Table Predicate Extraction and State-Machine Merging under Ship-of-Theseus Replacement

DevFeed: [How to Turn an Attention Neural Network into an Interpretable Hard-Logic Predicate Circuit: Truth-Table Predicate Extraction and State-Machine Merging under Ship-of-Theseus Replacement](<https://devfeed.tech/articles/how-to-turn-an-attention-neural-network-into-an-interpretable-hard-logic-predicate-circuit-truth-table-predicate-extraction-and-state-machine-merging-under-ship-of-theseu-143517.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24381>)

Author: Yuchen Li, Zongqi Fan, Xintong Li, Nguyen H. Tran, Ken-Tye Yong

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [fpga](<https://devfeed.tech/topics/fpga.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [circuit](<https://devfeed.tech/tags/circuit.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fpga](<https://devfeed.tech/tags/fpga.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [logic](<https://devfeed.tech/tags/logic.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [predicate](<https://devfeed.tech/tags/predicate.md>), [replacement](<https://devfeed.tech/tags/replacement.md>), [ship-of-theseus](<https://devfeed.tech/tags/ship-of-theseus.md>), [state-machine](<https://devfeed.tech/tags/state-machine.md>), [table](<https://devfeed.tech/tags/table.md>)

### AI overview

The paper replaces an attention network with a 0/1 predicate circuit whose mappings are minimized truth tables or implication rules. It replaces blocks from output to input, allowing the remaining soft network to adapt after each step. Attention selection uses either row-wise integer-score comparisons (order selection) or learned rules; both are fitted on ordinary training examples the teacher classifies correctly. On TweetEval-emotion, order selection outperforms rule selection across both models: DistilGPT2 reaches 66.5% versus 57.4% test accuracy, and BERT-base 70.0% versus 63.7%; both remain below their teachers. The circuit requires no multiplier or weight traffic, and its hardware description reproduces it bit for bit. Synthesis-based estimates put latency at 1-2 μs per tweet on FPGA for DistilGPT2 and 3-5 μs for BERT, or 5-9 μs on a palm-sized board drawing 7.5-15 W. The abstract reports a speed advantage of two to three orders of magnitude over batch-1 GPU inference, with seed-0 accuracy costs of 8.5 and 8.6 points for DistilGPT2 and BERT.

### Source excerpt

arXiv:2611.24381v1 Announce Type: new Abstract: We convert a trained attention network as a whole, attention included, into a hard-logic predicate circuit that runs in its place. Every unit of the circuit is a 0/1 predicate on tokens, and every map between predicates is a minimised truth table. Short tables, such as the product of a query code and a key code, are minimised exactly, and wide feature-to-feature transitions become implication rules by noise-tolerant minimisation. Blocks are replaced one at a time from the output down, in the manner of the Ship of Theseus, and after each replacement the remaining soft part re-adapts to the frozen circuit. Attention selection takes two forms, a row-wise comparison of integer scores (order selection) and a learned rule (rule selection). Both are fitted on the ordinary training set alone, and only on the examples that the teacher classifies correctly, so that the circuit imitates the teacher where the teacher is right. On TweetEval-emotion, the complete circuit of DistilGPT2 reaches a test gold accuracy of 66.5 $\pm$ 1.3% with order selection and 57.4 $\pm$ 2.5% with rule selection over three seeds, against 74.7% for the teacher and 64.0% for a bag of word pieces fitted on the same tweets, and that of BERT-base reaches 70.0 $\pm$ 0.5% with order selection and 63.7 $\pm$ 1.2% with rule selection against 77.8%. The circuit needs no multiplier and no weight traffic, and its exported hardware description reproduces it bit for bit. By a synthesis-based estimate it classifies one tweet in about 1-2 $\mu$s (DistilGPT2) and 3-5 $\mu$s (BERT) on an FPGA card and in 5-9 $\mu$s on a palm-sized board whose module draws 7.5-15 W, two to three orders of magnitude faster than the network on a GPU at batch 1 (measured, 0.9-2.6 ms for DistilGPT2 including an optimised CUDA-graph baseline), at a cost of 8.5 (DistilGPT2) and 8.6 (BERT) points of gold accuracy for the synthesised seed-0 circuits.

## Efficient Wide-Beam Generative Retrieval with Model-Native Trie Decoding and Shared-Context Attention

DevFeed: [Efficient Wide-Beam Generative Retrieval with Model-Native Trie Decoding and Shared-Context Attention](<https://devfeed.tech/articles/efficient-wide-beam-generative-retrieval-with-model-native-trie-decoding-and-shared-context-attention-140011.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22309>)

Author: Vedant Jhaveri, Arashpreet Singh Mor, Satyam Kumar, Alex Tsun, Qing Lan, Shaunak Mishra, Sriram Vasudevan

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [vllm](<https://devfeed.tech/topics/vllm.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [beam](<https://devfeed.tech/tags/beam.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [generative](<https://devfeed.tech/tags/generative.md>), [model](<https://devfeed.tech/tags/model.md>), [native](<https://devfeed.tech/tags/native.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [trie](<https://devfeed.tech/tags/trie.md>), [wide](<https://devfeed.tech/tags/wide.md>)

### AI overview

Trie-Head reduces two serving costs in wide-beam generative retrieval: it stores the identifier trie as validated GPU tensors, scores compact codebook candidates, and streams the full-vocabulary normalizer, avoiding Python constraint callbacks and vocabulary-wide candidate buffers. The authors say it preserves valid-token scores in real arithmetic without retraining. Shared-context attention reuses prompt key/value blocks across beams while keeping decode tails private. Both mechanisms are implemented in SGLang and evaluated for candidate sourcing in LinkedIn Hiring Assistant and qualification retrieval on the Qualification Retrieval Platform. At beam width 200 on the latter workload, the combined system reports up to 5.57x closed-loop throughput over a Python custom logit processor baseline and up to 2.63x over BF16 FlashRec.

### Source excerpt

arXiv:2611.22309v1 Announce Type: new Abstract: Generative retrieval (GR) predicts catalog entries as short semantic identifiers (SIDs). Trie-constrained beam search keeps these identifiers valid, but wide beams expose two serving costs: vocabulary-wide score tensors that are mostly discarded, and repeated attention over the same prompt. We present Trie-Head, a model-native output interface that stores the identifier trie as validated GPU tensors. Compared with a Python custom logit processor (CLP), it reduces CPU--GPU coordination and memory traffic by avoiding Python constraint callbacks and vocabulary-wide candidate buffers. It scores compact codebook candidates and streams the full-vocabulary normalizer, preserving the original model's valid-token scores in real arithmetic without retraining. Shared-context attention complements this output path by reusing prompt key/value blocks across beams while preserving their private decode tails. We implement both mechanisms in SGLang and evaluate candidate sourcing for LinkedIn Hiring Assistant and qualification retrieval for the Qualification Retrieval Platform. The combined system improves closed-loop throughput by up to 5.57x over the CLP baseline and up to 2.63x over BF16 FlashRec at a common beam width of 200 on the Qualification Retrieval Platform workload.

## Lost in a Few Heads: Diagnosing and Repairing Block Selection in Sparse Attention

DevFeed: [Lost in a Few Heads: Diagnosing and Repairing Block Selection in Sparse Attention](<https://devfeed.tech/articles/lost-in-a-few-heads-diagnosing-and-repairing-block-selection-in-sparse-attention-141694.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21917>)

Author: Wenhe Zhang, Boxing Chen

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [block](<https://devfeed.tech/tags/block.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [long-context](<https://devfeed.tech/tags/long-context.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [selection](<https://devfeed.tech/tags/selection.md>), [sparse-attention](<https://devfeed.tech/tags/sparse-attention.md>)

### AI overview

At fixed sparse-attention budgets, retrieval loss can come from selectors ranking decisive "needle" tokens too low, rather than from insufficient budget. A teacher-forced calibration probe identifies heads whose missed needle attention contributes to lost answer probability. In InfLLM-V2-family checkpoints and NOSA-8B, three blind heads share a failure mode: mean-pooled key summaries dilute a decisive token. Applying full attention to those heads or using a 4-bit scorer at the existing budget recovers most of the loss without training. On MiniCPM4.1-8B at 64K, either raises multi-key retrieval accuracy from 54.1% to 97.5%; full attention in those heads retains 84-89% of sparse attention's TTFT savings at 32K-128K. The authors report similar blind heads in SeerAttention-R, NSA, and DashAttention, but many such heads in MoBA. For Qwen3-8B with SeerAttention-R gates, full attention in 18 of 288 key-value heads raises accuracy from 29.8% to 97.7%.

### Source excerpt

arXiv:2611.21917v1 Announce Type: new Abstract: Sparse attention models read a few blocks of keys for every query at a fixed budget and can lose much of their long-context retrieval accuracy at that budget. We ask where this loss is made. Using exact attention mass selection at the shipped budget as an oracle, we show that the selector under-ranks the needle rather than running out of budget. We find attention heads that miss the needle by running a calibration probe that teacher-forces the answer and reads two quantities in every head. The first is the needle attention that the selector forfeits, and the second is the share of the answer's log-probability that fixing the head's selection restores, integrated along the path from the shipped selector to full attention. Four checkpoints of the InfLLM-V2 family and NOSA-8B lose multi-key retrieval in the same three blind heads, which attend to the needle while a mean-pooled key summary dilutes its single decisive token. Full attention in those heads or a 4-bit scorer that keeps the shipped budget recovers most of the loss without training. On MiniCPM4.1-8B at 64K, either fix lifts multi-key retrieval accuracy from 54.1 to 97.5%, level with full attention in every head. In production serving engines, full attention in those heads keeps 84 to 89% of the time to first token (TTFT) that sparse attention saves at 32K to 128K context length. Beyond the InfLLM-V2 selector lineage, one unified rule flags attention heads above a threshold swept on each checkpoint until they close most of the gap to full attention on calibration prompts. A similar small set of blind attention heads carry the loss for the SeerAttention-R gates, NSA, and DashAttention, whereas many such heads exist in MoBA. On Qwen3-8B with the SeerAttention-R gates, full attention in 18 of 288 key-value heads lifts multi-key retrieval accuracy from 29.8 to 97.7%.

## PANO: Parallel Latent Space Neural Operator for Solving PDEs

DevFeed: [PANO: Parallel Latent Space Neural Operator for Solving PDEs](<https://devfeed.tech/articles/pano-parallel-latent-space-neural-operator-for-solving-pdes-140079.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22782>)

Author: Zijiang Yang, Xiaomeng Wu, Chen Kuang, Dongmei Fu

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [attention](<https://devfeed.tech/tags/attention.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [code](<https://devfeed.tech/tags/code.md>), [compress](<https://devfeed.tech/tags/compress.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [effective](<https://devfeed.tech/tags/effective.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [errors](<https://devfeed.tech/tags/errors.md>), [existing](<https://devfeed.tech/tags/existing.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [field](<https://devfeed.tech/tags/field.md>), [independent](<https://devfeed.tech/tags/independent.md>), [information](<https://devfeed.tech/tags/information.md>), [neural](<https://devfeed.tech/tags/neural.md>), [operator](<https://devfeed.tech/tags/operator.md>), [parallel](<https://devfeed.tech/tags/parallel.md>)

### AI overview

PANO addresses information loss from compressing PDE observations with a single aggregation pattern by combining interaction results from multiple parallel latent spaces with shared parameters. Independent mappings create distinct compressed representations; interactions are learned in parallel, decoded results are fused to predict the physical field, and PaLA adaptively updates latent spaces through cross-space interactions. CAP provides bidirectionally consistent encoding and decoding mappings. Across nine benchmarks, the authors report state-of-the-art performance, including at least 28.3% lower equivalent plastic strain prediction error on Unilateral Stamping and at least 21.4% lower lift-coefficient prediction error on AirCraft compared with existing methods. This overview is based on the supplied abstract.

### Source excerpt

arXiv:2611.22782v1 Announce Type: cross Abstract: Aggregating observations into a limited number of latent tokens and learning physical interactions in the latent space has emerged as an effective approach for Transformer-based neural operators to solve Partial Differential Equation (PDE) problems involving large-scale observations. However, existing methods typically compress observations using a single aggregation pattern, making physical information not retained by that pattern invisible to the latent space and thereby constraining physical interactions within it. To address this limitation, we propose PArallel Latent Space Neural Operator (PANO), a novel neural operator for solving PDEs that formulates physical interactions among observations as an additive combination of interaction results from parallel latent spaces with shared parameters. Specifically, PANO transforms observations into multiple distinct compressed representations through independent mappings, learns physical interactions in parallel based on these representations, and finally fuses the decoded results to predict the target physical field. Within PANO, we further propose Coupled Assignment Projection (CAP) to construct bidirectionally consistent encoding and decoding mappings, and Pairwise Latent Space Attention Layer (PaLA) to jointly update latent representations of each latent space through adaptive cross-space interactions. Extensive experiments on nine benchmarks spanning diverse physical systems, geometries, and observation scales demonstrate that PANO achieves state-of-the-art performance. Notably, PANO reduces equivalent plastic strain prediction errors on Unilateral Stamping by 28.3% (or more) and lift-coefficient prediction errors on AirCraft by 21.4% (or more) compared with existing methods. Code is available at https://github.com/ZijiangY1116/PANO.

## A Transformer-Derived Iterative Preconditioner

DevFeed: [A Transformer-Derived Iterative Preconditioner](<https://devfeed.tech/articles/a-transformer-derived-iterative-preconditioner-141821.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22941>)

Author: Patrick Lutz, Themistoklis Haris, Aditya Gangrade, Venkatesh Saligrama

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [condition-number](<https://devfeed.tech/tags/condition-number.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [linear](<https://devfeed.tech/tags/linear.md>), [local](<https://devfeed.tech/tags/local.md>), [mixing](<https://devfeed.tech/tags/mixing.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [problems](<https://devfeed.tech/tags/problems.md>), [proof-of-concept](<https://devfeed.tech/tags/proof-of-concept.md>), [rapid](<https://devfeed.tech/tags/rapid.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [reduce](<https://devfeed.tech/tags/reduce.md>), [softmax](<https://devfeed.tech/tags/softmax.md>), [solve](<https://devfeed.tech/tags/solve.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

A transformer trained to solve linear systems implicitly performs conditioning improvements; tracing its computations led to RAPID, a standalone iterative preconditioner with no learned components. For symmetric positive definite matrices, RAPID combines randomized mixing with sparse local corrections and can be applied incrementally. Proof-of-concept tests on synthetic and real-world problems report improved conditioning and fewer conjugate-gradient iterations to reach a target accuracy. Under a Haar-mixing idealization, the authors prove convergence to any fixed target condition number regardless of the matrix's spectral distribution, with logarithmic dependence on initial conditioning.

### Source excerpt

arXiv:2611.22941v1 Announce Type: new Abstract: Transformers trained on in-context prediction problems can learn known numerical algorithms, e.g., gradient descent. We go further, showing that their internal computations can reveal new algorithms with provable guarantees. A softmax-attention transformer trained to solve linear systems $Xw=y$ implicitly improves matrix conditioning. Tracing $XX^\top$ reveals an iterative preconditioning primitive, which we develop into RAPID, a standalone algorithm requiring no learned components. RAPID is an anytime preconditioner for symmetric positive definite (SPD) matrices, using randomized mixing and sparse local corrections to improve conditioning. Proof-of-concept experiments on synthetic and real-world problems consistently improve conditioning and reduce conjugate gradient iterations needed to reach a target accuracy. For a Haar-mixing idealization, we prove convergence to any fixed target condition number regardless of spectral distribution, with only logarithmic dependence on initial conditioning. Together, these results show how a transformer can discover a broadly applicable numerical algorithm.

## ThermLeT: Transformer-based Temperature Prediction for 2.5D Chiplet Architecture

DevFeed: [ThermLeT: Transformer-based Temperature Prediction for 2.5D Chiplet Architecture](<https://devfeed.tech/articles/thermlet-transformer-based-temperature-prediction-for-2-5d-chiplet-architecture-139936.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21954>)

Author: Varun Darshana Parekh, Anusha Devulapally, Sivani Devarapalli, Cassius Henderson, Shimeng Yu, Vijaykrishnan Narayanan

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [2-5d](<https://devfeed.tech/tags/2-5d.md>), [ablation](<https://devfeed.tech/tags/ablation.md>), [ansys](<https://devfeed.tech/tags/ansys.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [chiplet](<https://devfeed.tech/tags/chiplet.md>), [chiplet-architecture](<https://devfeed.tech/tags/chiplet-architecture.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [design-space-exploration](<https://devfeed.tech/tags/design-space-exploration.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [faster](<https://devfeed.tech/tags/faster.md>), [heat](<https://devfeed.tech/tags/heat.md>), [hotspot](<https://devfeed.tech/tags/hotspot.md>), [inference](<https://devfeed.tech/tags/inference.md>), [invariant](<https://devfeed.tech/tags/invariant.md>), [layouts](<https://devfeed.tech/tags/layouts.md>), [maps](<https://devfeed.tech/tags/maps.md>), [material](<https://devfeed.tech/tags/material.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [temperature](<https://devfeed.tech/tags/temperature.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

ThermLeT predicts steady-state temperatures in 2.5D chiplet layouts in 13 ms, which the abstract reports is about four orders of magnitude faster than finite-element solvers such as ANSYS. The scale-invariant multimodal transformer combines power-density maps, spatial coordinate embeddings, material properties, and heat-transfer rate. It was trained on 6,221 layouts perturbed from three representative architectures. On a 150-sample test set, it achieved 3.16°C RMSE and 2.24°C MAE; the abstract reports 35% better accuracy than ThermEDGe and 71% better than HotSpot, with roughly 6x and 11x lower inference latency, respectively. Ablations are reported to show robust generalization to unseen floorplans.

### Source excerpt

arXiv:2611.21954v1 Announce Type: cross Abstract: The drive toward heterogeneous 2.5D chiplet packages promises unmatched performance and customization, yet the resulting density amplifies thermal challenges. To facilitate rapid design space exploration, we present ThermLeT, a multimodal transformer-based, scale invariant surrogate that integrates power density maps, spatial coordinate embeddings, material properties and heat transfer rate to predict steady state temperature in 13 ms, about four orders of magnitude faster than conventional finite element solvers such as ANSYS. To train the transformer model, we also create a dataset of 6,221 chiplet layouts perturbed from three representative architectures. On a 150 sample test set, ThermLeT achieves an RMSE of 3.16{\deg}C and an MAE of 2.24{\deg}C, improving accuracy by 35% over ThermEDGe and by 71% over HotSpot, while running approximately 6x and 11x lower inference latencies respectively. Ablation studies confirm robust generalization to unseen floorplans. Code and dataset available at https://github.com/MicroSystemsDesignLab/ThermLeT

## Zero-Dimensional Attention Topology for Linguistic Acceptability

DevFeed: [Zero-Dimensional Attention Topology for Linguistic Acceptability](<https://devfeed.tech/articles/zero-dimensional-attention-topology-for-linguistic-acceptability-141765.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22312>)

Author: Saeed AlMarri (Khalifa University)

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [bert](<https://devfeed.tech/topics/bert.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [graph theory](<https://devfeed.tech/topics/graph-theory.md>), [Feature Engineering](<https://devfeed.tech/topics/feature-engineering.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [bert](<https://devfeed.tech/tags/bert.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [persistent-homology](<https://devfeed.tech/tags/persistent-homology.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [topology](<https://devfeed.tech/tags/topology.md>)

### AI overview

The fixed H₀ attention representation did not show a predictive advantage over simpler attention statistics for linguistic acceptability. It summarizes minimum-spanning-tree merge values from 144 symmetrized BERT attention graphs into 1,440 features. In the retrospective evaluation, its out-of-domain Matthews correlation was 0.438, versus 0.508 for attention statistics and 0.545 for BERT. The analysis found that the representation retains token-count information while discarding some directional attention differences. The authors also report that uncertainty estimates from resampling BERT predictions depend on the sampling unit. Attention-feature scores are retained aggregate measurements, while BERT predictions were rescored and length controls refitted from accompanying records.

### Source excerpt

arXiv:2611.22312v1 Announce Type: cross Abstract: Topological summaries of Transformer attention provide sentence representations for linguistic acceptability classification. We examine a zero-dimensional persistent-homology ($H_0$) representation that summarizes minimum-spanning-tree merge values from 144 symmetrized BERT attention graphs into 1,440 features. The retrospective evaluation uses 8,517 downstream training sentences and 521 in-domain and 497 out-of-domain CoLA validation sentences after data-integrity filtering. The recorded out-of-domain Matthews correlation coefficients are 0.438 for the topological representation and 0.508 for attention statistics, compared with 0.545 for BERT. Attention-feature results are retained aggregate measurements; BERT predictions are rescored and length controls are refitted from the accompanying records. Analytical results show that the representation retains token-count information and discards a class of directional attention differences. Source resampling of BERT predictions also demonstrates the sensitivity of uncertainty estimates to the sampling unit. The reported results do not demonstrate a predictive advantage for this fixed topological representation over simpler attention summaries. They identify representation properties and evaluation controls relevant to subsequent studies of attention topology.

## Clustering in pure-attention hardmax transformers and its role in sentiment analysis

DevFeed: [Clustering in pure-attention hardmax transformers and its role in sentiment analysis](<https://devfeed.tech/articles/clustering-in-pure-attention-hardmax-transformers-and-its-role-in-sentiment-analysis-141515.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2407.01602>)

Author: Albert Alcalde, Giovanni Fantuzzi, Enrique Zuazua

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [clustering](<https://devfeed.tech/topics/clustering.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [math-ds](<https://devfeed.tech/tags/math-ds.md>), [sentiment-analysis](<https://devfeed.tech/tags/sentiment-analysis.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

For transformers with hardmax self-attention and normalization, the analysis treats layer-by-layer input evolution as a discrete-time dynamical system and proves that inputs converge asymptotically to clusters organized around special points called leaders. The authors use this account to build a fully interpretable sentiment-analysis transformer: less meaningful words cluster around leader words that carry more meaning, providing the model's account of context. The abstract says the work also outlines remaining challenges in connecting this mathematical analysis to real-world transformer implementation.

### Source excerpt

arXiv:2407.01602v3 Announce Type: replace-cross Abstract: Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. By viewing such transformers as discrete-time dynamical systems describing the evolution of points in a Euclidean space, and thanks to a geometric interpretation of the self-attention mechanism based on hyperplane separation, we show that the transformer inputs asymptotically converge to a clustered equilibrium determined by special points called \textit{leaders}. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model, which effectively captures `context' by clustering meaningless words around leader words carrying the most meaning. Finally, we outline remaining challenges to bridge the gap between the mathematical analysis of transformers and their real-life implementation.

## Depth-Aware Activation Steering via Attention Residuals

DevFeed: [Depth-Aware Activation Steering via Attention Residuals](<https://devfeed.tech/articles/depth-aware-activation-steering-via-attention-residuals-139892.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21655>)

Author: Shilong Zhao (Independent Researcher)

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [lora](<https://devfeed.tech/topics/lora.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [activation-steering](<https://devfeed.tech/tags/activation-steering.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [attention](<https://devfeed.tech/tags/attention.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [behavioral-shifts](<https://devfeed.tech/tags/behavioral-shifts.md>), [code](<https://devfeed.tech/tags/code.md>), [cosine](<https://devfeed.tech/tags/cosine.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [data](<https://devfeed.tech/tags/data.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fail](<https://devfeed.tech/tags/fail.md>), [generation](<https://devfeed.tech/tags/generation.md>), [j](<https://devfeed.tech/tags/j.md>), [js](<https://devfeed.tech/tags/js.md>), [layer](<https://devfeed.tech/tags/layer.md>), [layers](<https://devfeed.tech/tags/layers.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [llm](<https://devfeed.tech/tags/llm.md>), [steering](<https://devfeed.tech/tags/steering.md>)

### AI overview

On Qwen3-0.6B and Qwen3.5-0.8B, spreading a persona steering vector across layers using Attention Residuals-style depth weights produced stronger internal concept alignment and behavioral shifts than single-layer injection. The effect depended on direction: reversing the vector inverted six reported metrics, while random directions reduced alignment; a short LoRA fine-tune erased the imprint. Injection changed attention only downstream of its layer. Effects rose monotonically across intensities 0.3-0.7, but stronger settings were associated with generation collapse. The authors also report that keyword-rule metrics poorly matched LLM-judge assessments (κ ≤ 0.09), and retracted two rule-based conclusions. Training depth weights for 200 steps with the backbone frozen increased held-out alignment tenfold. Cross-persona vectors retained 96-99% of the effect, though specificity was shown against random directions, not semantically related ones. These findings are limited to the two small open models studied.

### Source excerpt

arXiv:2611.21655v1 Announce Type: cross Abstract: Activation steering is a simple, training-free way to modulate LLM behavior, yet existing methods typically inject vectors at a single layer and lack mechanisms for depth-aware propagation or internal verification. We combine two recent ideas - Attention Residuals (AttnRes) providing learnable cross-layer pathways, and Jacobian-lens (J-space) concept readout - into a lightweight depth-aware steering pipeline. On two small open models (Qwen3-0.6B and Qwen3.5-0.8B), distributing a persona vector according to AttnRes-style depth weights yields stronger internal alignment (deeper-layer concept cosine and rank, p<0.0001) and stronger behavioral shifts than single-layer injection. The effect is sign-specific: reverse vectors invert all six metrics, random directions degrade alignment (p<0.0001), and a short LoRA fine-tune erases the imprint. Injection rewires attention strictly downstream of the injection point (upstream JS = 0.00000). Three practical findings follow. First, a layer-wise geometric decomposition of cosine readouts: at the injection layer 123% of the increase is arithmetic, while 5-11 layers downstream 100% is dynamical. Second, an intensity window: effects are monotone over 0.3-0.7, then generation collapses (617 to 306 to 25 chars). Third, metric discipline: keyword-rule metrics fail LLM-judge validation (kappa <= 0.09), and two rule-based conclusions were retracted. Training the depth weights on a steering objective (200 steps, frozen backbone) raises final alignment 10x on held-out prompts. Cross-persona vectors retain 96-99% of the effect: specificity holds against random, not semantic, directions. We contribute a practical, low-overhead depth-aware steering method with a careful evaluation protocol. Code and data are released.

## Gradient Boosting vs. Random Forest vs. Neural Networks: A 2026 Performance Benchmark

DevFeed: [Gradient Boosting vs. Random Forest vs. Neural Networks: A 2026 Performance Benchmark](<https://devfeed.tech/articles/gradient-boosting-vs-random-forest-vs-neural-networks-a-2026-performance-benchmark-137767.md>)

Original publisher: [Read original article](<https://machinelearningmastery.com/gradient-boosting-vs-random-forest-vs-neural-networks-a-2026-performance-benchmark/>)

Author: Vinod Chugani

Published: 2026-11-17T11:00:00Z

Content type: comparison

Language: en

Sources: [MachineLearningMastery.com](<https://devfeed.tech/sources/machinelearningmastery-com.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [catboost](<https://devfeed.tech/topics/catboost.md>), [mlp](<https://devfeed.tech/topics/mlp.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [audio](<https://devfeed.tech/tags/audio.md>), [backpropagation](<https://devfeed.tech/tags/backpropagation.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [catboost](<https://devfeed.tech/tags/catboost.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cnns](<https://devfeed.tech/tags/cnns.md>), [compare](<https://devfeed.tech/tags/compare.md>), [compute](<https://devfeed.tech/tags/compute.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [gradient-boosting](<https://devfeed.tech/tags/gradient-boosting.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [performance](<https://devfeed.tech/tags/performance.md>), [practical-machine-learning](<https://devfeed.tech/tags/practical-machine-learning.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [speed](<https://devfeed.tech/tags/speed.md>)

### AI overview

The excerpt describes two regimes for tabular data: tabular foundation models (TFMs) lead TabArena accuracy results on small-to-medium datasets, while gradient boosting remains stronger for large-scale data and regression. TabPFN-3.5 is reported to support up to 1 million rows and 20,000 features, but TFM limits vary and inference requires GPU memory to hold data in context. XGBoost, LightGBM, and CatBoost offer CPU-based training and low-latency inference, making them practical under compute or latency constraints. Random forest is a forgiving, quickly tuned baseline, especially with noisy data or limited tuning resources. Specialized networks have narrower advantages: TabR can outperform XGBoost on some classification tasks, while TabM and RealMLP reach parity with boosted trees on large datasets when carefully tuned and ensembled. For images, text, and audio, transformer architectures lead.

### Source excerpt

In this article, you will learn how gradient boosting, random forest, neural networks, and tabular foundation models compare on accuracy, speed, and data requirements in...

## Radar Panoptic Transformer: Semantic-Guided Mask Queries for Dynamic Scene Understanding

DevFeed: [Radar Panoptic Transformer: Semantic-Guided Mask Queries for Dynamic Scene Understanding](<https://devfeed.tech/articles/radar-panoptic-transformer-semantic-guided-mask-queries-for-dynamic-scene-understanding-139008.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20203>)

Author: Mayank Mayank, Bharanidhar Duraisamy, Rohit Mohan, Abhinav Valada

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine Perception](<https://devfeed.tech/topics/machine-perception.md>), [Time-sequences](<https://devfeed.tech/topics/time-sequences.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [doppler](<https://devfeed.tech/tags/doppler.md>), [mask](<https://devfeed.tech/tags/mask.md>), [network](<https://devfeed.tech/tags/network.md>), [queries](<https://devfeed.tech/tags/queries.md>), [radar](<https://devfeed.tech/tags/radar.md>), [rcs](<https://devfeed.tech/tags/rcs.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

Radar Panoptic Transformer (RPT) addresses sparse radar point clouds by combining pose-aligned temporal features and per-point neighborhoods adapted to range, azimuth, and radar cross section (RCS). Point-wise semantic predictions seed foreground queries and guide masked cross-attention to produce instance masks without geometric clustering; an auxiliary loss encourages predicted and annotated instances to have consistent RCS statistics. The abstract reports 71.6% mIoU and 82.2% PQ on RadarScenes, and says RPT outperforms evaluated baselines on MAN TruckScenes in a full-surround setting with sensing ranges up to 230 m. RadarScenes inference runs at 31 Hz, above the sensor frame rate.

### Source excerpt

arXiv:2611.20203v1 Announce Type: cross Abstract: Automotive radar provides long-range sensing and direct Doppler velocity measurements, but its sparse and non-uniformly sampled point clouds pose significant challenges for panoptic segmentation. Dynamic objects often yield only a few detections amid a dominant static background, providing limiting the geometric cues for separating object instances. We propose \emph{Radar Panoptic Transformer (RPT)}, an end-to-end point-based network that incorporates radar-adaptive feature extraction with semantic-guided mask prediction. Its backbone aggregates pose-aligned temporal context through grouped vector attention and learns per-point neighborhood radii conditioned on range, azimuth, and radar cross section (RCS). Point-wise semantic predictions initialize queries at likely foreground detections and guide masked cross-attention through query-point class agreement, producing instance masks without geometric clustering. An auxiliary RCS-consistency loss matches instance-level statistics between predicted and annotated instances. RPT achieves state-of-the-art performance on RadarScenes with $71.6\%$ mIoU and $82.2\%$ PQ. We further evaluate RPT on MAN TruckScenes, where it outperforms the evaluated baselines in a long-range, full-surround radar setting with sensing ranges up to $230\,\mathrm{m}$. RPT achieves real-time end-to-end inference at $31$\,Hz on RadarScenes, exceeding the sensor frame rate.

## When Zero Is Competitive: A Descriptive Pilot of Transformer and Gate-Graph Surrogates on Random-Angle Quantum Circuits

DevFeed: [When Zero Is Competitive: A Descriptive Pilot of Transformer and Gate-Graph Surrogates on Random-Angle Quantum Circuits](<https://devfeed.tech/articles/when-zero-is-competitive-a-descriptive-pilot-of-transformer-and-gate-graph-surrogates-on-random-angle-quantum-circuits-138971.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19985>)

Author: Mahesh Reddy Pagadala

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>)

Tags: [absolute](<https://devfeed.tech/tags/absolute.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [all](<https://devfeed.tech/tags/all.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [arm](<https://devfeed.tech/tags/arm.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [but](<https://devfeed.tech/tags/but.md>), [can](<https://devfeed.tech/tags/can.md>), [circuit](<https://devfeed.tech/tags/circuit.md>), [circuits](<https://devfeed.tech/tags/circuits.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [gate](<https://devfeed.tech/tags/gate.md>), [graph](<https://devfeed.tech/tags/graph.md>), [pilot](<https://devfeed.tech/tags/pilot.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [quantum-circuits](<https://devfeed.tech/tags/quantum-circuits.md>), [random](<https://devfeed.tech/tags/random.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [zero](<https://devfeed.tech/tags/zero.md>)

### AI overview

In a 63-cell workload of noiseless 4-10-qubit RX/RY/RZ-CNOT circuits, a literal-zero predictor achieved held-out-template equal-instance MAE 0.193399, and no evaluated transformer checkpoint or final gate-graph network endpoint scored lower. Although 44.89% of training targets were within 0.05 of zero, this is a descriptive result, not an explanation: on a separate product-state control, zero scored MAE 0.514811 versus 0.007755 for the model. A training-fitted mean constant improved on zero by only 0.14% under training MSE, without uncertainty analysis. The pilot makes no causal, equivalence, or impossibility claim. The transformer's 815,233-parameter identity was locally verified, but packed-input semantics were not independently audited; exact datasets and model-run evidence remain private.

### Source excerpt

arXiv:2611.19985v1 Announce Type: new Abstract: We examine supervised classical prediction of all one-qubit Z and two-qubit ZZ expectation values on noiseless, layered 4-10-qubit RX/RY/RZ-CNOT circuits. In a 63-cell stratified workload with disjoint circuit-template tests, we evaluate one transformer encoder and gate-dependency graph neural networks against literal-zero and training-derived constant predictors. On the Arm K training labels, 44.89% of scalar targets have absolute magnitude below 0.05. The held-out-template zero predictor has equal-instance mean absolute error 0.193399, and none of the evaluated transformer checkpoints or final graph-network endpoints achieves a lower score. A separate product-state positive control achieves MAE 0.007755 versus 0.514811 for zero, but does not explain the failure on the entangled workload. The difference between zero and a training-fitted mean constant under training MSE is only 0.14%, a descriptive point estimate without uncertainty analysis. The transformer's 815,233-parameter source identity is locally verified; detailed packed-input semantics have not been independently audited. These results constitute a descriptive pilot, not a causal explanation, formal equivalence test, or impossibility claim. Comparable circuits can be regenerated from the workload specification, but exact frozen datasets and model-run evidence remain private.

## Symmetry Fingerprinting with Geometric Learnable Invariants Transformers

DevFeed: [Symmetry Fingerprinting with Geometric Learnable Invariants Transformers](<https://devfeed.tech/articles/symmetry-fingerprinting-with-geometric-learnable-invariants-transformers-138998.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20150>)

Author: Bart M. N. Smets, Gijs Bellaard, Pieter Wils

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Transformer](<https://devfeed.tech/topics/transformer.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [data augmentation](<https://devfeed.tech/topics/data-augmentation.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-augmentation](<https://devfeed.tech/tags/data-augmentation.md>), [image-classification](<https://devfeed.tech/tags/image-classification.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [symmetry](<https://devfeed.tech/tags/symmetry.md>), [training-data](<https://devfeed.tech/tags/training-data.md>)

### AI overview

The proposed soft-equivariant neural network is designed to avoid fixing a potentially unsuitable symmetry bias in advance: it selects from chosen symmetry groups during training, becoming equivariant to those that fit the training data. After training, inspecting the network yields a symmetry fingerprint indicating which symmetries mattered and to what degree. The authors demonstrate the architecture and fingerprinting process on image classification. The supplied text does not report comparative performance results or further implementation details.

### Source excerpt

arXiv:2611.20150v1 Announce Type: new Abstract: Geometric deep learning has made available a large variety of structurally equivariant neural network architectures, meaning a particular transformation of their input causes a predictable transformation of their output. But structural equivariance is a very strong inductive bias that can limit performance if chosen poorly. In contrast, choosing an architecture with low geometric bias necessitates data augmentation to obtain correct behavior; extending the training process. We propose a novel neural network architecture that is soft equivariant; we select a set of symmetry groups and during training the network will become equivariant only to those symmetry groups that are appropriate given the training data. After training we can inspect the network to obtain a symmetry fingerprint that will tell us exactly which symmetries ended up being relevant and to what degree. We show how our architecture and the fingerprinting process work in the case of an image classification task.

## Infrastructure-Based Multi-Agent Reinforcement Learning with Attention for Highway Ramp Merging

DevFeed: [Infrastructure-Based Multi-Agent Reinforcement Learning with Attention for Highway Ramp Merging](<https://devfeed.tech/articles/infrastructure-based-multi-agent-reinforcement-learning-with-attention-for-highway-ramp-merging-138988.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20078>)

Author: Chenhao Tong, Steffen M\\"uller

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [autonomous vehicles](<https://devfeed.tech/topics/autonomous-vehicles.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [AI agents in autonomous vehicles](<https://devfeed.tech/topics/ai-agents-in-autonomous-vehicles.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [autonomous-vehicles](<https://devfeed.tech/tags/autonomous-vehicles.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [flow](<https://devfeed.tech/tags/flow.md>), [highway](<https://devfeed.tech/tags/highway.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [multi-agent](<https://devfeed.tech/tags/multi-agent.md>), [ramp](<https://devfeed.tech/tags/ramp.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [safety](<https://devfeed.tech/tags/safety.md>), [sumo](<https://devfeed.tech/tags/sumo.md>), [traffic](<https://devfeed.tech/tags/traffic.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

The paper proposes a transformer-based multi-agent reinforcement learning controller for highway ramp merging. A roadside controller receives CAV states through V2I and sends commands; an attention-based encoder lets each vehicle select relevant information from global traffic. In SUMO comparisons with neighbor-only, global-information, and rule-based baselines, the authors report the highest traffic flow, safety, and smoothness. They also report avoiding jams during a four-hour dynamic-demand simulation, effectiveness with human-driven vehicles, and collision-free operation with maintained throughput under realistic V2I errors despite ideal-communication training. These are simulation results described in the supplied abstract.

### Source excerpt

arXiv:2611.20078v1 Announce Type: new Abstract: On-ramp merging is a typical safety and throughput problem on highways. Without proper coordination, human drivers and self-interested autonomous vehicles may make risky manoeuvres and cause accidents or congestion. With recent advances in connected autonomous vehicles (CAVs) and multi-agent reinforcement learning (MARL), vehicles can be trained to coordinate for safe and efficient merging. However, in dense traffic, forming an effective observation of the traffic state is challenging. For example, observing only the neighbouring vehicles may lead to locally optimal decisions, while observing the entire traffic can introduce much irrelevant information. In this paper, we propose a transformer-based MARL vehicle control model to address these problems. A roadside infrastructure controller receives the states of all CAVs via vehicle-to-infrastructure (V2I) communication and sends control commands back to them. An attention-based observation encoder is designed where each vehicle can filter the relevant vehicle information from the global traffic state. The proposed model is evaluated in SUMO against three baselines: a model that observes only the neighbouring vehicles, a model that encodes the global traffic information, and the rule-based SUMO model. The results show that our model achieves the highest traffic flow, safety and smoothness. Furthermore, in a four-hour simulation with dynamic traffic demand, our model successfully avoids traffic jams, while other baseline models fail at high traffic flow. We also show that our model remains effective when human-driven vehicles are present. We further show that our model, trained in ideal communication conditions, can still remain collision-free and maintain throughput under realistic V2I communication errors. Finally, we visualise the relevance weights that an ego vehicle assigns to other vehicles in multiple traffic situations.

[Next page](<https://devfeed.tech/topics/transformer.md?cursor=WyIyMDI2LTExLTE3VDA1OjAwOjAwKzAwOjAwIiwgImMwYTViNjhlLWZmNzQtNGJjYy05NjM1LTA3ODA5NWMyYzZmMCJd>)