# Deep neural networks

A deep neural network is a neural network containing more than one hidden layer.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## DL101 The Representation Capability of Neural Network Models

DevFeed: [DL101 The Representation Capability of Neural Network Models](<https://devfeed.tech/articles/dl101-the-representation-capability-of-neural-network-models-144995.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24980>)

Author: Fernando Berzal

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [approximation](<https://devfeed.tech/tags/approximation.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ne](<https://devfeed.tech/tags/cs-ne.md>), [deep](<https://devfeed.tech/tags/deep.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [error](<https://devfeed.tech/tags/error.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [regions](<https://devfeed.tech/tags/regions.md>), [relu](<https://devfeed.tech/tags/relu.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [tangent](<https://devfeed.tech/tags/tangent.md>), [training](<https://devfeed.tech/tags/training.md>), [universal-approximation](<https://devfeed.tech/tags/universal-approximation.md>), [universality](<https://devfeed.tech/tags/universality.md>)

### AI overview

Universal approximation shows that shallow neural networks with non-polynomial activations can approximate any continuous function on compact sets, but it gives no guarantees about model size, successful training, or generalization. Barron's results provide quantitative approximation rates for functions with bounded Fourier norm. Montúfar's result shows that deep networks can form exponentially more linear regions as depth increases; the excerpt states that, for the same parameter count, they are more expressive than shallow networks. These results explain representational capacity, not whether training will achieve it.

### Source excerpt

arXiv:2611.24980v1 Announce Type: cross Abstract: Universal approximation theorems (Cybenko, Funahashi, Hornik & Leshno) provide the mathematical justification for why neural networks are powerful enough to model complex, nonlinear phenomena. Shallow networks can approximate any continuous function on compact sets. Leshno's theorem clarified that non-polynomial activations are necessary for universality, which explains why ReLU, logistic, hyperbolic tangent, and other common activations work (but pure polynomials do not). However, approximation theorems provide no efficiency, training, nor generalization guarantees. They do not indicate how many neurons are needed for a given accuracy, whether gradient descent will find the right parameters, or whether networks trained on finite data will generalize well. Barron provided quantitative error bounds for particular function classes (approximation rates for functions with bounded Fourier norm), whereas Mont\'ufar was the first to show that expressivity grows exponentially with depth (in terms of the number of linear regions). Deep networks are strictly more expressive than shallow ones with the same number of parameters. Depth is fundamentally more powerful than width, providing a rigorous mathematical explanation for the use of deep learning models.

## Mamba Flow Matching Neural Processes: Linear-Time Inference for Irregularly Observed Spatial Fields

DevFeed: [Mamba Flow Matching Neural Processes: Linear-Time Inference for Irregularly Observed Spatial Fields](<https://devfeed.tech/articles/mamba-flow-matching-neural-processes-linear-time-inference-for-irregularly-observed-spatial-fields-145168.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26355>)

Author: Cosmo Santoni, Giovanni Charles, Timothy James Hitge, Oliver Watson, Elizaveta Semenova

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Mamba](<https://devfeed.tech/topics/mamba.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attention](<https://devfeed.tech/tags/attention.md>), [based](<https://devfeed.tech/tags/based.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [but](<https://devfeed.tech/tags/but.md>), [combined](<https://devfeed.tech/tags/combined.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fields](<https://devfeed.tech/tags/fields.md>), [flow](<https://devfeed.tech/tags/flow.md>), [flow-matching](<https://devfeed.tech/tags/flow-matching.md>), [inference](<https://devfeed.tech/tags/inference.md>), [linear](<https://devfeed.tech/tags/linear.md>), [mamba](<https://devfeed.tech/tags/mamba.md>), [matching](<https://devfeed.tech/tags/matching.md>), [neural](<https://devfeed.tech/tags/neural.md>), [processes](<https://devfeed.tech/tags/processes.md>), [spatial](<https://devfeed.tech/tags/spatial.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

MambaFlowNP uses conditional flow matching and a multi-directional Mamba-2 scan to jointly predict irregularly observed spatial fields with linear compute. Context and query tokens share a sequence, and each query reads the scan state at its own position; this avoids reducing context to one hidden summary and makes predictions consistent across query sets. On synthetic Matérn Gaussian-process benchmarks it was competitive with leading attention-based joint-predictive baselines. For NOAA temperature interpolation, it achieved the lowest reported RMSE and CRPS, near-nominal coverage across tiers, and ran 114-784 times faster than the strongest joint-predictive baseline. A forward pass handled one million points in under a second on an NVIDIA L40S, though predictive quality was evaluated only up to 5,580 points.

### Source excerpt

arXiv:2611.26355v1 Announce Type: new Abstract: Interpolating a spatial field, such as daily maximum temperature, from sparse, irregularly located observations recurs across the environmental and health sciences. Standard Gaussian-process kriging refits hyperparameters and solves an $O(N^3)$ system for every new field. When many related fields are observed, such as daily maps from one sensor network, neural processes (NPs) instead learn the interpolation map from past fields and predict a new field in one forward pass. Joint-predictive variants, which sample whole fields rather than independent per-location values, typically use self-attention over context and query points, costing $O(K^2)$ in the combined point count $K$ and limiting practical use to a few thousand points. State-space models offer a linear-time alternative, but a direct state-space adaptation compresses the context into one hidden state before reading queries, reducing what the decoder sees. We propose MambaFlowNP, a linear-compute joint-predictive NP trained with conditional flow matching. A multi-directional Mamba-2 scan reads context and query tokens in one shared sequence, so each query's velocity is read from its own scan position rather than a compressed context summary, and because queries only read the scan state, predictions are consistent across query sets. MambaFlowNP is competitive with the strongest attention-based joint-predictive baselines on synthetic Mat\'ern Gaussian-process benchmarks, and on regional to continental-scale NOAA temperature interpolation it attains the lowest RMSE and CRPS and near-nominal coverage on every tier, while running 114--784$\times$ faster than the strongest joint-predictive baseline. A single forward pass scales to $10^6$ points on one NVIDIA L40S GPU in under a second. This makes joint prediction feasible for continental-scale, irregularly sampled sensor networks; predictive quality is evaluated up to $K=5{,}580$.

## SlotFlow-GB: A Neural Framework for Trans-Dimensional Inference of Overlapping Galactic Binaries

DevFeed: [SlotFlow-GB: A Neural Framework for Trans-Dimensional Inference of Overlapping Galactic Binaries](<https://devfeed.tech/articles/slotflow-gb-a-neural-framework-for-trans-dimensional-inference-of-overlapping-galactic-binaries-145133.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26057>)

Author: Niklas Houba, Giovanni Giarda, Davide Piras

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>)

Tags: [astro-ph-im](<https://devfeed.tech/tags/astro-ph-im.md>), [binaries](<https://devfeed.tech/tags/binaries.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gr-qc](<https://devfeed.tech/tags/gr-qc.md>), [inference](<https://devfeed.tech/tags/inference.md>), [neural](<https://devfeed.tech/tags/neural.md>), [overlap](<https://devfeed.tech/tags/overlap.md>), [physics-data-an](<https://devfeed.tech/tags/physics-data-an.md>), [physics-ins-det](<https://devfeed.tech/tags/physics-ins-det.md>)

### AI overview

SlotFlow-GB estimates both how many Galactic binaries are present and each binary's parameters in one evaluation, without requiring a source count in advance. In year-long LISA simulations with up to ten binaries per 2 μHz band, the inferred count was within one of the true count in 94% of cases, and the model identified binaries despite strong frequency overlap. A complete evaluation with parameter sampling took 41 ms per band on an NVIDIA GH200 GPU or 530 ms on eight laptop CPU threads. Its output distributions can also initialize later likelihood-based inference. The reported demonstration focuses on high-SNR cases where overlap, rather than individual-source detectability, is the main challenge.

### Source excerpt

arXiv:2611.26057v1 Announce Type: cross Abstract: The Laser Interferometer Space Antenna (LISA) will observe the millihertz gravitational-wave sky, including millions of long-lived Galactic binaries (GBs), tens of thousands of which are expected to be individually resolvable throughout the mission's lifetime. Extracting these sources requires jointly determining their number and physical parameters from strongly overlapping signals, posing a computationally demanding trans-dimensional inference problem. We introduce SlotFlow-GB, a neural framework that constructs a learned approximation to the trans-dimensional posterior, producing a probability distribution over source multiplicity and parameter densities for individual binaries in a single evaluation, without requiring their number to be specified in advance. We demonstrate this framework using year-long LISA simulations containing up to ten binaries within a 2 $\mu$Hz frequency interval, focusing on a high-SNR regime in which source overlap, rather than individual-source detectability, is the primary challenge. The inferred source count differs from the true count by at most one in 94% of cases, while the model identifies individual binaries even when their signals overlap strongly in frequency. A complete evaluation, including parameter sampling, takes just 41 ms per band on one NVIDIA GH200 GPU, or 530 ms on eight laptop CPU threads. We further demonstrate that these distributions can be used to initialize subsequent likelihood-based inference, connecting fast amortized predictions with joint parameter estimation.

## MetaSieve: Faster Relational Deep Learning through SQL-Based Metapath Selection

DevFeed: [MetaSieve: Faster Relational Deep Learning through SQL-Based Metapath Selection](<https://devfeed.tech/articles/metasieve-faster-relational-deep-learning-through-sql-based-metapath-selection-142283.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.25903>)

Author: Fahim Shahriar Khan, Ashraf Aboulnaga

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [rdbms](<https://devfeed.tech/topics/rdbms.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [JOIN](<https://devfeed.tech/topics/join.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [faster](<https://devfeed.tech/tags/faster.md>), [gnn](<https://devfeed.tech/tags/gnn.md>), [relational-databases](<https://devfeed.tech/tags/relational-databases.md>), [selection](<https://devfeed.tech/tags/selection.md>), [sql](<https://devfeed.tech/tags/sql.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

MetaSieve reduces the subgraphs used to train relational graph neural networks by pruning metapaths that appear uninformative, aiming to cut training time while maintaining accuracy. For each candidate metapath extension, it uses SQL joins and aggregations to calculate statistics, then scores the candidate for low cost and informativeness; extensions below a threshold are pruned. The selection layer relies on database statistics and task labels, not GNN parameters, and is designed to work with different GNN architectures for classification and regression. On RelBench with multiple GNN backbones, the authors report large per-epoch training-time reductions while maintaining and often improving accuracy.

### Source excerpt

arXiv:2608.25903v2 Announce Type: replace Abstract: Relational Deep Learning (RDL) is an effective approach to machine learning over multi-table relational databases. In RDL, a database is modeled as a graph in which each row is a node and each foreign-key relation is an edge, and a graph neural network (GNN) is trained on this graph. Training a GNN requires sampling a subgraph around every seed node in the training set, and the cost of training is largely determined by the size of these subgraphs. This paper aims to reduce subgraph size by leveraging the join and aggregation capabilities of relational database systems. We observe that sampled subgraphs are obtained by following metapaths composed of foreign-key links, and that many of these metapaths can be pruned without loss of accuracy. We present MetaSieve, a metapath selection layer that determines which metapaths to retain and which to prune. For each candidate metapath extension, MetaSieve computes statistics via SQL join and aggregation queries and evaluates the extension based on a novel scoring function that prefers lightweight but informative candidates. Metapaths whose scores fall below a threshold are deemed uninformative and pruned. Metapath selection in MetaSieve is lightweight since it relies only on database statistics and task labels, and it is independent of GNN parameters, so it integrates with diverse GNN architectures for classification and regression. Our evaluation on the RelBench benchmark with multiple GNN backbones shows that MetaSieve consistently reduces per-epoch training time by large margins while maintaining and often improving accuracy.

## Response-Relevant Information Is Not Sufficient for Neural Training-State Equivalence

DevFeed: [Response-Relevant Information Is Not Sufficient for Neural Training-State Equivalence](<https://devfeed.tech/articles/response-relevant-information-is-not-sufficient-for-neural-training-state-equivalence-142619.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23245>)

Author: Truong Xuan Khanh

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>)

Tags: [checkpoint](<https://devfeed.tech/tags/checkpoint.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [modular](<https://devfeed.tech/tags/modular.md>), [networks](<https://devfeed.tech/tags/networks.md>), [neural](<https://devfeed.tech/tags/neural.md>), [response](<https://devfeed.tech/tags/response.md>), [state](<https://devfeed.tech/tags/state.md>), [training](<https://devfeed.tech/tags/training.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

The study defines training-state equivalence by whether checkpoints respond alike to continued training after a specified intervention. In fresh runs of a one-layer transformer learning modular addition, responses to a weight-decay pulse were reproducible relative to between-checkpoint differences (ρ_state = 0.934 at 500 steps). Fifteen descriptors of weight spectra, representation geometry, and optimizer geometry predicted held-out response deviations beyond weight norm and loss. Yet their apparent 16.1% advantage for matching checkpoints by future response did not exceed a conditional placebo that preserved each descriptor's fitted dependence on conventional observables. A calibrated positive control made simple insensitivity an unlikely explanation. The findings suggest that response-relevant information in descriptors does not by itself establish that their pairwise similarity captures training-state equivalence.

### Source excerpt

arXiv:2611.23245v1 Announce Type: cross Abstract: Response-relevant information is not sufficient evidence that two neural networks are in the same training state. We define training-state equivalence operationally: two checkpoints are equivalent, relative to an intervention family and horizon, if they respond alike when training is perturbed and continues. On fresh runs of a one-layer transformer learning modular addition, response to a weight-decay pulse is highly reproducible relative to between-checkpoint differences (rho_state = 0.934 at 500 steps), establishing the target as measurable. Fifteen descriptors of weight spectra, representation geometry and optimiser geometry carry within-run response-relevant information beyond weight norm and loss on a development dataset, predicting held-out response deviation across seed blocks. But their apparent 16.1% advantage at matching checkpoints by future response does not separate from a conditional placebo preserving each descriptor's fitted dependence on the conventional observables, and a calibrated positive control makes simple insensitivity an unlikely explanation. Information a decoder can extract need not be organised in the pairwise geometry that similarity judgments require.

## Optimal Deep Nonparametric Regression with Uniformly Bounded Lipschitz Constants

DevFeed: [Optimal Deep Nonparametric Regression with Uniformly Bounded Lipschitz Constants](<https://devfeed.tech/articles/optimal-deep-nonparametric-regression-with-uniformly-bounded-lipschitz-constants-143467.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23979>)

Author: Jiaxin Shi, Ziyuan Chen, Jian Huang, Hansheng Wang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep](<https://devfeed.tech/tags/deep.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [regression](<https://devfeed.tech/tags/regression.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>)

### AI overview

The paper presents deep ReLU regression with controlled input sensitivity while retaining near-optimal statistical performance. Its split spectral norm product constraint supports arbitrary-accuracy approximation of sufficiently smooth Hölder functions as network depth grows, even with the norm-product bound fixed. A penalized least squares estimator then achieves the optimal nonparametric rate up to a logarithmic factor with a uniformly bounded Lipschitz constant. The authors also report optimal prediction rates for two-stage learning and robustness guarantees under adversarial perturbations. Numerical studies find controlled sensitivity and prediction accuracy comparable to classical deep network methods.

### Source excerpt

arXiv:2611.23979v1 Announce Type: cross Abstract: Stability under input perturbations is necessary for deep neural network estimators in many applications involving noisy covariates or estimated features. Weight norm constraints are widely used to control sensitivity to input perturbations by bounding the Lipschitz constant of neural networks. However, existing approximation results under fixed bounds on standard norm products are insufficient to establish optimal statistical rates. In this paper, we establish a sharp approximation theory for deep ReLU networks under a split spectral norm product constraint. This theoretical result shows that sufficiently smooth H\"{o}lder functions can be uniformly approximated to arbitrary accuracy as the network depth increases, with the norm-product bound held fixed. We then propose a penalized least squares estimator for deep nonparametric regression. The resulting estimator has a uniformly bounded Lipschitz constant while still achieving the optimal nonparametric rate up to a logarithmic factor. As applications, we establish optimal prediction rates for two-stage learning and provide robustness guarantees under adversarial input perturbations. Numerical studies show that the penalized estimator has well-controlled sensitivity to input perturbations and prediction accuracy comparable to that of classical deep network methods.

## Memorization capacity of deep ReLU neural networks characterized by width and depth

DevFeed: [Memorization capacity of deep ReLU neural networks characterized by width and depth](<https://devfeed.tech/articles/memorization-capacity-of-deep-relu-neural-networks-characterized-by-width-and-depth-141545.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.09589>)

Author: Xin Yang, Yunfei Yang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [c](<https://devfeed.tech/tags/c.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-na](<https://devfeed.tech/tags/cs-na.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [deep](<https://devfeed.tech/tags/deep.md>), [delta](<https://devfeed.tech/tags/delta.md>), [lower-bounds](<https://devfeed.tech/tags/lower-bounds.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [math-na](<https://devfeed.tech/tags/math-na.md>), [minimal](<https://devfeed.tech/tags/minimal.md>), [networks](<https://devfeed.tech/tags/networks.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [relu](<https://devfeed.tech/tags/relu.md>), [size](<https://devfeed.tech/tags/size.md>)

### AI overview

The paper constructs ReLU networks with adjustable width W and depth L that can memorize any N points in the unit ball separated by distance δ, with labels from a finite set of size C. The construction satisfies W²L² log W = O(N(log(δ⁻¹) + log C)); the paper also establishes width-depth lower bounds that explicitly depend on δ and C. When log(δ⁻¹) is on the order of log N, suitable width and depth give W²L² on the order of N. The authors state this order is optimal when log C is on the order of log N, and optimal up to logarithmic factors when log C grows more slowly than log N.

### Source excerpt

arXiv:2603.09589v2 Announce Type: replace Abstract: This paper studies the memorization capacity of deep neural networks with ReLU activation. Specifically, we investigate the minimal size of such networks to memorize any $N$ data points in the unit ball with pairwise separation distance $\delta$ and labels from a finite set of size $C$. Most prior studies characterize the memorization capacity by the number of parameters or neurons. We extend these results by constructing ReLU networks with adjustable width $W$ and depth $L$ satisfying $W^2L^2\log W=\mathcal{O}(N(\log(\delta^{-1})+\log C))$ that can memorize any such dataset. We also establish width-depth lower bounds that depend explicitly on $\delta$ and $C$ for networks capable of memorizing all such datasets. When $\log(\delta^{-1})\asymp\log N$, our construction achieves $W^2L^2\asymp N$ for suitable choices of width and depth. This order is optimal if $\log C\asymp\log N$, and optimal up to logarithmic factors if $\log C=o(\log N)$.

## BernBRS: Certified Backward Reachability for Deep Bernstein Neural Networks

DevFeed: [BernBRS: Certified Backward Reachability for Deep Bernstein Neural Networks](<https://devfeed.tech/articles/bernbrs-certified-backward-reachability-for-deep-bernstein-neural-networks-141731.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22083>)

Author: Ibrahim Albool, Yasser Shoukry

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Formal verification](<https://devfeed.tech/topics/formal-verification.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [activation-functions](<https://devfeed.tech/tags/activation-functions.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [reachability-analysis](<https://devfeed.tech/tags/reachability-analysis.md>), [safety](<https://devfeed.tech/tags/safety.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

BernBRS computes certified inner and outer approximations of the backward reachable set for DeepBern-Nets, which use Bernstein polynomial activations. It refines an input-space cover by analyzing activation patterns and splitting where an exact decomposition of the relaxation gap predicts the greatest error reduction, rather than relying on a single convex bound. The abstract reports scalability to complex architectures and evaluates the method on a linearized closed-loop Half-Cheetah controller and a distilled Bernstein student of a Transformer block; it also empirically examines how these model-level guarantees relate to the original Half-Cheetah and TinyBERT classifier. The supplied evidence does not give quantitative results.

### Source excerpt

arXiv:2611.22083v1 Announce Type: new Abstract: Ensuring the safety of complex neural network systems requires mathematically rigorous verification. While backward reachability analysis is a critical tool for identifying safe initial states, computing tight, formal bounds remains notoriously difficult to scale. Recent advancements have demonstrated that DeepBern-Nets neural networks utilizing Bernstein polynomial activation functions possess advantageous mathematical properties that enable exceptionally fast and tight forward reachability analysis. Building upon this efficiency, this paper introduces BernBRS, a highly accurate and scalable framework for the backward reachability analysis of DeepBern-Nets. BernBRS simultaneously computes certified inner and outer approximations of the Backward Reachable Set (BRS). Rather than relying on single convex bounding, our method intelligently refines a cover of the input space by analyzing activation patterns, splitting the input domain precisely where an exact decomposition of the relaxation gap predicts the most significant reduction in error. This targeted refinement enables high levels of scalability to highly complex and massive architectures that remain intractable for existing certified methods. We demonstrate the efficacy of our framework on high-dimensional models, including a linearized closed-loop approximation of the Half-Cheetah controller and a distilled Bernstein student of a Transformer block and we empirically study how these model-level guarantees relate to the original Half-Cheetah and TinyBERT classifier, respectively.

## Spatially local mini-batching for implicit neural representations

DevFeed: [Spatially local mini-batching for implicit neural representations](<https://devfeed.tech/articles/spatially-local-mini-batching-for-implicit-neural-representations-141755.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22236>)

Author: Minseok Choi, Swarup Dhar, Heejae Kwon, Kookjin Lee, Donsub Rim

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [mlp](<https://devfeed.tech/topics/mlp.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [batching](<https://devfeed.tech/tags/batching.md>), [cells](<https://devfeed.tech/tags/cells.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [hidden-states](<https://devfeed.tech/tags/hidden-states.md>), [law](<https://devfeed.tech/tags/law.md>), [local](<https://devfeed.tech/tags/local.md>), [low-rank](<https://devfeed.tech/tags/low-rank.md>), [mesh](<https://devfeed.tech/tags/mesh.md>), [network](<https://devfeed.tech/tags/network.md>), [neural](<https://devfeed.tech/tags/neural.md>), [operator](<https://devfeed.tech/tags/operator.md>), [points](<https://devfeed.tech/tags/points.md>), [problems](<https://devfeed.tech/tags/problems.md>), [rank](<https://devfeed.tech/tags/rank.md>), [time](<https://devfeed.tech/tags/time.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

Low-rank neural representations (LRNRs) can be trained on spatially local batches: each step uses points from one time snapshot and one small adaptive-mesh cell, with cells and snapshots sampled according to mesh refinement. The authors propose that LRNR structure helps because local updates reach shared weight bases through only a small set of layer coefficients; a Kolmogorov-width argument suggests hidden states within a cell lie near a low-dimensional subspace. Across six hyperbolic conservation-law problems, one recipe achieved 1.1-2.3% relative L² error using one tenth the parameters of a same-width dense network. In a same-size, same-budget comparison, hypernetwork-SIREN, DeepONet and FNO baselines reached 16-78% error while LRNR reached 4.9%. The abstract also reports that local training affects per-cell low-dimensional structure in a wide hypernetwork-SIREN.

### Source excerpt

arXiv:2611.22236v1 Announce Type: new Abstract: Neural surrogates based on implicit neural representations (INRs) are typically trained on mini-batches spread over the whole domain, since a generic INR loses accuracy when every point of a batch lies in one small region. We show that low-rank neural representations (LRNRs), INRs whose weight matrices are products of shared bases with coefficients emitted by a small hypernetwork, can be trained with spatially local mini-batches. Each step uses the points of one time snapshot of an adaptive-mesh numerical solution inside one small cell, with snapshots and cells drawn according to the refinement of the mesh. We posit that such training is enabled by the structural property of LRNRs, as a local step reaches the shared bases only through the $r$ coefficients of each layer, and keeps one cell from rewriting the network. By a Kolmogorov-width argument, we show that when restricted to a small cell, the hidden states of LRNRs lie close to a low-dimensional subspace, which suggests the local steps only affect low-rank subspaces of the shared bases. On six hyperbolic-conservation-law problems with solutions computed using adaptive mesh refinement, one local training recipe reaches 1.1-2.3% relative $L^2$ error with a tenth of the parameters of a dense network of the same width. Finally, we provide a comparison by training a hypernetwork-SIREN, a DeepONet, and a Fourier neural operator (FNO) of the same size with the same local batches and training budget. We observe that these baselines reach 16-78% where the LRNR reaches 4.9%. Interestingly, we also observe that local training has a significant effect on the per-cell low-dimensional structure even in a wide generic hypernetwork-SIREN.

## Proximal Targets: Training Looped Refiners on Problems with Many Correct Answers

DevFeed: [Proximal Targets: Training Looped Refiners on Problems with Many Correct Answers](<https://devfeed.tech/articles/proximal-targets-training-looped-refiners-on-problems-with-many-correct-answers-140072.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22737>)

Author: Syed Affan Daimi

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>)

Tags: [capacity](<https://devfeed.tech/tags/capacity.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep](<https://devfeed.tech/tags/deep.md>), [draft](<https://devfeed.tech/tags/draft.md>), [errors](<https://devfeed.tech/tags/errors.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [found](<https://devfeed.tech/tags/found.md>), [gram](<https://devfeed.tech/tags/gram.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [grid](<https://devfeed.tech/tags/grid.md>), [hrm](<https://devfeed.tech/tags/hrm.md>), [input](<https://devfeed.tech/tags/input.md>), [list](<https://devfeed.tech/tags/list.md>), [model](<https://devfeed.tech/tags/model.md>), [more](<https://devfeed.tech/tags/more.md>), [network](<https://devfeed.tech/tags/network.md>), [node](<https://devfeed.tech/tags/node.md>), [nodes](<https://devfeed.tech/tags/nodes.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [problems](<https://devfeed.tech/tags/problems.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The authors argue that training every refinement pass toward one stored answer creates an irreducible loss floor when puzzles have many valid answers, and can make the model blend them into an invalid answer. Their proximal target instead trains each pass toward a nearby correct answer, found from an answer list, by starting a search solver at the draft, or within a quality band controlled by a knob. They report that, with the same 1.7M-parameter network and five seeds, this approach solves 98.8% of Sudoku puzzles with 2-64 answers versus 2.5% for the standard target, and 93.8% of sparse Sudoku puzzles with over 10,000 answers without an answer list versus 0.0%. It also reaches 90.9% on 200-node 3-colourings after training on graphs up to 50 nodes, versus 36.9%. Against GRAM, it has higher single-run correctness and finds more answers across twenty runs, though GRAM performs slightly better on its own 10-node colouring benchmark. The excerpt says the knob serves as a calibrated strength dial in Scrabble.

### Source excerpt

arXiv:2611.22737v1 Announce Type: new Abstract: Looped refiners such as HRM and TRM apply one small network repeatedly to a draft answer, and deep supervision trains every pass toward one stored answer per puzzle. We prove that when a puzzle has many correct answers this target sets a loss floor that no model capacity removes, and that the loss-minimising pass blends the answers into an invalid grid. We propose the proximal target: train each pass toward the correct answer nearest to its own input, found by scanning a list of answers, by a search solver started at the draft, or within a band of quality set by a knob; correct answers become fixed points, and a pass only has to make fewer errors than its input has for the loop to converge, which its training loss bounds. With the same 1.7M-parameter network and 5 seeds, the proximal target solves 98.8% of Sudoku puzzles with 2-64 answers (standard target: 2.5%), 93.8% of sparse puzzles with more than 10,000 answers without any list of answers (0.0%), and 90.9% of 200-node 3-colourings after training on graphs of at most 50 nodes (36.9%), and different random starts end at different answers. Against GRAM, a stochastic-latent reasoner, a single run is correct far more often (98.8% vs. 57.7%) and twenty runs find more of the answers (53.6% vs. 37.4%), while GRAM is slightly ahead on its own 10-node colouring benchmark; in Scrabble the knob becomes a calibrated strength dial.

## Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Time

DevFeed: [Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Time](<https://devfeed.tech/articles/training-multi-layer-over-parametrized-neural-network-in-subquadratic-time-138702.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2112.07628>)

Author: Zhao Song, Lichen Zhang, Ruizhe Zhang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ds](<https://devfeed.tech/tags/cs-ds.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [network](<https://devfeed.tech/tags/network.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [subquadratic](<https://devfeed.tech/tags/subquadratic.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The framework trains the last layer of an over-parameterized network with amortized per-iteration cost m^{2−Ω(1)}, after O(m²nL) initialization. It uses a shifted ReLU with a threshold proportional to each layer input's norm, producing sparse activations and positively homogeneous layers. For fixed, sparse inputs to the trained layer, simpler algorithms without a data structure achieve the same per-iteration cost; maintaining the last layer's products with those inputs reduces it to mn^{ω−1+o(1)}. The convergence guarantee assumes random frozen layers and a width polynomial in depth, sample count, and inverse smallest kernel eigenvalues. For pre-trained models, only the product-maintenance per-iteration cost is stated to carry over.

### Source excerpt

arXiv:2112.07628v3 Announce Type: replace Abstract: We consider the problem of training the last layer of a multi-layer over-parametrized neural network to minimize the empirical risk induced by a loss function. In the typical setting of over-parametrization, the network width $m$ is much larger than the data dimension $d$ and the number of training samples $n$ ($m=\mathrm{poly}(n,d)$), which induces a prohibitively large weight matrix $W\in\mathbb{R}^{m\times m}$ per layer above the first. Naively, when the layer inputs are dense, one has to pay $O(m^2)$ time to read the weight matrix and evaluate the neural network function in both forward and backward computation. To reduce the training cost per iteration, we propose a framework that uses $O(m^2nL)$ time in the initialization phase and achieves a truly subquadratic amortized cost per iteration in terms of $m$, i.e., $m^{2-\Omega(1)}$ per iteration. The speedup comes from a sparse activation, a shifted ReLU whose threshold at each layer is proportional to the norm of the layer input. With this threshold every layer is positively homogeneous, so our convergence proof needs a width only polynomial in the depth $L$, in $n$ and in the inverse smallest kernel eigenvalues. The inputs of the trained layer are fixed and sparse. In this setting, simpler algorithms without a data structure reach the same cost per iteration, and maintaining the products of the last layer with its inputs lowers it to $mn^{\omega-1+o(1)}$, where $\omega$ is the matrix multiplication exponent. Our result can be viewed as an idealized model of maintaining a small amount of state on top of a frozen pre-trained model to speed up fine-tuning, a core procedure to deploy large language models (LLMs). Our guarantees assume that the frozen layers are random, and for a pre-trained model only the cost per iteration of maintaining these products carries over.

## Generalization Bounds for Neural Networks with Sparse Connectivity

DevFeed: [Generalization Bounds for Neural Networks with Sparse Connectivity](<https://devfeed.tech/articles/generalization-bounds-for-neural-networks-with-sparse-connectivity-138867.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19382>)

Author: Nong Minh Hieu, Antoine Ledent

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [sparsity](<https://devfeed.tech/tags/sparsity.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [theory](<https://devfeed.tech/tags/theory.md>)

### AI overview

The paper derives non-asymptotic generalization-gap bounds for deep neural networks whose layer-wise weight matrices satisfy ℓp quasi-norm constraints for 0 < p ≤ 1. The result interpolates between norm-based guarantees and parameter-counting bounds; as p approaches zero, it yields square-root dependence on the number of active (nonzero) parameters, which the authors report is tighter than existing parameter-counting approaches. Experiments show milder dependence on the initial architecture in the over-parameterized regime and suggest the bounds capture generalization benefits associated with sparsity learned from natural data.

### Source excerpt

arXiv:2611.19382v1 Announce Type: cross Abstract: Empirical evidence suggests that large neural networks rarely make effective use of all available parameters, with learned solutions exhibiting pronounced sparsity. Despite this ubiquity, existing generalization theory only partially explains how such sparsity influences statistical performance. In this paper, we derive non-asymptotic generalization gap bounds for deep neural networks whose layer-wise weight matrices are restricted by $\ell_p$ quasi-norm ($0

## Lesion-Aware Explainable AI for Diabetic Retinopathy Grading from Fundus Images

DevFeed: [Lesion-Aware Explainable AI for Diabetic Retinopathy Grading from Fundus Images](<https://devfeed.tech/articles/lesion-aware-explainable-ai-for-diabetic-retinopathy-grading-from-fundus-images-137462.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19550>)

Author: Truong Thinh Nguyen, Hai Phong Nguyen, Truong Thanh Hung Nguyen, Giang Son Tran, Hung Cao

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Computer vision](<https://devfeed.tech/topics/computer-vision.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Image](<https://devfeed.tech/topics/image.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [dr](<https://devfeed.tech/tags/dr.md>), [explainable-ai](<https://devfeed.tech/tags/explainable-ai.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [images](<https://devfeed.tech/tags/images.md>), [reliability](<https://devfeed.tech/tags/reliability.md>)

### AI overview

The proposed Consensus-Calibrated Explanation Map (CCEM) combines Grad-CAM++, Ada-SISE, and SmoothGrad Integrated Gradients with image-specific weights from annotation-free reliability cues, aiming to make DR-grading explanations more lesion-supported and faithful to the model. It explains a frozen EfficientNet-B7 ordinal grader. On 193 MESSIDOR images with expert lesion annotations, CCEM achieved a 69.61% Soft Pointing Game score (a relative saliency measure, not a lesion detection rate), a 0.570 deletion AUC, and a 0.115 deletion-insertion balance; each result was significantly better than the strongest single explainer on paired tests. The grader's reported QWK was 0.920 on the hidden Kaggle test set and 0.559 on external MAPLES-DR. The excerpt does not provide further methodological or evaluation details.

### Source excerpt

arXiv:2611.19550v1 Announce Type: cross Abstract: Post-hoc explanations of deep-learning models for diabetic retinopathy (DR) grading are difficult to trust because different attribution methods highlight different retinal regions for the same image, and a visually plausible heatmap does not show that the highlighted pixels drive the prediction. We propose the Consensus-Calibrated Explanation Map (CCEM), which fuses Grad-CAM++, Ada-SISE, and SmoothGrad Integrated Gradients (SG-IG) using image-specific weights derived from annotation-free reliability cues: deletion sensitivity, inter-method agreement, compactness, retinal containment, and peak structure. CCEM explains a frozen ordinal grader, an EfficientNet-B7 with CBAM and a scalar regression head, pre-trained on EyePACS 2015 in Stage 1 and fine-tuned on APTOS 2019 in Stage 2 with quadratic weighted kappa (QWK)-optimized thresholds; it reaches a QWK of 0.920 on the hidden Kaggle test set and 0.559 on the external MAPLES-DR cohort. Explanations are evaluated on MAPLES-DR, 198 MESSIDOR images with expert pixel-level lesion annotations that are never used for training, threshold fitting, or fusion. On the 193 images with annotated lesions, CCEM obtains the highest Soft Pointing Game score (69.61%; 95% CI 63.7-75.3), defined as the ratio of the strongest lesion-supported saliency to the global map maximum rather than a lesion detection rate, together with the lowest deletion AUC (0.570) and the best deletion-insertion balance (0.115); each gain is significant against the strongest single explainer on paired tests. Annotation-free, per-image reliability weighting therefore turns three disagreeing explainers into one map whose peak evidence is more lesion-supported and more faithful to the model than that of any single explainer.

## DNA: A Double-Helix Hierarchical Deep Interaction Network Architecture for Industrial Digital Twin

DevFeed: [DNA: A Double-Helix Hierarchical Deep Interaction Network Architecture for Industrial Digital Twin](<https://devfeed.tech/articles/dna-a-double-helix-hierarchical-deep-interaction-network-architecture-for-industrial-digital-twin-139073.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20607>)

Author: Haoran Yang, Yanan Xu, Yinan Zhang, Wenjie Zhang, Peiyu Liu, Jiahui Xiang, Wenhai Wang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Physics-guided deep learning](<https://devfeed.tech/topics/physics-guided-deep-learning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep](<https://devfeed.tech/tags/deep.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [digital](<https://devfeed.tech/tags/digital.md>), [digital-twin](<https://devfeed.tech/tags/digital-twin.md>), [dna](<https://devfeed.tech/tags/dna.md>), [helix](<https://devfeed.tech/tags/helix.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [industrial](<https://devfeed.tech/tags/industrial.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [models](<https://devfeed.tech/tags/models.md>), [network](<https://devfeed.tech/tags/network.md>), [network-architecture](<https://devfeed.tech/tags/network-architecture.md>), [novel](<https://devfeed.tech/tags/novel.md>), [structure](<https://devfeed.tech/tags/structure.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

DNA combines mechanistic and data-driven modeling through two interaction strategies, then uses a hierarchical, multi-stage structure implemented with two tailored deep learning frameworks to model complex industrial systems. Evaluated on two industrial systems, it reduced MAE by 3.704% and 3.175%, respectively, compared with the strongest baseline. The supplied abstract does not describe the interaction strategies or frameworks in detail.

### Source excerpt

arXiv:2611.20607v1 Announce Type: new Abstract: High-fidelity modeling is fundamental to industrial digital twin, yet a trade-off persists between the physical transparency of mechanistic models and the superior predictive power of data-driven approaches. Existing hybrid methods often fail to bridge this gap, suffering from shallow integration and rigid one-shot modeling strategies that struggle with complex industrial systems. To address these issues, we propose a \textbf{Double-Helix Hierarchical Deep Interaction Network Architecture (DNA)}. Inspired by the double-helix structure of biological \textit{DNA}, our architecture intertwines mechanistic and data-driven modeling strands through two novel interaction strategies, forming a symbiotic architecture in which the two strands mutually enhance predictive capability. Furthermore, the hierarchical structure of DNA allows it to model complex systems in a multi-stage manner through two tailored deep learning frameworks. We evaluate the proposed approach on two distinct industrial systems, where DNA achieves reductions in MAE of 3.704\% and 3.175\%, respectively, compared to the strongest baseline. These results demonstrate the effectiveness of DNA and suggest its potential to provide new insights for the development of industrial digital twins.

## Hyperlyve: Parallelized Enumeration and Exact Lyapunov Stability Certification for Deep ReLU Networks

DevFeed: [Hyperlyve: Parallelized Enumeration and Exact Lyapunov Stability Certification for Deep ReLU Networks](<https://devfeed.tech/articles/hyperlyve-parallelized-enumeration-and-exact-lyapunov-stability-certification-for-deep-relu-networks-139199.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21599>)

Author: Brian Yarbrough, Jesse Wayment, Jingbo Wang, Philip E. Par\\'{e}, Shreyas Sundaram

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [certification](<https://devfeed.tech/tags/certification.md>), [computational-complexity](<https://devfeed.tech/tags/computational-complexity.md>), [controllers](<https://devfeed.tech/tags/controllers.md>), [cpa](<https://devfeed.tech/tags/cpa.md>), [cs-cg](<https://devfeed.tech/tags/cs-cg.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-lo](<https://devfeed.tech/tags/cs-lo.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [deep](<https://devfeed.tech/tags/deep.md>), [domain](<https://devfeed.tech/tags/domain.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [enumeration](<https://devfeed.tech/tags/enumeration.md>), [framework](<https://devfeed.tech/tags/framework.md>), [hyperplane](<https://devfeed.tech/tags/hyperplane.md>), [inductive](<https://devfeed.tech/tags/inductive.md>), [math-ds](<https://devfeed.tech/tags/math-ds.md>), [neural](<https://devfeed.tech/tags/neural.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [region](<https://devfeed.tech/tags/region.md>), [relu](<https://devfeed.tech/tags/relu.md>), [sound](<https://devfeed.tech/tags/sound.md>), [systems](<https://devfeed.tech/tags/systems.md>), [tests](<https://devfeed.tech/tags/tests.md>), [times](<https://devfeed.tech/tags/times.md>)

### AI overview

Hyperlyve certifies asymptotic stability for nonlinear systems, including systems with neural network controllers, using continuous piecewise-affine neural Lyapunov functions in a counterexample-guided synthesis loop. Its parallel algorithm enumerates the regions induced by deep ReLU activations and checks Lyapunov conditions in each. The authors report sound and complete regional tests for the exact smooth dynamics within a compact domain, and classify CPA neural Lyapunov verification as co-NP-complete. In reported experiments, Hyperlyve enumerated ReLU networks over 150 times faster than competing tools, enabling proofs for systems described as intractable for state-of-the-art verifiers.

### Source excerpt

arXiv:2611.21599v1 Announce Type: cross Abstract: We introduce Hyperlyve (Hyperplane Partitioned Lyapunov Verification and Enumeration), a framework for proving the asymptotic stability of nonlinear dynamical systems, including those with neural network controllers. Hyperlyve learns and formally certifies continuous piecewise-affine (CPA) neural Lyapunov functions within a counterexample-guided inductive synthesis (CEGIS) loop. Hyperlyve's key innovation is a parallel algorithm that enumerates the hyperplane arrangement formed by the ReLU activations of deep networks and certifies that Lyapunov conditions hold within each polyhedral region. We derive sound and complete regional tests to verify the asymptotic stability of the exact smooth dynamics (rather than a piecewise-linear approximation) within a compact domain of interest. Furthermore, we formally characterize the computational complexity of the CPA neural Lyapunov verification problem as co-NP-complete, filling a key gap in the literature. Empirical results demonstrate that Hyperlyve enumerates ReLU networks over 150-times faster than competing tools, which contributes to Hyperlyve proving the asymptotic stability of systems that are currently intractable for state-of-the-art verifiers.

## PIRanNN: Random-Feature Pricing and Calibration under Jump-Diffusion and Stochastic-Volatility Models

DevFeed: [PIRanNN: Random-Feature Pricing and Calibration under Jump-Diffusion and Stochastic-Volatility Models](<https://devfeed.tech/articles/pirannn-random-feature-pricing-and-calibration-under-jump-diffusion-and-stochastic-volatility-models-138932.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19792>)

Author: Saurabh Bansal, Srikanth K. Iyer, Shashi Jain

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-na](<https://devfeed.tech/tags/cs-na.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [error](<https://devfeed.tech/tags/error.md>), [errors](<https://devfeed.tech/tags/errors.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [feature](<https://devfeed.tech/tags/feature.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [math-na](<https://devfeed.tech/tags/math-na.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [models](<https://devfeed.tech/tags/models.md>), [network](<https://devfeed.tech/tags/network.md>), [neural](<https://devfeed.tech/tags/neural.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [options](<https://devfeed.tech/tags/options.md>), [physics-informed](<https://devfeed.tech/tags/physics-informed.md>), [prices](<https://devfeed.tech/tags/prices.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [q-fin-cp](<https://devfeed.tech/tags/q-fin-cp.md>), [quotes](<https://devfeed.tech/tags/quotes.md>), [random](<https://devfeed.tech/tags/random.md>), [standard](<https://devfeed.tech/tags/standard.md>)

### AI overview

PIRanNN prices and calibrates European options with a single regularised least-squares fit: a one-hidden-layer probit network uses fixed random inner weights, and requires no iterative training. For Merton and Kou jump models, each feature's jump integral is explicit, avoiding quadrature; one matrix factorisation yields prices, Greeks and exact parameter derivatives. Subtracting the Black-Scholes price removes the payoff kink and cuts jump-model pricing error by about 100 times. At equal runtime, PIRanNN is then about as accurate as standard Crank-Nicolson, while finite differences with the same subtraction are about 100 times more accurate. On cited benchmark problems, it matches a physics-informed neural network's accuracy in seconds rather than roughly half an hour. Calibration also depends on pricing accuracy: without subtraction, jump parameters can be off by up to 50%; with it, they are recovered within 3%. In Heston and SABR tests, errors are about 1% and calibrated parameters are within about 15% of their true values.

### Source excerpt

arXiv:2611.19792v1 Announce Type: cross Abstract: We study a random-feature method for pricing and calibrating European options under jump-diffusion (Merton, Kou) and stochastic-volatility (Heston, SABR) models. The network has one hidden layer with a probit activation and fixed random inner weights, so fitting it to the pricing equation, the payoff and the boundary conditions is a single regularised least-squares problem, with no iterative training. For Merton and Kou the jump integral of each feature is explicit, so no quadrature is needed. One factorisation of the least-squares matrix gives the price at every state and time, its Greeks, and the exact derivatives of the prices with respect to the model parameters for gradient-based calibration. For the jump models, subtracting the Black-Scholes price removes the payoff kink and reduces the pricing error by about two orders of magnitude. The method is then about as accurate as a standard Crank-Nicolson finite-difference scheme at the same run time, although a finite-difference scheme that also subtracts the Black-Scholes price is about two orders of magnitude more accurate still. On the test problems of a recent physics-informed neural network study, the method matches that network's accuracy in seconds instead of about half an hour. The calibration experiments show that a solver can fit the quotes and still return wrong parameters when its pricing error is not small compared with the effect of the parameters on the prices: without the subtraction the solver misses the jump parameters by up to 50%, and with it they are recovered to within 3%. For Heston and SABR, used as an applicability study with boundary conditions that need no reference prices, pricing errors are about one percent and the calibrated parameters are within about 15% of the true values.

## Network-Native DNN Architecture Synthesis for Distributed Inference

DevFeed: [Network-Native DNN Architecture Synthesis for Distributed Inference](<https://devfeed.tech/articles/network-native-dnn-architecture-synthesis-for-distributed-inference-136993.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20824>)

Author: Fan Yang, Xiaolong Wu, Qi Cao, Fen Hou, Yulin Shao

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [distributed algorithm](<https://devfeed.tech/topics/distributed-algorithm.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [communication](<https://devfeed.tech/tags/communication.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-it](<https://devfeed.tech/tags/cs-it.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-inference](<https://devfeed.tech/tags/distributed-inference.md>), [dnn](<https://devfeed.tech/tags/dnn.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [linear](<https://devfeed.tech/tags/linear.md>), [linear-programming](<https://devfeed.tech/tags/linear-programming.md>), [math-it](<https://devfeed.tech/tags/math-it.md>), [model](<https://devfeed.tech/tags/model.md>), [native](<https://devfeed.tech/tags/native.md>), [network](<https://devfeed.tech/tags/network.md>), [networks](<https://devfeed.tech/tags/networks.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [synthesis](<https://devfeed.tech/tags/synthesis.md>)

### AI overview

The paper proposes designing distributed DNN architectures around a physical network's topology, which defines dependencies between nodes; computation is allocated subject to communication, compute, storage, synchronization, and latency constraints. Its LP-based Architecture Synthesis (LPAS) algorithm produces feasible architectures with a computable suboptimality bound. In experiments, LPAS had relative gaps below 0.6% to an LP-derived upper bound and ran substantially faster than direct mixed-integer LP optimization on larger networks. Under matched deadlines, network-native models used more compute nodes and model-storage capacity than model-first deployments; after training, they achieved higher classification accuracy across evaluated dataset-precision settings, with average gains up to 10.69%.

### Source excerpt

arXiv:2611.20824v1 Announce Type: cross Abstract: Distributed deep neural network (DNN) inference typically follows a model-first paradigm, in which a predefined DNN is partitioned and deployed over a physical network. On irregular communication-computation topologies, however, this approach can create transmission bottlenecks and leave distributed resources underutilized. This paper reverses this perspective and asks: given a physical network, what kind of DNN can it naturally support? To answer this question, we propose a network-native DNN architecture synthesis paradigm in which the physical network topology directly defines inter-node neural dependencies. Neural computation is allocated across nodes under communication, computation, storage, synchronization, and latency constraints. We establish structural properties of the synthesis problem and develop linear programming (LP)-based Architecture Synthesis (LPAS), an efficient topology-guided algorithm that produces a feasible architecture with a computable bound on suboptimality. Experiments on physical networks of different sizes show that LPAS achieves relative gaps below 0.6% to the LP-derived upper bound, with substantially lower runtime than direct mixed-integer LP optimization on larger networks. Under matched network deadlines, the network-native architectures activate more compute nodes and utilize a larger proportion of available model-storage capacity than the model-first baseline. After training, the network-native models achieve higher classification accuracy across all evaluated dataset-precision settings, with the average gain of up to 10.69%. This network-native design paradigm thus turns the physical network from a deployment constraint into an architecture design primitive.

## Deep Latents, Long Memory: Latent persistence explains temporal correlations in transformer activations

DevFeed: [Deep Latents, Long Memory: Latent persistence explains temporal correlations in transformer activations](<https://devfeed.tech/articles/deep-latents-long-memory-latent-persistence-explains-temporal-correlations-in-transformer-activations-135865.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18089>)

Author: Francesco Cagnetta, Santiago Acevedo

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [abstraction](<https://devfeed.tech/tags/abstraction.md>), [autoencoder](<https://devfeed.tech/tags/autoencoder.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [features](<https://devfeed.tech/tags/features.md>), [growth](<https://devfeed.tech/tags/growth.md>), [memory](<https://devfeed.tech/tags/memory.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [spans](<https://devfeed.tech/tags/spans.md>), [structure](<https://devfeed.tech/tags/structure.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [synthetic-data](<https://devfeed.tech/tags/synthetic-data.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [text](<https://devfeed.tech/tags/text.md>), [transformer](<https://devfeed.tech/tags/transformer.md>)

### AI overview

The paper uses layer-wise activation autocorrelation--the average dot product between activations at positions separated by a context lag--as a measurable way to study temporal structure in transformer representations. It reports that autocorrelation tails grow across layers in synthetic and natural data. In controlled synthetic data, persistent latent codes governing long input spans predict the autocorrelation structure and explain most empirical measures. For real text, sparse-autoencoder features serve as proxies for latent variables: their persistence correlates with autocorrelation magnitude across datasets and LLMs, and persistence time increases with depth. Ablating persistent features also reduces autocorrelations. The evidence is summarized in the supplied abstract.

### Source excerpt

arXiv:2611.18089v1 Announce Type: new Abstract: The hidden layers of Large Language Models (LLMs) are believed to encode representations of the data whose abstraction increases with depth. However, in the absence of an operational notion of abstraction, this statement is hard to test quantitatively. Here, we propose an alternative view, building on the principle that the correlations of a sequence reflect the structure of the process that generated it. We define the autocorrelation function of a layer as the average dot product between its activations at two context positions separated by a lag. Latent variables encoded persistently along the context create slowly decaying tails in the autocorrelation, and we find these tails to grow across layers in both synthetic and natural data. In synthetic data with controlled hierarchical latent structure, this growth stems from the encoding of abstract latent variables, which govern long stretches of the input. Indeed, the persistence of latent codes predicts the qualitative structure of autocorrelations and accounts for most of the empirical measures. In real text, we take sparse-autoencoder features as proxies for latent variables, and find that their persistence correlates strongly with the magnitude of autocorrelations across datasets and LLMs. In particular, the persistence time increases with depth, as do autocorrelations, implying that deeper latent features persist for longer context spans. We complement this analysis by explicitly ablating the persistent sparse-autoencoder features and observing a consequent decrease in the autocorrelations.

## ERGo: Energy-Efficient Hybrid Graph Neural Network Training on Processing-In-Memory Architectures

DevFeed: [ERGo: Energy-Efficient Hybrid Graph Neural Network Training on Processing-In-Memory Architectures](<https://devfeed.tech/articles/ergo-energy-efficient-hybrid-graph-neural-network-training-on-processing-in-memory-architectures-135781.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17478>)

Author: Pratyush Dhingra, Chibuike E. Ugwu, Janardhan Rao Doppa, Partha Pratim Pande

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [Green Software](<https://devfeed.tech/topics/green-software.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [based](<https://devfeed.tech/tags/based.md>), [can](<https://devfeed.tech/tags/can.md>), [cells](<https://devfeed.tech/tags/cells.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [energy](<https://devfeed.tech/tags/energy.md>), [ergo](<https://devfeed.tech/tags/ergo.md>), [graph](<https://devfeed.tech/tags/graph.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [hybrid-graph](<https://devfeed.tech/tags/hybrid-graph.md>), [memory](<https://devfeed.tech/tags/memory.md>), [network](<https://devfeed.tech/tags/network.md>), [neural](<https://devfeed.tech/tags/neural.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [processing](<https://devfeed.tech/tags/processing.md>), [processing-in-memory](<https://devfeed.tech/tags/processing-in-memory.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

ERGo reduces costly off-chip access during GNN training on processing-in-memory (PIM) architectures by automatically switching from full-parameter training to a parameter-efficient phase that represents weights at low rank. The abstract reports negligible loss in predictive accuracy, up to 3.15x speedup, and 10.5x greater energy efficiency than conventional full-parameter training on a PIM platform. On heterogeneous NVM-SRAM systems, which use NVM for forward passes and SRAM for backward passes, ERGo freezes weights in the parameter-efficient phase, reducing NVM writes associated with updates. Experiments report up to 33x longer architecture lifetime than the full-parameter implementation. These are reported maxima; the excerpt gives no experimental conditions or broader evaluation details.

### Source excerpt

arXiv:2611.17478v1 Announce Type: new Abstract: Processing-in-memory (PIM) has been proposed as an alternative computing paradigm for training Deep Neural Networks, including Graph Neural Networks (GNNs). Despite these advancements, training GNN workloads on PIM devices necessitates off-chip memory access. This off-chip access is expensive in terms of performance and energy, thereby impacting the overall energy efficiency of the training process on PIM platforms. In this paper, we propose a novel hybrid training framework called ERGo that automatically switches from a full-parameter phase to a parameter-efficient phase during the training process with negligible loss in predictive accuracy. The parameter-efficient phase employs low-rank representations of GNN weights, effectively reducing the number of trainable parameters. This helps in reducing the off-chip access during the end-to-end training process on PIM architectures, leading to notable improvements in both performance and energy efficiency. ERGo outperforms the conventional full-parameter GNN training on a PIM-based platform by up to 3.15x in speedup and enhances energy efficiency by 10.5x. Furthermore, ERGo offers additional advantages when utilized on heterogeneous architectures incorporating non-volatile memory (NVM) and static random-access memory (SRAM). Training on the heterogeneous NVM-SRAM architecture typically utilizes NVM for the forward-pass and SRAM for the backward-pass. However, NVM cells suffer from low device endurance and repeated write operations due to weight updates can lead to poor lifetime. ERGo improves the lifetime of NVM devices by freezing the weights and preventing weight updates in the parameter-efficient phase. Experimental results demonstrate that ERGo improves the lifetime of heterogeneous NVM-SRAM architecture by up to 33x in comparison to full-parameter implementation.

## Neural Classifiers as Error-Correcting Codes

DevFeed: [Neural Classifiers as Error-Correcting Codes](<https://devfeed.tech/articles/neural-classifiers-as-error-correcting-codes-135882.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18239>)

Author: Sergey Salishev, Anton Makarov, Oleg Granichin, Renata Avros, Zeev Volkovich

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Decoding](<https://devfeed.tech/topics/decoding.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>)

Tags: [classification](<https://devfeed.tech/tags/classification.md>), [classifiers](<https://devfeed.tech/tags/classifiers.md>), [cs-it](<https://devfeed.tech/tags/cs-it.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [deep](<https://devfeed.tech/tags/deep.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [error-correcting-codes](<https://devfeed.tech/tags/error-correcting-codes.md>), [feature](<https://devfeed.tech/tags/feature.md>), [hamming](<https://devfeed.tech/tags/hamming.md>), [knowledge-distillation](<https://devfeed.tech/tags/knowledge-distillation.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [math-it](<https://devfeed.tech/tags/math-it.md>), [models](<https://devfeed.tech/tags/models.md>), [noise](<https://devfeed.tech/tags/noise.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [redundancy](<https://devfeed.tech/tags/redundancy.md>), [shannon](<https://devfeed.tech/tags/shannon.md>), [space](<https://devfeed.tech/tags/space.md>)

### AI overview

The abstract proposes viewing supervised deep classification as error-correcting-code decoding: cross-entropy aligns softmax classifiers with maximum-likelihood decoding, with logits corresponding to Hamming distances to class-specific codewords. It says this framing accounts for standard divergence objectives and offers a geometric basis for decision-boundary stability. Neural Collapse and LeJEPA geometries are described as optimal decoding constellations under isotropic noise, while deep networks learn feature-space codebooks implicitly rather than using classical Error-Correcting Output Codes. The authors propose applying this view to lossless quantization, knowledge distillation, and architectural regularization through code redundancy, and extend it to self-supervised encoders. These are claims and proposed applications in the abstract; supporting details are not included.

### Source excerpt

arXiv:2611.18239v1 Announce Type: new Abstract: A unified framework is proposed that characterizes supervised deep classification as the decoding of error-correcting codes in the Shannon sense. Cross-entropy optimization is shown to align softmax classifiers with maximum-likelihood decoding, where logits map to Hamming distances to class-specific codewords. This correspondence explains the emergence of standard divergence objectives (e.g., Kullback-Leibler, Jeffreys) and provides a geometric basis for the stability of decision boundaries. Furthermore, the geometries of Neural Collapse and LeJEPA are identified as optimal constellations for decoding under isotropic noise. Unlike classical Error-Correcting Output Codes, deep networks are shown to learn effective codebooks in the feature space implicitly. This insight is leveraged to derive novel criteria for lossless quantization, improved knowledge distillation, and architectural regularization based on code redundancy. Finally, this perspective is extended to self-supervised encoders, offering a cohesive explanation for the geometry of latent representations in modern neural models.

## Deep Neural Networks Training Efficiency: Does Grokking Modify its Dynamics?

DevFeed: [Deep Neural Networks Training Efficiency: Does Grokking Modify its Dynamics?](<https://devfeed.tech/articles/deep-neural-networks-training-efficiency-does-grokking-modify-its-dynamics-135897.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18396>)

Author: Eduardo Cueto-Mendoza, John D. Kelleher

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>)

Tags: [64-bit](<https://devfeed.tech/tags/64-bit.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deep-neural-networks](<https://devfeed.tech/tags/deep-neural-networks.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [grokking](<https://devfeed.tech/tags/grokking.md>), [mnist](<https://devfeed.tech/tags/mnist.md>), [modular](<https://devfeed.tech/tags/modular.md>), [out-of-distribution-generalization](<https://devfeed.tech/tags/out-of-distribution-generalization.md>), [overfitting](<https://devfeed.tech/tags/overfitting.md>), [precision](<https://devfeed.tech/tags/precision.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The abstract reports that grokking did not materially change training efficiency, measured as accuracy per unit energy, in the tested settings. Across modular arithmetic and MNIST, two network models, and 16-, 32-, and 64-bit precision, short runs of up to 500 epochs showed smoothly decreasing efficiency; progressively stronger early Orthograd did not induce grokking. In runs of up to 20,000 epochs, Orthograd induced grokking, but efficiency remained approximately monotonic, with no appreciable local increase at the test-accuracy transition. The conclusion is limited to these experiments.

### Source excerpt

arXiv:2611.18396v1 Announce Type: new Abstract: We study whether grokking, the delayed emergence of out-of-distribution generalization after apparent overfitting, alters training efficiency in deep neural networks, where efficiency is measured as accuracy per unit energy. Using two canonical settings, modular arithmetic and MNIST, we analyze two neural network models under 16-, 32-, and 64-bit precision and use Orthograd to test whether grokking produces noticeable deviations from the typically decreasing dynamics of training efficiency. We consider both short training runs of up to 500 epochs and long training runs of up to 20,000 epochs, and in both cases identify grokking as a sharp improvement in test accuracy after a sustained train-test gap. In the short-horizon experiments, applying Orthograd at progressively stronger levels during early training does not induce grokking, and efficiency remains smoothly decreasing. In the long-horizon experiments, Orthograd induces grokking, but the efficiency curves remain approximately monotonic and show no appreciable local increase at the grokking transition. These results suggest that, in our setting, grokking does not materially alter overall training-efficiency dynamics.

## Understanding Contextual Bias in Agricultural CNNs Through Hidden-Neuron Concept Induction

DevFeed: [Understanding Contextual Bias in Agricultural CNNs Through Hidden-Neuron Concept Induction](<https://devfeed.tech/articles/understanding-contextual-bias-in-agricultural-cnns-through-hidden-neuron-concept-induction-135914.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18530>)

Author: Abhilekha Dalal, Avishek Das, Samatha Ereshi Akkamahadevi

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [background](<https://devfeed.tech/tags/background.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [bias](<https://devfeed.tech/tags/bias.md>), [classification](<https://devfeed.tech/tags/classification.md>), [classification-accuracy](<https://devfeed.tech/tags/classification-accuracy.md>), [cnns](<https://devfeed.tech/tags/cnns.md>), [convolutional-neural-networks](<https://devfeed.tech/tags/convolutional-neural-networks.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [delta](<https://devfeed.tech/tags/delta.md>), [edge](<https://devfeed.tech/tags/edge.md>), [feature](<https://devfeed.tech/tags/feature.md>), [features](<https://devfeed.tech/tags/features.md>), [graph](<https://devfeed.tech/tags/graph.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [induction](<https://devfeed.tech/tags/induction.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [layers](<https://devfeed.tech/tags/layers.md>), [location](<https://devfeed.tech/tags/location.md>), [neuron](<https://devfeed.tech/tags/neuron.md>)

### AI overview

The study finds that interpretable hidden-neuron features and a CNN's reliance on image context capture different aspects of behavior. Researchers applied Concept Induction to patch descriptors encoded in a knowledge graph across five agricultural datasets, four CNN architectures, three depths, and five patch locations. On held-out images, 13,164 of 48,013 neuron-feature pairs (27.4%) were validated. Validation was more frequent in mid layers than deep layers (40.2% versus 12.0% on average); edge, Gabor, and Tamura texture descriptors validated most often, and no patch location consistently dominated. Validation rates did not rise with contextual bias: moderately biased Sugarcane had the highest rate (40.5%), while highly biased Cassava had the lowest (19.2%). The authors suggest Concept Induction can complement patch-based bias detection.

### Source excerpt

arXiv:2611.18530v1 Announce Type: cross Abstract: Convolutional Neural Networks (CNNs) can classify images using background-dominated patches that contain little or none of the target object, showing that they exploit contextual information. Previous studies detected this behavior through classification accuracy but say little about what CNN hidden layers represent. We investigate whether Concept Induction can reveal interpretable visual properties associated with hidden-neuron activations on contextual image patches. Using five agricultural datasets with high, moderate, and low contextual bias, we analyze four CNN architectures at three network depths and five patch locations. Quantitative image descriptors of each patch are encoded in a knowledge graph and used as background knowledge for Concept Induction, which derives logical expressions characterizing neuron activation. These expressions are tested on held-out images using the Mann-Whitney U test and Cliff's delta. Of 48,013 tested neuron-feature pairs, 13,164 (27.4%) were validated. Validation was most frequent at intermediate depths (mean 40.2% at mid layers versus 12.0% at deep layers), and no patch location dominated across datasets. Among the tested descriptors, edge, Gabor, and Tamura texture features had the highest validation rates. Stronger contextual bias did not correspond to higher neuron-feature validation rates: moderately biased Sugarcane had the highest rate (40.5%) and highly biased Cassava the lowest (19.2%). These results suggest that behavioral context dependence and the semantic interpretability of hidden neurons capture different aspects of CNN behavior, and that Concept Induction can complement patch-based bias detection.

## A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

DevFeed: [A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks](<https://devfeed.tech/articles/a-closed-form-depth-width-law-for-neural-collapse-in-deep-linear-networks-132844.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.21820>)

Author: Pankhil Gawade, Kirsten Fischer, Guy Wolf, Ewa Szczurek

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [class](<https://devfeed.tech/tags/class.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [deep](<https://devfeed.tech/tags/deep.md>), [feature](<https://devfeed.tech/tags/feature.md>), [function](<https://devfeed.tech/tags/function.md>), [input](<https://devfeed.tech/tags/input.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [law](<https://devfeed.tech/tags/law.md>), [learning](<https://devfeed.tech/tags/learning.md>), [linear](<https://devfeed.tech/tags/linear.md>), [means](<https://devfeed.tech/tags/means.md>), [networks](<https://devfeed.tech/tags/networks.md>), [neural](<https://devfeed.tech/tags/neural.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [properties](<https://devfeed.tech/tags/properties.md>), [ratio](<https://devfeed.tech/tags/ratio.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [theory](<https://devfeed.tech/tags/theory.md>)

### AI overview

For deep linear networks with an exchangeable input kernel, the learned feature kernel captures three neural-collapse properties: within-class variability collapse (NC1), simplex class means (NC2), and classifier alignment (NC3). The kernel equations give an exact geometric law within this theory for NC1's decrease across layers. At large width N, the accumulated decrease depends on the depth-to-width ratio L/N, with its rate determined self-consistently by the data and architecture. This yields a scaling law for representation geometry; the source describes a theoretical result under these assumptions.

### Source excerpt

arXiv:2610.21820v2 Announce Type: replace Abstract: How the depth and width of neural networks shape learned representations is difficult to predict from the data and architecture. We use neural collapse (NC) as a probe of the representation geometry and express its first three properties, within-class variability collapse (NC1), simplex class means (NC2) and classifier alignment (NC3), through the learned feature kernel, extending a known identity for NC1 to NC2 and NC3. For deep linear networks with an exchangeable input kernel, the kernel equations yield a geometric law, exact within this theory, for the layerwise decrease of NC1. At large width $N$ the accumulated decrease becomes a function of the depth-to-width ratio $L/N$, with its rate determined self-consistently by the data and architecture, yielding a scaling law for representation geometry.

## Hybrid FedGraph: An efficient hybrid federated learning algorithm using graph convolutional neural network

DevFeed: [Hybrid FedGraph: An efficient hybrid federated learning algorithm using graph convolutional neural network](<https://devfeed.tech/articles/hybrid-fedgraph-an-efficient-hybrid-federated-learning-algorithm-using-graph-convolutional-neural-network-131503.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2404.09443>)

Author: Jaeyeon Jang, Diego Klabjan, Veena Mendiratta, Fanfei Meng

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Federated Learning](<https://devfeed.tech/topics/federated-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Deep neural networks](<https://devfeed.tech/topics/deep-neural-networks.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [clients](<https://devfeed.tech/tags/clients.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-privacy](<https://devfeed.tech/tags/data-privacy.md>), [federated](<https://devfeed.tech/tags/federated.md>), [federated-learning](<https://devfeed.tech/tags/federated-learning.md>), [graph](<https://devfeed.tech/tags/graph.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>)

### AI overview

FedGraph targets hybrid federated learning, a setting the authors describe as common but less studied than horizontal or vertical distributions. It uses a graph convolutional neural network to capture feature-sharing information while learning from a subset of clients, and a clustering algorithm to aggregate features produced by each client's deep neural network while preserving data privacy. The excerpt gives no performance results or further implementation details.

### Source excerpt

arXiv:2404.09443v2 Announce Type: replace-cross Abstract: Federated learning is an emerging paradigm for decentralized training of machine learning models on distributed clients, without revealing the data to the central server. Most existing works have focused on horizontal or vertical data distributions, where each client possesses different samples with shared features, or each client fully shares only sample indices, respectively. However, the hybrid scheme is much less studied, even though it is much more common in the real world. Therefore, in this paper, we propose a generalized algorithm, FedGraph, that introduces a graph convolutional neural network to capture feature-sharing information while learning features from a subset of clients. We also develop a simple but effective clustering algorithm that aggregates features produced by the deep neural networks of each client while preserving data privacy.

[Next page](<https://devfeed.tech/topics/deep-neural-networks.md?cursor=WyIyMDI2LTExLTEzVDA1OjAwOjAwKzAwOjAwIiwgImI3NmIyMzBmLWViMGEtNDMxMS04ZTEwLWY1NjVlZjYyOTdhYyJd>)