# linear-regression

Linear regression is a statistical modeling method that fits data using a function linear in its unknown parameters.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Statistical Inference with Ill-Conditioned Random Weight Neural Network Sieves

DevFeed: [Statistical Inference with Ill-Conditioned Random Weight Neural Network Sieves](<https://devfeed.tech/articles/statistical-inference-with-ill-conditioned-random-weight-neural-network-sieves-145176.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26410>)

Author: Giovanni Ballarin, Lyudmila Grigoryeva, Lukas Gonon

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [polynomials](<https://devfeed.tech/topics/polynomials.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [based](<https://devfeed.tech/tags/based.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [beta](<https://devfeed.tech/tags/beta.md>), [condition](<https://devfeed.tech/tags/condition.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [covariance](<https://devfeed.tech/tags/covariance.md>), [covering](<https://devfeed.tech/tags/covering.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [develop](<https://devfeed.tech/tags/develop.md>), [econ-em](<https://devfeed.tech/tags/econ-em.md>), [inference](<https://devfeed.tech/tags/inference.md>), [network](<https://devfeed.tech/tags/network.md>), [neural](<https://devfeed.tech/tags/neural.md>), [neural-network](<https://devfeed.tech/tags/neural-network.md>), [random](<https://devfeed.tech/tags/random.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [weight](<https://devfeed.tech/tags/weight.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The paper shows that random weight neural network sieves become increasingly ill-conditioned as their dimension grows: under polynomial covering and Lipschitz conditions, the population covariance eigenvalues have geometry-dependent polynomial upper bounds, so the smallest eigenvalue vanishes. It develops theory that accounts for this behavior, including faster-than-Monte-Carlo L² approximation and stratified-sampling L∞ bounds under a β-Hölder source condition, plus ridge least-squares consistency guarantees based on effective dimension and stability conditions. For inference, it establishes asymptotic linearity and Gaussian limits for linear directions despite the vanishing eigenvalue; directional nondegeneracy is expressed through a ridge-normalized Rayleigh quotient and extended to smooth functionals using Riesz representation. Consistent ridge-adjusted variance estimation enables feasible studentized inference. The abstract reports numerical experiments on conditioning and finite-sample behavior.

### Source excerpt

arXiv:2611.26410v1 Announce Type: cross Abstract: Random weight neural networks (RWNNs) are shallow networks whose hidden-layer parameters are randomly generated and then kept fixed, while only the output coefficients are estimated. Conditional on the sampled parameters, RWNN regression is a linear sieve problem with random approximation space. We show that such random sieves cannot remain uniformly well-conditioned as their dimension grows. For parametrized feature families satisfying polynomial covering and Lipschitz conditions, the eigenvalues of the population covariance matrix are proven to have polynomial upper bounds determined by the covering geometry of the parameter space, leading to a vanishing smallest eigenvalue. We develop approximation, estimation, and inference theory that explicitly treats the ill-conditioning. Under a $\beta$-H\"older source condition, we obtain both faster-than-Monte-Carlo $L^2$ approximation rates and new $L^\infty$ approximation bounds based on stratified sampling. We then derive improved consistency guarantees for ridge-regularized least squares driven by ridge effective dimension, spectral, and leverage-based stability conditions. For inference, we establish asymptotic linearity and Gaussian limits for linear functional directions while allowing the smallest eigenvalue of the population covariance matrix to vanish. Nondegeneracy is directional: we characterize it with a ridge-normalized Rayleigh quotient and extend it to smooth functionals via Riesz representation. Consistency of ridge-adjusted variance estimators is proved, yielding feasible studentized inference. Numerical experiments illustrate the deterioration of random-feature conditioning and the finite-sample behavior of the resulting estimation and inference procedures.

## DSFlow: A Neuro-Symbolic Data Workflow Agent with Statistical Diagnostic Guardrails

DevFeed: [DSFlow: A Neuro-Symbolic Data Workflow Agent with Statistical Diagnostic Guardrails](<https://devfeed.tech/articles/dsflow-a-neuro-symbolic-data-workflow-agent-with-statistical-diagnostic-guardrails-145469.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25111>)

Author: Junfeng Nie, Xiaohui Chen

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Data Science](<https://devfeed.tech/topics/data-science.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [clustering](<https://devfeed.tech/topics/clustering.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data](<https://devfeed.tech/tags/data.md>), [diagnostic](<https://devfeed.tech/tags/diagnostic.md>), [guardrails](<https://devfeed.tech/tags/guardrails.md>), [langgraph](<https://devfeed.tech/tags/langgraph.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [regression](<https://devfeed.tech/tags/regression.md>), [reports](<https://devfeed.tech/tags/reports.md>), [skills](<https://devfeed.tech/tags/skills.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

DSFlow combines reusable skills selected by an LLM with deterministic statistical checks to guide data-science workflows. Its symbolic validation can accept a proposed method, ask for clarification or replanning, override an unsuitable choice, or provide a safe fallback. Regression and clustering case studies show diagnostic evidence correcting method choices, with decisions recorded in auditable reports. The supplied text is an abstract and does not provide implementation details or case-study results.

### Source excerpt

arXiv:2611.25111v1 Announce Type: new Abstract: Data science workflows often require tools that presume non-trivial technical knowledge. Large language models (LLMs) reduce this barrier, but a gap remains between flexible general-purpose agents and statistically controlled pipelines. We introduce DSFlow, a LangGraph-based neuro-symbolic data-agent platform that combines reusable, LLM-selected skills with deterministic diagnostic guardrails. Skills provide extensible task knowledge. Symbolic validation can accept a proposed method, request clarification or replanning, override an unsuitable method, or supply a safe fallback. Regression and clustering case studies illustrate how DSFlow uses diagnostic evidence to correct method choices and documents those decisions in auditable reports.

## Inverse ECG Reconstruction via a Preconditioned Variational Quantum Linear Solver

DevFeed: [Inverse ECG Reconstruction via a Preconditioned Variational Quantum Linear Solver](<https://devfeed.tech/articles/inverse-ecg-reconstruction-via-a-preconditioned-variational-quantum-linear-solver-145057.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25450>)

Author: Xizhuo Zhang, Chao Lu, Bing Yao

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Quantum Computing](<https://devfeed.tech/topics/quantum-computing.md>), [Medical imaging](<https://devfeed.tech/topics/medical-imaging.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [applications](<https://devfeed.tech/tags/applications.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [ecg](<https://devfeed.tech/tags/ecg.md>), [experimental](<https://devfeed.tech/tags/experimental.md>), [framework](<https://devfeed.tech/tags/framework.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [imaging](<https://devfeed.tech/tags/imaging.md>), [linear](<https://devfeed.tech/tags/linear.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [quantum-classical](<https://devfeed.tech/tags/quantum-classical.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>)

### AI overview

The paper proposes a hybrid method for inverse ECG reconstruction, which infers heart-surface potentials from body-surface measurements. To address the dense, ill-conditioned systems involved, it first uses Laplacian-regularized pre-encoding to reduce system dimension and improve conditioning, then solves the reduced system with a coherent variational quantum linear solver (VQLS). The authors state that a theoretical proof shows the procedure preserves the original problem's exact solution, and report accurate reconstruction in experiments on a realistic heart-torso geometry. The abstract does not provide comparative performance or computational-cost results.

### Source excerpt

arXiv:2611.25450v1 Announce Type: new Abstract: Inverse electrocardiography (ECG) reconstructs heart-surface potentials from body-surface measurements, enabling noninvasive mapping of cardiac electrical activity. However, the reconstruction requires solving dense and ill-conditioned linear systems, whose computational cost grows rapidly with the refined heart mesh. Quantum computing offers new opportunities for solving such systems. This paper presents a hybrid quantum-classical framework for inverse ECG reconstruction, which, to the best of our knowledge, is the first to apply a quantum linear solver to this problem. We first propose a Laplacian-regularized pre-encoding procedure to reduce the dimension of the system while improving its conditioning. Second, the resulting reduced system is encoded and solved using a coherent variational quantum linear solver (VQLS). Finally, we provide a rigorous theoretical proof establishing that the proposed framework preserves the exact solution of the original reconstruction problem throughout all stages of the computational process. Experimental results on a realistic heart-torso geometry demonstrate that our approach holds significant ability for accurate inverse ECG reconstruction. These findings highlight the potential of quantum solvers for noninvasive cardiac imaging, providing a foundation for further research into quantum-enhanced cardiac assessment and other healthcare applications.

## Agnostic Smoothed Online Regression with Adversarial Responses

DevFeed: [Agnostic Smoothed Online Regression with Adversarial Responses](<https://devfeed.tech/articles/agnostic-smoothed-online-regression-with-adversarial-responses-144936.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.32478>)

Author: Xuanyu Chen, Yue Yu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [experts](<https://devfeed.tech/tags/experts.md>), [hedge](<https://devfeed.tech/tags/hedge.md>), [linear](<https://devfeed.tech/tags/linear.md>), [lower-bound](<https://devfeed.tech/tags/lower-bound.md>), [matching](<https://devfeed.tech/tags/matching.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [noise](<https://devfeed.tech/tags/noise.md>), [online](<https://devfeed.tech/tags/online.md>), [open](<https://devfeed.tech/tags/open.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [regression](<https://devfeed.tech/tags/regression.md>), [response](<https://devfeed.tech/tags/response.md>), [responses](<https://devfeed.tech/tags/responses.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>)

### AI overview

Hedge-Cover achieves sublinear regret of Õ(√(Pdim(F) C_cov T)) for smoothed online regression with bounded adversarial responses and function classes of bounded pseudo-dimension. It combines a constructed family of experts with Hedge, using a prior tied to disagreements between a consistent selector and target; covariate smoothness bounds those disagreements. The paper reports a matching lower bound for linear predictors, resolving an open question about minimax adaptive regret. In the well-specified setting with conditionally ν²-subGaussian response noise, it gives a minimax expected regret lower bound Ω((1∨ν)√((C_cov−1)dT)) for VC dimension d, matched in its dependence on ν by an ERM upper bound.

### Source excerpt

arXiv:2609.32478v2 Announce Type: replace-cross Abstract: We study smoothed online prediction with bounded adversarial responses. This widely studied framework bridges i.i.d. sampling and adversarial covariate selection through a smoothness parameter $\textsf{C}_{\textsf{cov}}$, which bounds conditional covariate densities relative to a fixed, unknown base measure. We propose Hedge-Cover, an information-theoretic algorithm that achieves sublinear regret $\widetilde{O}(\sqrt{\text{Pdim}(\mathcal{F}) \textsf{C}_{\textsf{cov}} T})$ for function classes with bounded pseudo-dimension. The algorithm aggregates a carefully constructed family of experts using Hedge, with a prior that links regret to the number of disagreements between a consistent selector and a target function. We bound this number by exploiting covariate smoothness. This answers an open problem posed in \cite{blanchard2025agnostic} on the minimax optimal adaptive regret of the smoothed online regression problem. We establish a matching lower bound for the class of linear predictors. The main intricacy of the lower bound lies in explicitly constructing a challenging sequential covariate distribution supported on mutually orthogonal hyperplanes. This construction may be of independent technical interest. Finally, we revisit the well-specified setting and quantify the effect of response noise. For conditionally $\nu^2$-subGaussian responses, we extend the existing lower bound under realizable responses by showing that the minimax expected regret is $\Omega((1\vee \nu)\sqrt{(\textsf{C}_{\textsf{cov}}-1)dT})$ for a function class of VC dimension $d$. A corresponding upper bound for ERM matches this dependence on $\nu$.

## Meta-RL with Bayesian Linear Task Models

DevFeed: [Meta-RL with Bayesian Linear Task Models](<https://devfeed.tech/articles/meta-rl-with-bayesian-linear-task-models-142501.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2512.20974>)

Author: Jingyang You, Hanna Kurniawati

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>)

Tags: [basis](<https://devfeed.tech/tags/basis.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [bayesian-inference](<https://devfeed.tech/tags/bayesian-inference.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [learning](<https://devfeed.tech/tags/learning.md>), [linear](<https://devfeed.tech/tags/linear.md>), [manipulation](<https://devfeed.tech/tags/manipulation.md>), [meta](<https://devfeed.tech/tags/meta.md>), [models](<https://devfeed.tech/tags/models.md>), [mujoco](<https://devfeed.tech/tags/mujoco.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rl](<https://devfeed.tech/tags/rl.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [zero-shot](<https://devfeed.tech/tags/zero-shot.md>)

### AI overview

GLiBRL is a meta-reinforcement-learning framework for adapting to unseen tasks that replaces variational inference with conjugate Bayesian updates. It combines generalised linear task models with learnable nonlinear basis functions to make exact sequential updates to task parameters and model noise, and computes a closed-form marginal likelihood. Its permutation-invariant updates can be used with both off-policy and on-policy algorithms. The abstract also reports an exact kernel identity linking distances between learned task representations to kernel discrepancies over task contexts. In comparisons with eight representative or recent methods, GLiBRL had the highest aggregate zero-shot test performance on the MuJoCo locomotion and MetaWorld manipulation benchmarks.

### Source excerpt

arXiv:2512.20974v5 Announce Type: replace-cross Abstract: Deep Bayesian reinforcement learning adapts to unseen tasks by inferring latent transition and reward models, but existing methods typically rely on variational posteriors and evidence lower bounds, introducing approximation error and unstable task representations. We introduce GLiBRL, a deep Bayesian RL framework that combines generalised linear task models with learnable non-linear basis functions. GLiBRL features conjugate Bayesian inference, yielding exact, sequential posterior updates over task parameters and model noise, together with a closed-form marginal likelihood that eliminates variational inference. The update is naturally permutation-invariant, allowing GLiBRL to integrate with both off- and on-policy algorithms. GLiBRL also learns task representation admitting an exact kernel identity, relating distances between task representations to kernel discrepancies over the task contexts. Compared against eight representative or recent meta reinforcement learning methods, GLiBRL achieves the highest aggregate zero-shot test performance on both the MuJoCo locomotion and MetaWorld manipulation benchmarks.

## A Free Lunch: Gradient Averaging Achieves Optimal Statistical Efficiency in Newton Sketch

DevFeed: [A Free Lunch: Gradient Averaging Achieves Optimal Statistical Efficiency in Newton Sketch](<https://devfeed.tech/articles/a-free-lunch-gradient-averaging-achieves-optimal-statistical-efficiency-in-newton-sketch-143428.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23671>)

Author: Haoxuan Wang, Jianliang Ye, Sen Na

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>), [Covariance](<https://devfeed.tech/topics/covariance.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [central-limit-theorem](<https://devfeed.tech/tags/central-limit-theorem.md>), [covariance](<https://devfeed.tech/tags/covariance.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [inference](<https://devfeed.tech/tags/inference.md>), [linear](<https://devfeed.tech/tags/linear.md>), [logistic-regression](<https://devfeed.tech/tags/logistic-regression.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [minimax](<https://devfeed.tech/tags/minimax.md>), [newton](<https://devfeed.tech/tags/newton.md>), [online](<https://devfeed.tech/tags/online.md>), [randomness](<https://devfeed.tech/tags/randomness.md>), [regression](<https://devfeed.tech/tags/regression.md>), [sketch](<https://devfeed.tech/tags/sketch.md>), [stat-co](<https://devfeed.tech/tags/stat-co.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>), [trade](<https://devfeed.tech/tags/trade.md>)

### AI overview

For online sketched Newton methods, moving-average gradient estimates can reduce the statistical cost of randomized sketching without extra oracle evaluations or changing the inner solver's contraction bound. The last iterate's limiting covariance is no larger than without gradient averaging, while the averaged iterate reaches the minimax optimal covariance matching exact Newton. The analysis handles the loss of the martingale difference structure by studying coupled two-timescale recursions; experiments on linear and logistic regression support the covariance results.

### Source excerpt

arXiv:2611.23671v1 Announce Type: cross Abstract: We study statistical inference for online sketched Newton methods, in which each Newton system is approximately solved by an inner randomized sketching solver. Existing work reveals a computational--statistical trade-off: the sketching solver (and its accelerated variants) reduces the per-step cost of Newton methods from $O(d^3)$ to $O(d^2)$, but the randomness introduced by sketching also inflates the limiting covariance of both the last and the averaged Newton iterates. In this work, we address this trade-off via gradient averaging. We show that simply replacing the stochastic gradient with a moving average at each step can mitigate, and in some cases eliminate, the trade-off in Newton sketch. Importantly, gradient averaging requires no additional oracle evaluations and leaves the contraction bound of the inner sketching solver unchanged. In particular, we prove that the last sketched Newton iterate with gradient averaging attains a limiting covariance no larger than that without gradient averaging, while the averaged iterate even attains the minimax optimal limiting covariance, matching that of the exact Newton method. To our knowledge, this is the first analysis showing that randomness introduced to improve computational efficiency need not compromise statistical efficiency as long as proper gradient estimates are used. The key analytical challenge is that gradient averaging destroys the crucial martingale difference structure underlying central limit theorem arguments. We overcome this challenge by analyzing the coupled two-timescale recursions governing the iterate and the gradient estimator. Numerical experiments on linear and logistic regression support our theoretical covariance comparisons.

## Improved Leverage Score Sampling for Constrained Active Linear Regression

DevFeed: [Improved Leverage Score Sampling for Constrained Active Linear Regression](<https://devfeed.tech/articles/improved-leverage-score-sampling-for-constrained-active-linear-regression-143447.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23801>)

Author: Aarshvi Gajjar, Syamantak Kumar, Aditya Makkar, Christopher Musco, Joey Zhou

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-na](<https://devfeed.tech/tags/cs-na.md>), [learning](<https://devfeed.tech/tags/learning.md>), [leverage](<https://devfeed.tech/tags/leverage.md>), [linear-regression](<https://devfeed.tech/tags/linear-regression.md>), [math-na](<https://devfeed.tech/tags/math-na.md>), [regression](<https://devfeed.tech/tags/regression.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

The paper proposes ellipsoid-regularized leverage score sampling for active constrained linear regression. It approximates the constraint set with an outer ellipsoid and uses its shape to focus queries on directions important under the constraint geometry. The resulting query bound depends on an anisotropic effective dimension no greater than the ambient dimension: O(d_eff(C) log d / ε²) queries suffice for a (1+ε)-approximate objective value. For convex constraints, the authors improve the ε dependence to 1/ε. Their lower bounds show d/ε² dependence is unavoidable without convexity, and d_eff(C)/ε can be necessary for ellipsoidal constraints in some regimes. Experiments on synthetic regression and parametric heat conduction report lower regression error than standard baselines at similar query budgets.

### Source excerpt

arXiv:2611.23801v1 Announce Type: cross Abstract: In this paper we study active learning for constrained linear regression. Given a design matrix $A\in\mathbb{R}^{n\times d}$, a constraint set $C\subseteq\mathbb{R}^d$, and query access to $b\in\mathbb{R}^n$, we seek an approximate solution to $\min_{x\in C}\|Ax-b\|_2^2$ using as few queries as possible. This problem arises in many scientific and engineering applications, and randomized leverage score sampling provides a powerful method to solve it. However, existing leverage score methods ignore the constraint or encode it through ridge regularization, suffering unfavorable query complexity for anisotropic constraint sets. We propose ellipsoid-regularized leverage scores, where we approximate the constraint set by an outer ellipsoid and use its shape matrix to bias sampling toward directions that matter under the constraint geometry. We prove that the query complexity is governed by an anisotropic effective dimension $d_{\text{eff}}(C)\le d$. With appropriate parameters, $O(d_{\text{eff}}(C)\log d/\varepsilon^2)$ queries suffice to return a point in $C$ with objective value at most $(1+\varepsilon)$ times the optimum. This bound can substantially improve on the standard leverage-score bound, since anisotropic constraints can make $d_{\text{eff}}(C)$ much smaller than $d$. For convex constraints, we improve the dependence on $\varepsilon$ from $1/\varepsilon^2$ to $1/\varepsilon$. We also prove two lower bounds: first, the $d/\varepsilon^2$ dependence is unavoidable without convexity assumptions on $C$, and second, in some regimes, the effective dimension dependence $d_{\mathrm{eff}}(C)/\varepsilon$ is necessary for ellipsoidal constraints. Experiments on synthetic regression problems and a parametric heat conduction problem show lower regression error than standard baselines for similar query budget.

## High-Dimensional Tail Index Regression

DevFeed: [High-Dimensional Tail Index Regression](<https://devfeed.tech/articles/high-dimensional-tail-index-regression-141514.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2403.01318>)

Author: Yuya Sasaki, Jing Tao, Yulong Wang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [X (Twitter)](<https://devfeed.tech/topics/twitter.md>)

Tags: [consistency](<https://devfeed.tech/tags/consistency.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [econ-em](<https://devfeed.tech/tags/econ-em.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [health](<https://devfeed.tech/tags/health.md>), [inference](<https://devfeed.tech/tags/inference.md>), [media](<https://devfeed.tech/tags/media.md>), [model](<https://devfeed.tech/tags/model.md>), [monte-carlo](<https://devfeed.tech/tags/monte-carlo.md>), [posts](<https://devfeed.tech/tags/posts.md>), [regression](<https://devfeed.tech/tags/regression.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [social-media](<https://devfeed.tech/tags/social-media.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [text](<https://devfeed.tech/tags/text.md>), [twitter](<https://devfeed.tech/tags/twitter.md>), [viral](<https://devfeed.tech/tags/viral.md>), [x](<https://devfeed.tech/tags/x.md>)

### AI overview

The paper introduces high-dimensional tail index regression for studying power-law outcomes. It proposes a regularized estimator, establishes consistency and a convergence rate, then constructs a debiased estimator for inference and proves asymptotic normality. Monte Carlo simulations support the theoretical results. The methods are applied to text analysis of viral posts on X (formerly Twitter) and to abnormally high health expenditures.

### Source excerpt

arXiv:2403.01318v4 Announce Type: replace-cross Abstract: Motivated by the empirical observation of power-law distributions in the credits (e.g., ``likes'') of viral posts on social media, we introduce a high-dimensional tail index regression model and propose methods for estimation and inference of its parameters. First, we propose a regularized estimator, establish its consistency, and derive its convergence rate. Second, we construct a debiased version of the regularized estimator to facilitate inference and prove its asymptotic normality. Monte Carlo simulation studies corroborate our theoretical findings. We apply these methods (1) to the text analysis of viral posts on X (formerly Twitter) and (2) to the analysis of abnormally high health expenditures.

## Tight Sequential Confidence Intervals for Online Kernel Regression

DevFeed: [Tight Sequential Confidence Intervals for Online Kernel Regression](<https://devfeed.tech/articles/tight-sequential-confidence-intervals-for-online-kernel-regression-138929.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19772>)

Author: Haichen Hu, David Simchi-Levi

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [2021](<https://devfeed.tech/tags/2021.md>), [2023](<https://devfeed.tech/tags/2023.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [building](<https://devfeed.tech/tags/building.md>), [case](<https://devfeed.tech/tags/case.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [d](<https://devfeed.tech/tags/d.md>), [delta](<https://devfeed.tech/tags/delta.md>), [design](<https://devfeed.tech/tags/design.md>), [designs](<https://devfeed.tech/tags/designs.md>), [domain](<https://devfeed.tech/tags/domain.md>), [elements](<https://devfeed.tech/tags/elements.md>), [error](<https://devfeed.tech/tags/error.md>), [f](<https://devfeed.tech/tags/f.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [math-pr](<https://devfeed.tech/tags/math-pr.md>), [online](<https://devfeed.tech/tags/online.md>), [regression](<https://devfeed.tech/tags/regression.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

The paper determines the worst-case minimax width of online confidence intervals for RKHS functions, resolving a discrete-time open problem. For a kernel formed by summing a linear kernel and a periodic Sobolev kernel, it shows that for sufficiently large n and δ ∈ (0, 1/4], the width scales as B + (R/λ)√(γₙ + log(1/δ)), where γₙ is maximal information gain and grows polynomially in n. It also gives an adaptive design on a fixed two-dimensional domain under Gaussian noise where normalized error at a prespecified point is at least order B + (R/λ)√γₙ with constant probability. This rules out polylogarithmic width in the worst case over kernels and online designs.

### Source excerpt

arXiv:2611.19772v1 Announce Type: cross Abstract: We characterize, in the worst case over kernels, the sharp minimax width of online confidence intervals for elements of a reproducing kernel Hilbert space (RKHS). This resolves the open problem posed by Vakili et al. (2021) in its original discrete-time form. A learner observes $y_i=f(x_i)+\epsilon_i$ at adaptively chosen points, with $\|f\|_{\mathcal H_k}\le B$ and i.i.d. $R$-sub-Gaussian noise, and reports the interval $|f(x)-\mu_n(x)|\le\rho_n(\delta)\sigma_n(x)$, where $\mu_n$ and $\sigma_n$ are the posterior mean and standard deviation of a surrogate Gaussian process model with regularization parameter $\lambda$. Writing $\rho_n^*(\delta)$ for the minimax confidence width for a fixed kernel, we show that, for the sum of a linear kernel and a periodic Sobolev kernel, all sufficiently large $n$, and all $\delta\in(0,1/4]$, $\rho_n^*(\delta)\asymp B+(R/\lambda)\sqrt{\gamma_n+\log(1/\delta)}$, where $\gamma_n$ is the maximal information gain and grows polynomially in $n$. Building on Lattimore (2023), we construct an adaptive design on a fixed two-dimensional domain for which, under Gaussian noise, the normalized error at a prespecified point is $\Omega(B+(R/\lambda)\sqrt{\gamma_n})$ with constant probability. This rules out the conjectured polylogarithmic width in the worst case over kernels and online designs.

## The Manifold-Index Model: provably and efficiently avoiding the statistical and computational curse of dimensionality

DevFeed: [The Manifold-Index Model: provably and efficiently avoiding the statistical and computational curse of dimensionality](<https://devfeed.tech/articles/the-manifold-index-model-provably-and-efficiently-avoiding-the-statistical-and-computational-curse-of-dimensionality-139019.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20275>)

Author: Yantao Wu, Mauro Maggioni

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [computational complexity](<https://devfeed.tech/topics/computational-complexity.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [polynomials](<https://devfeed.tech/topics/polynomials.md>)

Tags: [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [curse-of-dimensionality](<https://devfeed.tech/tags/curse-of-dimensionality.md>), [dimension](<https://devfeed.tech/tags/dimension.md>), [error](<https://devfeed.tech/tags/error.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [intrinsic](<https://devfeed.tech/tags/intrinsic.md>), [link](<https://devfeed.tech/tags/link.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [noise](<https://devfeed.tech/tags/noise.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [projection](<https://devfeed.tech/tags/projection.md>), [regression](<https://devfeed.tech/tags/regression.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>), [tangent](<https://devfeed.tech/tags/tangent.md>), [vector](<https://devfeed.tech/tags/vector.md>)

### AI overview

The Manifold-Index Model assumes predictors lie in a tube around an unknown m-dimensional manifold and that the mean response depends on their nearest-point projection through a possibly non-injective link. MANOR estimates tangent planes using multiscale PCA on predictors with similar responses, then refines them by pooling local slope outer products; its neighborhoods narrow tangentially while retaining full normal fibers. Under the stated geometric and sampling conditions, a Lipschitz link differential, bounded response noise, and p >= m, the reported mean tangent-space error is n^{-2/(m+4)} and integrated squared prediction error is n^{-4/(m+4)}, up to logarithmic factors. With intrinsic dimension and specified model and scale quantities fixed, required sample size grows polynomially in D+p and computation polynomially in n, D, and p. Experiments cover curves, surfaces with vector responses, and hitting-probability prediction under an approximate model.

### Source excerpt

arXiv:2611.20275v1 Announce Type: cross Abstract: We study vector-valued regression with predictors filling a tube around an unknown $m$-dimensional manifold $\mathcal{M}$ in $\mathbb{R}^D$. In the Manifold-Index Model, the mean response is $F=f\circ\Pi_{\mathcal{M}}$, where $\Pi_{\mathcal{M}}$ is the nearest-point projection onto $\mathcal{M}$ and $f:\mathcal{M}\to\mathbb{R}^p$ is a possibly non-injective link function. Multiscale ANisotropic lOcal Regression (MANOR) estimates tangent planes by multiscale PCA of predictors with similar responses, then refines them by pooling outer products of local slopes. The neighborhoods narrow tangentially while retaining complete normal fibers. We assume a Lipschitz link differential and bounded response noise. For $p\ge m$, under our geometric and sampling conditions, $n$ independent observations give mean tangent-space error of order $n^{-2/(m+4)}$ and integrated squared prediction error at the $m$-dimensional nonparametric rate $n^{-4/(m+4)}$, up to logarithmic factors. In a scaling regime with fixed intrinsic dimension, model and scale parameters, and ratio of manifold volume to the $m$-th power of its reach, the required sample size grows polynomially in $D+p$, and computation is polynomial in $n$, $D$ and $p$. Numerical experiments on curves with non-monotone links and surfaces with vector-valued responses compare observed convergence rates with the theoretical bounds; a Langevin dynamics example tests prediction of hitting probabilities under an approximate manifold-index model.

## Ensemble Modeling for Time Series Forecasting: an Adaptive Robust Optimization Approach

DevFeed: [Ensemble Modeling for Time Series Forecasting: an Adaptive Robust Optimization Approach](<https://devfeed.tech/articles/ensemble-modeling-for-time-series-forecasting-an-adaptive-robust-optimization-approach-138704.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2304.04308>)

Author: L\\'eonard Boussioux, Henry Mao, Dimitris Bertsimas

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Time Series](<https://devfeed.tech/topics/time-series.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [Temporal data](<https://devfeed.tech/topics/temporal-data.md>), [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [air-pollution](<https://devfeed.tech/tags/air-pollution.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [energy-consumption](<https://devfeed.tech/tags/energy-consumption.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [linear](<https://devfeed.tech/tags/linear.md>), [linear-regression](<https://devfeed.tech/tags/linear-regression.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [methodology](<https://devfeed.tech/tags/methodology.md>), [model-weights](<https://devfeed.tech/tags/model-weights.md>), [modeling](<https://devfeed.tech/tags/modeling.md>), [models](<https://devfeed.tech/tags/models.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [regression](<https://devfeed.tech/tags/regression.md>), [risk](<https://devfeed.tech/tags/risk.md>), [robust](<https://devfeed.tech/tags/robust.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [shift](<https://devfeed.tech/tags/shift.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [time-series](<https://devfeed.tech/tags/time-series.md>), [time-series-forecasting](<https://devfeed.tech/tags/time-series-forecasting.md>)

### AI overview

The paper uses Adaptive Robust Optimization (ARO) to build linear regression ensembles with weights that adapt over time as time-series data shift. In real-world forecasting applications for air pollution, energy consumption, and tropical cyclone intensity, the method reports 16-26% lower RMSE and 14-28% lower 15% conditional value at risk than the best ensemble member in hindsight, and improves on other compared ensemble methods on most metrics. Across eight synthetic disturbance scenarios, it has lower mean RMSE than rolling ridge in seven, with statistically significant improvements in four. A limitation is that an abrupt regime shift appearing only after model selection remains difficult for the approach.

### Source excerpt

arXiv:2304.04308v2 Announce Type: replace Abstract: Accurate time-series forecasting is critical for many problems involving temporal data. Ensemble modeling is a well-established technique that leverages multiple predictive models to improve accuracy and robustness, since a single predictor's performance can vary widely due to shifts in the underlying data distribution. This paper proposes a new methodology for building robust ensembles of time series forecasting models. Our approach uses Adaptive Robust Optimization (ARO) to construct a linear regression ensemble whose model weights adapt over time. We demonstrate the effectiveness of our method through synthetic experiments covering eight disturbance scenarios, from no drift to abrupt regime shifts, and real-world applications, including air pollution management, energy consumption forecasting, and tropical cyclone intensity forecasting. In the real-world applications, our adaptive ensembles outperform the best ensemble member in hindsight by 16-26% in root mean square error (RMSE) and 14-28% in conditional value at risk at the 15% level and improve over the other ensemble methods we compare on most metrics. In the synthetic experiments, they have lower mean RMSE than a rolling-ridge comparator in seven of the eight scenarios, significantly in four, while an abrupt regime shift that first appears after model selection remains a limitation.

## Pretty Linear Liars: Unregularized Probes Favor Intervention-Unfaithful Features

DevFeed: [Pretty Linear Liars: Unregularized Probes Favor Intervention-Unfaithful Features](<https://devfeed.tech/articles/pretty-linear-liars-unregularized-probes-favor-intervention-unfaithful-features-137565.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20371>)

Author: Karl Vilhelmsson Emneby, Julian Yocum, Cameron Allen

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [causality](<https://devfeed.tech/topics/causality.md>)

Tags: [causality](<https://devfeed.tech/tags/causality.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [features](<https://devfeed.tech/tags/features.md>), [interpretability](<https://devfeed.tech/tags/interpretability.md>), [linear-probing](<https://devfeed.tech/tags/linear-probing.md>), [llama](<https://devfeed.tech/tags/llama.md>), [regression](<https://devfeed.tech/tags/regression.md>), [transformer](<https://devfeed.tech/tags/transformer.md>), [variance](<https://devfeed.tech/tags/variance.md>)

### AI overview

The study finds that unregularized OLS probes can favor features that are decodable but do not faithfully reflect the model's active use of them: theoretically, OLS amplifies low-variance activation directions spuriously correlated with the target. In experiments with a toy transformer and Llama-2-7B, intervention faithfulness rises with ridge regularization to an optimum, then falls as regularization increases further. At that optimum, probe features resemble those learned by directly optimizing intervention outcomes in both direction and variance. The results support regularizing regression probes, while indicating that the strength matters.

### Source excerpt

arXiv:2611.20371v1 Announce Type: cross Abstract: Linear probing is a common technique in mechanistic interpretability despite the fact that the decodability of a given feature does not necessarily imply the network uses that feature. Intervention faithfulness, not merely decodability, is the appropriate criterion for whether a feature serves a defined active role in the model. Meanwhile, probe regularization has been used for feature discovery, but the impact of regularization on feature causality remains unclear. We study probe regularization systematically, starting with unregularized ordinary least squares (OLS) regression and considering increasing amounts of regularization via ridge regression. We show theoretically that OLS probes amplify low-variance directions in activation space that have spurious correlations with the target feature, whereas regularization reduces influence from these directions. Empirically, we find in a toy transformer and Llama-2-7B that intervention faithfulness increases up to a point, and then decreases again, and that the optimal regularization strength leads to features bearing striking similarity (in both direction and variance) to those learned by optimizing for intervention outcomes directly. Together, these results provide mechanistic justification for regularizing regression probes.

## Denoising the US Census: Succinct Block Hierarchical Regression

DevFeed: [Denoising the US Census: Succinct Block Hierarchical Regression](<https://devfeed.tech/articles/denoising-the-us-census-succinct-block-hierarchical-regression-135672.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.10099>)

Author: Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Adam Sealfon

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [2020](<https://devfeed.tech/tags/2020.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [census](<https://devfeed.tech/tags/census.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [denoising](<https://devfeed.tech/tags/denoising.md>), [differential-privacy](<https://devfeed.tech/tags/differential-privacy.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [error](<https://devfeed.tech/tags/error.md>), [housing](<https://devfeed.tech/tags/housing.md>), [linear](<https://devfeed.tech/tags/linear.md>), [linear-regression](<https://devfeed.tech/tags/linear-regression.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [noise](<https://devfeed.tech/tags/noise.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [race](<https://devfeed.tech/tags/race.md>), [regression](<https://devfeed.tech/tags/regression.md>), [research](<https://devfeed.tech/tags/research.md>), [spending](<https://devfeed.tech/tags/spending.md>), [structure](<https://devfeed.tech/tags/structure.md>), [us](<https://devfeed.tech/tags/us.md>)

### AI overview

BlueDown postprocesses the 2020 Census's differentially private measurements, combining linear regression with constrained optimization and using measurement structure to improve efficiency. On Census data, it reduced error by 21-36% for county-level race groupings and 9-58% across county-level groupings; at the tract level, housing-type grouping error fell by 38-61%. The abstract reports these improvements without added confidentiality cost or loss of computational efficiency.

### Source excerpt

arXiv:2603.10099v2 Announce Type: replace Abstract: The 2020 United States Census adopted differential privacy to protect individual confidentiality, adding calibrated noise to summary demographic measurements that are then reconciled into population tables. We developed BlueDown, a postprocessing algorithm that combines techniques from linear regression and constrained optimization while exploiting measurement structure for efficiency. On 2020 Census data, BlueDown reduced the error of groupings by race at the county level by 21-36% and of all county-level groupings by 9-58%; at the tract level, it reduced error of groupings by housing type by 38-61%. These gains came at no cost to confidentiality or computational efficiency, and would improve the validity of research relying on this definitive dataset and analyses used to guide the distribution of over $1.5 trillion in annual federal spending.

## Benign Overfitting for Regression with Trained Two-Layer ReLU Networks

DevFeed: [Benign Overfitting for Regression with Trained Two-Layer ReLU Networks](<https://devfeed.tech/articles/benign-overfitting-for-regression-with-trained-two-layer-relu-networks-134943.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2410.06191>)

Author: Junhyung Park, Patrick Bloebaum, Shiva Prasad Kasiviswanathan

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [mlp](<https://devfeed.tech/topics/mlp.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [bounded](<https://devfeed.tech/tags/bounded.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [flow](<https://devfeed.tech/tags/flow.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [overfitting](<https://devfeed.tech/tags/overfitting.md>), [regression](<https://devfeed.tech/tags/regression.md>), [relu](<https://devfeed.tech/tags/relu.md>), [risk](<https://devfeed.tech/tags/risk.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [two-layer](<https://devfeed.tech/tags/two-layer.md>)

### AI overview

For least-squares regression trained by gradient flow, the paper reports benign overfitting in finite-width, two-layer fully connected ReLU networks within the neural tangent kernel regime. Its generalization result assumes only that the regression function and noise are bounded. The authors analyze excess risk by separating estimation and approximation errors, treating gradient flow as an implicit regularizer; under the same setting, they also establish that the trained network overfits the data.

### Source excerpt

arXiv:2410.06191v2 Announce Type: replace-cross Abstract: We study the least-square regression problem with a two-layer fully-connected neural network, with ReLU activation function, trained by gradient flow. Our first result is a generalization result, that requires no assumptions on the underlying regression function or the noise other than that they are bounded. We operate in the neural tangent kernel regime, and our generalization result is developed via a decomposition of the excess risk into estimation and approximation errors, viewing gradient flow as an implicit regularizer. This decomposition in the context of neural networks is a novel perspective of gradient descent, and helps us avoid uniform convergence traps. In this work, we also establish that under the same setting, the trained network overfits to the data. Together, these results, establishes the first result on benign overfitting for finite-width ReLU networks for arbitrary regression functions.

## Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning

DevFeed: [Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning](<https://devfeed.tech/articles/training-dynamics-of-softmax-self-attention-fast-global-convergence-via-preconditioning-132762.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2603.01514>)

Author: Gautam Goel, Mahdi Soltanolkotabi, Peter Bartlett

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [model architecture](<https://devfeed.tech/topics/model-architecture.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [attention](<https://devfeed.tech/tags/attention.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [global](<https://devfeed.tech/tags/global.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [infinite](<https://devfeed.tech/tags/infinite.md>), [layers](<https://devfeed.tech/tags/layers.md>), [linear](<https://devfeed.tech/tags/linear.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [model](<https://devfeed.tech/tags/model.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [psd](<https://devfeed.tech/tags/psd.md>), [risk](<https://devfeed.tech/tags/risk.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

In the infinite-data limit, the paper connects linear regression training in multilayer softmax self-attention to matrix factorization, motivating Hera, a first-order optimizer that reparameterizes and rescales the model to keep empirical loss gradients close to those of the auxiliary factorization problem. Hera's reparameterized updates correspond to preconditioned updates in the original parameters. For Hera-PSD, the authors prove that with small Gaussian initialization excess risk decreases exponentially with gradient descent iterations, up to an error that falls with training-set size as 1/n (up to logarithmic factors); the bound does not worsen with the number of attention layers. In a stylized nonlinear-dynamics time-series experiment at the same iteration budget, Hera had 98.1% lower mean-squared test error than AdamW and 97.7% lower than Muon.

### Source excerpt

arXiv:2603.01514v2 Announce Type: replace Abstract: We study the training dynamics of gradient descent in a multilayer softmax self-attention network trained to perform linear regression and show that in the infinite-data limit this regression problem is equivalent to a matrix factorization problem. This connection motivates the design of Hera, a new first-order optimization algorithm for training self-attention networks. Key features of Hera include a reparameterization of the model arising from the matrix factorization geometry and a rescaling of the parameters which ensures that the empirical loss gradients remain close to the auxiliary matrix factorization gradients throughout training. Hera updates in the reparameterized model are equivalent to a certain preconditioned update in the original parameterization. We also propose a variant of Hera called Hera-PSD; our main theoretical result is that under small Gaussian initialization the excess risk of the model trained by Hera-PSD decreases exponentially fast in the number of gradient descent iterations, up to an additional error term that decreases as $1/n$, up to logarithmic factors, in the size of the training set. A particularly appealing aspect of our excess risk bound is that it does not degrade as the number of self-attention layers increases. In a stylized time-series prediction experiment in a nonlinear dynamical system, we find that a model trained by Hera achieves $98.1\%$ lower mean-squared test error than a model trained by AdamW and $97.7\%$ lower than a model trained by Muon at the same iteration budget.

## Calibeating Prediction-Powered Inference

DevFeed: [Calibeating Prediction-Powered Inference](<https://devfeed.tech/articles/calibeating-prediction-powered-inference-131903.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2604.21260>)

Author: Lars van der Laan, Mark Van Der Laan

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Calibrated uncertainty](<https://devfeed.tech/topics/calibrated-uncertainty.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [aipw](<https://devfeed.tech/tags/aipw.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [econ-em](<https://devfeed.tech/tags/econ-em.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [inference](<https://devfeed.tech/tags/inference.md>), [linear](<https://devfeed.tech/tags/linear.md>), [model](<https://devfeed.tech/tags/model.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [original](<https://devfeed.tech/tags/original.md>), [package](<https://devfeed.tech/tags/package.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [q-bio-qm](<https://devfeed.tech/tags/q-bio-qm.md>), [regression](<https://devfeed.tech/tags/regression.md>), [sample](<https://devfeed.tech/tags/sample.md>), [scale](<https://devfeed.tech/tags/scale.md>), [score](<https://devfeed.tech/tags/score.md>), [simulations](<https://devfeed.tech/tags/simulations.md>), [standard](<https://devfeed.tech/tags/standard.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [study](<https://devfeed.tech/tags/study.md>), [variance](<https://devfeed.tech/tags/variance.md>)

### AI overview

Calibrated Prediction-Powered Inference post-hoc calibrates a black-box prediction score on the labeled sample before using it for semisupervised mean estimation, requiring no model retraining. The paper studies linear and isotonic calibration: isotonic calibration has first-order optimality guarantees, while linear calibration is first-order equivalent to PPI++. It describes the original PPI estimator as a special case of AIPW that can be inefficient when predictions are accurate, and PPI++ as AIPW with empirical efficiency maximization. In simulations and real-data experiments, calibrated estimators often outperform PPI and are competitive with or outperform AIPW and PPI++. These are abstract-level claims; the excerpt gives no experiment-specific results.

### Source excerpt

arXiv:2604.21260v2 Announce Type: replace-cross Abstract: We study semisupervised mean estimation with a small labeled sample, a large unlabeled sample, and a black-box prediction model whose output may be miscalibrated. A standard approach in this setting is augmented inverse-probability weighting (AIPW) [Robins et al., 1994], which protects against prediction-model misspecification but may be inefficient when the prediction score is poorly aligned with the outcome scale. We introduce Calibrated Prediction-Powered Inference, which post-hoc calibrates the prediction score on the labeled sample before using it for semisupervised estimation. This simple step requires no retraining and can calibeat the original score, improving it both as a predictor of the outcome and as a regression adjustment for semisupervised inference. We study both linear and isotonic calibration. For isotonic calibration, we establish first-order optimality guarantees: isotonic post-processing can improve predictive accuracy and estimator efficiency relative to the original score and simpler post-processing rules, while replacing the fitted isotonic score by its conditional-mean oracle yields no additional first-order variance reduction. For linear calibration, we show first-order equivalence to PPI++. We also clarify the relationship between existing estimators, showing that the original PPI estimator is a special case of AIPW and can be inefficient when the prediction model is accurate, while PPI++ is AIPW with empirical efficiency maximization [Rubin and van der Laan, 2008]. In simulations and real-data experiments, our calibrated estimators often outperform PPI and are competitive with, or outperform, AIPW and PPI++. We provide an accompanying Python package, ppi_aipw, at https://larsvanderlaan.github.io/ppi-aipw/.

## Linear Regression with Self-Selection Bias in Polynomial Time

DevFeed: [Linear Regression with Self-Selection Bias in Polynomial Time](<https://devfeed.tech/articles/linear-regression-with-self-selection-bias-in-polynomial-time-133194.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17228>)

Author: Felix Zhou

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [scikit-learn](<https://devfeed.tech/topics/scikit-learn.md>)

Tags: [bias](<https://devfeed.tech/tags/bias.md>), [cs-ds](<https://devfeed.tech/tags/cs-ds.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [linear](<https://devfeed.tech/tags/linear.md>), [linear-regression](<https://devfeed.tech/tags/linear-regression.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [regression](<https://devfeed.tech/tags/regression.md>), [selection](<https://devfeed.tech/tags/selection.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

The paper presents a polynomial-time algorithm in n, k, and 1/ε for estimating k linear regressors under maximum self-selection, including the unknown-index variant and correlated noise between regressors. For max-affine regression, it reports the first polynomial-time algorithm, replacing earlier approaches with exponential dependence on k. The method uses degree-log(n) Hermite moments of covariates among observations with large responses: these approximately decompose into tensor powers of the regressors. An implicit tensor decomposition recovers directions parallel to the regressors, then a variant of ordinary least squares estimates the regressors. The supplied abstract does not give detailed assumptions or guarantees beyond these claims.

### Source excerpt

arXiv:2611.17228v1 Announce Type: cross Abstract: We revisit the problem of estimating $k$ linear regressors with self-selection bias in $n$ dimensions up to error $\varepsilon$ with the maximum self-selection criterion, as introduced by Cherapanamjeri, Daskalakis, Ilyas, and Zampetakis [CDIZ23, STOC]. Our main result is a $\mathrm{poly}(n,k,1/\varepsilon)$ time algorithm for the unknown-index variant of this problem, thereby improving upon the running times of the best known algorithms due to Gaitonde and Mossel [GM24] and Kalavasis, Mehrotra, and Zhou [KMZ26, COLT] with exponential dependence on $k$. Our algorithm succeeds in the presence of correlated noise between regressors, which captures the related setting of max-affine regression. Consequently, we also give the first polynomial-time algorithm for this problem, improving on the prior works of Ghosh, Pananjady, Guntuboyina, and Ramchandran [GPGR22, IEEE Transactions on Information Theory] and Gaitonde and Mossel [GM24] with exponential dependence on $k$. Prior works exhaustively search over a $k$-dimensional subspace spanned by the regressors, using conditional moments of the response to test candidates. In contrast, our algorithm is driven by an improved understanding of the moments of the covariates. We show that, restricting to observations with large response values, the degree-$(\log n)$ Hermite moment of the covariates approximately decomposes as a sum of tensor powers of the regressors. The first step of our algorithm can be viewed as a tensor decomposition algorithm for these implicitly represented high-order tensors, which outputs unit vectors parallel to the regressors. A secondary step based on a simple variant of ordinary least squares extracts estimates for the regressors given these unit vectors. We believe our analytic insight and algorithmic components may have applications to designing efficient algorithms for other latent-variable problems.

## Settling the Burn-in Gap: Sharp Risk Bounds for Tail-Averaged SGD in Linear Regression

DevFeed: [Settling the Burn-in Gap: Sharp Risk Bounds for Tail-Averaged SGD in Linear Regression](<https://devfeed.tech/articles/settling-the-burn-in-gap-sharp-risk-bounds-for-tail-averaged-sgd-in-linear-regression-128701.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13383>)

Author: Qingyue Zhao, Xuheng Li, Quanquan Gu

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [burn-in](<https://devfeed.tech/tags/burn-in.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [effective](<https://devfeed.tech/tags/effective.md>), [gap](<https://devfeed.tech/tags/gap.md>), [gradient-descent](<https://devfeed.tech/tags/gradient-descent.md>), [linear-regression](<https://devfeed.tech/tags/linear-regression.md>), [lower-bound](<https://devfeed.tech/tags/lower-bound.md>), [lower-bounds](<https://devfeed.tech/tags/lower-bounds.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [online](<https://devfeed.tech/tags/online.md>), [regression](<https://devfeed.tech/tags/regression.md>), [risk](<https://devfeed.tech/tags/risk.md>), [spectrum](<https://devfeed.tech/tags/spectrum.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [variance](<https://devfeed.tech/tags/variance.md>)

### AI overview

The paper derives matching, initialization-explicit upper and lower excess-risk bounds for constant-stepsize, tail-averaged SGD in high-dimensional linear regression, assuming a trajectory spectral feedback condition. It identifies the burn-in mechanism behind a gap in prior bounds: the direct fast-mode variance from head-eigenspace initialization decays exponentially, while fourth-moment energy scattered into slow modes can persist into the averaging window. The condition is shown to hold for arbitrary covariance spectra with a preselected, horizon-dependent stepsize over any finite horizon, and for polynomially decaying spectra with a positive horizon-independent stepsize. One- and two-dimensional constructions also show that fourth-moment assumptions alone do not determine the asymptotic exponential decay rate.

### Source excerpt

arXiv:2611.13383v1 Announce Type: new Abstract: We study the excess risk of constant-stepsize online stochastic gradient descent (SGD) with tail-averaging for high-dimensional linear regression. Building on the instance-dependent spectral framework of Zou et al. (2021), we establish matching initialization-explicit upper and lower bounds on the total excess risk of tail-averaged SGD in terms of the population covariance spectrum $\mathbf{H}$ and initialization $\mathbf{w}_0-\mathbf{w}^*$ whenever a trajectory spectral feedback condition holds. Existing analyses exhibit a fundamental gap in the effective variance: the upper bound of Zou et al. (2021) retains an undamped contribution from the initial error in the head eigenspace $\mathbf{I}_{0:k^\dagger}$, whereas their lower bound omits the variance induced by the head initialization altogether. We show that both the existing upper and lower bounds are loose: the single-hop fast-mode variance contribution from the head initialization is exponentially damped during the burn-in phase, whereas fourth-moment energy scattered from the head initialization into slow spectral modes during the early burn-in steps survives undamped into the averaging window. We prove that our spectral feedback condition is satisfied in two concrete constant-stepsize regimes: (i) for arbitrary covariance spectra $\mathbf{H}$ and any finite horizon, under an explicit horizon-dependent stepsize chosen prior to execution; and (ii) for polynomially decaying spectra $\lambda_i\asymp i^{-a}$ ($a>1$), under a single horizon-independent positive stepsize. Finally, we complement these positive results with explicit one- and two-dimensional separation constructions proving an exponential-versus-polynomial gap in the lower bound of Zou et al. (2021) and showing why fourth-moment assumptions alone cannot determine the asymptotic exponential decay rate without additional structural conditions.

## Multifidelity Operator Inference for Non-intrusive Model Reduction under Data Scarcity

DevFeed: [Multifidelity Operator Inference for Non-intrusive Model Reduction under Data Scarcity](<https://devfeed.tech/articles/multifidelity-operator-inference-for-non-intrusive-model-reduction-under-data-scarcity-128741.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13624>)

Author: Vivian Zhang, Tomoki Koike, Atticus Rex, Elizabeth Qian

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [Finite Element Method (FEM)](<https://devfeed.tech/topics/finite-element-method.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [approach](<https://devfeed.tech/tags/approach.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [budget](<https://devfeed.tech/tags/budget.md>), [can](<https://devfeed.tech/tags/can.md>), [code](<https://devfeed.tech/tags/code.md>), [critical](<https://devfeed.tech/tags/critical.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-na](<https://devfeed.tech/tags/cs-na.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [development](<https://devfeed.tech/tags/development.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fields](<https://devfeed.tech/tags/fields.md>), [framework](<https://devfeed.tech/tags/framework.md>), [important](<https://devfeed.tech/tags/important.md>), [inference](<https://devfeed.tech/tags/inference.md>), [math-na](<https://devfeed.tech/tags/math-na.md>), [model](<https://devfeed.tech/tags/model.md>), [operator](<https://devfeed.tech/tags/operator.md>), [reduction](<https://devfeed.tech/tags/reduction.md>), [stat-co](<https://devfeed.tech/tags/stat-co.md>), [under](<https://devfeed.tech/tags/under.md>)

### AI overview

Multifidelity Operator Inference (MF-OpInf) addresses scarce high-fidelity training data for non-intrusive reduced models. It supplements high-fidelity snapshots with more abundant data from coarser discretizations of the same problem, then uses multifidelity linear regression to estimate reduced operators. In numerical experiments on model PDEs, it predicted solution fields more accurately than high-fidelity-only Operator Inference (OpInf) under the same computational budget. The evidence is limited to the abstract and does not specify broader validation or performance beyond those experiments.

### Source excerpt

arXiv:2611.13624v1 Announce Type: new Abstract: Many-query analyses, in which a computational model is repeatedly evaluated, play an important role in uncertainty quantification and optimization for engineering design. However, many-query analyses can be prohibitively expensive when considering high-fidelity numerical models, especially those governed by partial differential equations (PDEs). Surrogate modeling replaces costly simulations with cheap approximations, enabling practical uncertainty quantification and design workflows. Machine learning (ML) approaches to surrogate modeling have gained significant traction in the development of surrogate models due to their non-intrusive nature; they do not require access to the underlying simulation code. The Operator Inference (OpInf) approach, in particular, fits reduced operators to state and corresponding time derivative data. However, high-fidelity training data is scarce due to limited computational budgets, and OpInf models trained on such scarce data often exhibit high variance, a critical liability in high-risk engineering design. We propose Multifidelity Operator Inference (MF-OpInf), which reduces this variance by supplementing scarce high-fidelity snapshots with abundant low-fidelity data drawn from coarser discretizations of the same problem. MF-OpInf handles data across varying fidelity levels by utilizing a multifidelity linear regression framework to estimate the reduced operators. Numerical experiments on model PDE problems demonstrate that MF-OpInf predicts PDE solution fields with higher accuracy compared to the high-fidelity-only OpInf with the same computational budget.

## Computationally Efficient Offline Policy Optimization with All-Policy Linear Realizability

DevFeed: [Computationally Efficient Offline Policy Optimization with All-Policy Linear Realizability](<https://devfeed.tech/articles/computationally-efficient-offline-policy-optimization-with-all-policy-linear-realizability-125376.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11423>)

Author: Zhijie Zheng

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [covariance](<https://devfeed.tech/tags/covariance.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [linear](<https://devfeed.tech/tags/linear.md>), [lower-bound](<https://devfeed.tech/tags/lower-bound.md>), [markov](<https://devfeed.tech/tags/markov.md>), [offline](<https://devfeed.tech/tags/offline.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [policy](<https://devfeed.tech/tags/policy.md>), [regression](<https://devfeed.tech/tags/regression.md>), [sample](<https://devfeed.tech/tags/sample.md>), [state](<https://devfeed.tech/tags/state.md>)

### AI overview

The paper gives an offline policy optimization algorithm for finite-horizon Markov decision processes where every deterministic Markov policy has a linearly realizable action-value function with coefficient norm at most B. From independent complete trajectories of one unknown behavior policy, and assuming all-policy occupancy coverage, it achieves Õ(ε⁻²) sample dependence with computation polynomial in log(2+B), using exactly one convex regression per layer. A single consistent Markov witness policy supplies the stopped-return targets; a shared reference design and bounded predictions yield pointwise confidence, while truncated leverage controls performance loss without covariance approximation at scale B⁻². The guarantee permits any fixed initial-state distribution, with realizability and coverage checked only for deterministic policies. A covariance-coverage variant allows uniform approximate realizability with polynomial dependence on B. A two-action lower bound establishes that the accuracy exponent 2 is necessary. These claims are based on the supplied abstract.

### Source excerpt

arXiv:2611.11423v1 Announce Type: new Abstract: We study offline policy optimization in finite-horizon Markov decision processes where every deterministic Markov policy has an action-value function linear in known features, with coefficient norm at most $B$. From independent complete trajectories of one unknown Markov behavior policy, and under all-policy occupancy coverage, we give an algorithm with $\widetilde O(\varepsilon^{-2})$ sample dependence and computational complexity polynomial in $\log(2+B)$. The algorithm performs exactly one convex regression per layer. Its stopped-return targets are the values of a single consistent Markov witness policy. A shared reference design and bounded prediction constraints convert fast clipped-regression error into pointwise confidence for the raw linear predictions. Truncated leverage controls the performance loss without requiring empirical covariance approximation at scale $B^{-2}$. Rounding confidence radii upward preserves the exact witness and yields a polynomial-time bit implementation for rational numerical inputs. The guarantees allow any fixed initial-state distribution; realizability and coverage need only be checked for deterministic policies. We also establish a covariance-coverage variant with uniform approximate realizability and polynomial dependence on $B$. A two-action lower bound shows that the accuracy exponent $2$ is necessary.

## Spectral Bias Reduction in Local Gated ReLU Architectures

DevFeed: [Spectral Bias Reduction in Local Gated ReLU Architectures](<https://devfeed.tech/articles/spectral-bias-reduction-in-local-gated-relu-architectures-125239.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10700>)

Author: Victorita Dolean

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [architectures](<https://devfeed.tech/tags/architectures.md>), [bias](<https://devfeed.tech/tags/bias.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-na](<https://devfeed.tech/tags/cs-na.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [gram](<https://devfeed.tech/tags/gram.md>), [hat](<https://devfeed.tech/tags/hat.md>), [layer](<https://devfeed.tech/tags/layer.md>), [learning](<https://devfeed.tech/tags/learning.md>), [local](<https://devfeed.tech/tags/local.md>), [math-na](<https://devfeed.tech/tags/math-na.md>), [partition](<https://devfeed.tech/tags/partition.md>), [physics-informed](<https://devfeed.tech/tags/physics-informed.md>), [reduction](<https://devfeed.tech/tags/reduction.md>), [regression](<https://devfeed.tech/tags/regression.md>), [relu](<https://devfeed.tech/tags/relu.md>), [windows](<https://devfeed.tech/tags/windows.md>)

### AI overview

Gating ReLU features with hat partition-of-unity weights reduces spectral bias for L² regression, but does not resolve it for the H¹ energy form. In the studied shallow models, the gated Gram matrices have block-banded structure. For L² regression, learning-rate spread falls from 1.9x10⁵ to 4.5 at matched model size, uniformly with the number of windows. For H¹, the symbol vanishes at zero frequency, so conditioning remains Θ(N²) and frequency ordering is inverted; a preconditioner from the interior stencil is needed to correct the bias. Experiments on 1D regression and Poisson problems support this difference, and the regression effect persists when hidden-layer parameters are trained.

### Source excerpt

arXiv:2611.10700v1 Announce Type: new Abstract: Neural networks trained by gradient-based methods learn the low-frequency content of a target function faster than its high-frequency content. This spectral bias is well documented, but it is much less clear which architectural choices mitigate it and which cannot. We study the question for shallow ReLU models linear in their trainable parameters, where each mode is learned at a rate given by a Gram-matrix eigenvalue, and for a general symmetric bilinear form $a(\cdot,\cdot)$, so that least-squares regression ($a=$ the $L^2$ inner product) and variational or physics-informed losses ($a=$ the $H^1$ energy form) are two instances of one analysis. Within this framework we propose gated local ReLU dictionaries, obtained by multiplying each ReLU feature by a hat partition-of-unity weight, in the spirit of partition-of-unity finite elements and overlapping Schwarz methods. The resulting Gram matrices are block-banded for any local bilinear form, with a block-Toeplitz interior whose matrix-valued symbol governs the spectrum. The two forms then behave very differently. For $L^2$ regression, the symbol is bounded away from zero, the ratio is uniform in the number of windows, and the measured spread of frequency-dependent learning rates falls from $1.9\times10^{5}$ to $4.5$ at matched model size. For the $H^1$ stiffness matrix, the symbol vanishes at zero frequency: gating leaves the $\Theta(N^2)$ conditioning intact and inverts the frequency ordering instead of removing the bias. A preconditioner built from the same interior stencil will fix the spectral bias, showing that architecture alone is not enough. Experiments on $1$D regression and Poisson problems confirm the dichotomy, and the regression effect persists when the hidden-layer parameters are trained as well.

## Quantile Regression Caught the House Price OLS Missed by $55K

DevFeed: [Quantile Regression Caught the House Price OLS Missed by $55K](<https://devfeed.tech/articles/quantile-regression-caught-the-house-price-ols-missed-by-55k-123821.md>)

Original publisher: [Read original article](<https://towardsdatascience.com/quantile-regression-caught-the-house-price-ols-missed-by-55k/>)

Author: Gustavo Santos

Published: 2026-11-09T12:00:00Z

Content type: article

Language: en

Sources: [Towards Data Science](<https://devfeed.tech/sources/towards-data-science.md>)

Topics: [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [house](<https://devfeed.tech/tags/house.md>), [models](<https://devfeed.tech/tags/models.md>), [ols-estimator](<https://devfeed.tech/tags/ols-estimator.md>), [python](<https://devfeed.tech/tags/python.md>), [quantile](<https://devfeed.tech/tags/quantile.md>), [regression](<https://devfeed.tech/tags/regression.md>), [sales](<https://devfeed.tech/tags/sales.md>), [statistics](<https://devfeed.tech/tags/statistics.md>)

### AI overview

With heteroscedastic house prices, a single OLS estimate can conceal widening uncertainty: in the 88-sale example, OLS predicted $164K for a $219K home. The post fits separate 5th- and 95th-quantile regressions to form a 90% prediction interval; that home's interval was $136K-$227K and contained its actual price. The models also show feature effects can differ across the price distribution: bedroom impact is small at the 5th percentile but larger at the 95th, and lot-size impact is nearly four times greater in the upper tail than for average-priced homes, while square footage is more consistent. Wider intervals for larger, higher-priced homes signal that the median alone is less informative there. These findings illustrate the method on this dataset and do not establish that every outlier will fall inside the interval.

### Source excerpt

On 88 house sales, OLS predicted $164K for a $219K home, while a 90% interval from 5th and 95th percentile models spanned $136K to $227K. The post Quantile Regression Caught the House Price OLS Missed by $55K appeared first on Towards Data Science.

## On Differentially Private Mechanisms for Linear Regression

DevFeed: [On Differentially Private Mechanisms for Linear Regression](<https://devfeed.tech/articles/on-differentially-private-mechanisms-for-linear-regression-122786.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08737>)

Author: Bardiya Aryanfard, Monika Henzinger, Farhood Rostamkhani

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [linear-regression](<https://devfeed.tech/topics/linear-regression.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [batch](<https://devfeed.tech/tags/batch.md>), [box](<https://devfeed.tech/tags/box.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-ds](<https://devfeed.tech/tags/cs-ds.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [differential-privacy](<https://devfeed.tech/tags/differential-privacy.md>), [linear-regression](<https://devfeed.tech/tags/linear-regression.md>), [lower-bounds](<https://devfeed.tech/tags/lower-bounds.md>)

### AI overview

ContinualAdaSSP addresses unconstrained least-squares regression under continual differential privacy, producing parameter estimates at each step of a data stream. The abstract reports a suboptimality gap proportional to √d-polylog(n), space O(d² log n), and update time O(d² log n + d^ω). The paper also gives the first lower bounds for box-constrained least-squares regression: proportional to min(n, √d/log^1.5 d) for (1, O(1/n))-DP and min(n, d/ε) for ε-DP, derived via a fingerprinting-code reduction. Experiments on real-world datasets reportedly show higher accuracy than other known continual mechanisms, including rerunning a batch algorithm at each time step.

### Source excerpt

arXiv:2611.08737v1 Announce Type: cross Abstract: We study the classic least-squares linear regression problem under differential privacy (DP) in the batch and the continual observation setting. In the latter setting, the mechanism receives a stream of $n$ data points $(x_i,y_i)$ ($x_i\in\mathbb{R}^d$ and $y_i\in\mathbb{R}$) and must output an estimate of the optimal parameters at each time step $i$. Although differentially private linear regression has been extensively studied in the batch setting, the problem remains underexplored for the continual setting, with prior work being restricted to constrained linear regression and update time polynomial in $n$. In this work, we introduce ContinualAdaSSP, the first mechanism for unconstrained linear regression under continual differential privacy. Our mechanism's suboptimality gap is proportional to $\sqrt{d}\operatorname{polylog}(n)$. It requires $O(d^2\log{n})$ space and $O(d^2\log{n}+d^{\omega})$ update time, where $d$ is the dimension of the input space. We also study fundamental limits of differentially private linear regression by showing the first lower bounds for the box-constrained least-squares linear regression problem. Our lower bounds are proportional to $\min(n,\frac{\sqrt{d}}{\log^{1.5}d})$ for $(1,O(\frac{1}{n}))$-DP, and $\min(n,\frac{d}{\epsilon})$ for $\epsilon$-DP, and are derived using a reduction from a fingerprinting code problem from [Peter et al., 2024], providing new insights into the inherent hardness of this canonical task. Our experimental evaluation on real-world datasets shows that the accuracy of our continual observation mechanism clearly outperforms any other known differentially private mechanism for this problem such as rerunning the batch algorithm at each time step.

## Self-Distillation is Optimal Among Spectral Shrinkage Estimators

DevFeed: [Self-Distillation is Optimal Among Spectral Shrinkage Estimators](<https://devfeed.tech/articles/self-distillation-is-optimal-among-spectral-shrinkage-estimators-123154.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.17778>)

Author: Radu Lecoiu, Debarghya Mukherjee, Pragya Sur

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [linear-regression](<https://devfeed.tech/topics/linear-regression.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [covariance](<https://devfeed.tech/tags/covariance.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-centers](<https://devfeed.tech/tags/data-centers.md>), [distillation](<https://devfeed.tech/tags/distillation.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [federated](<https://devfeed.tech/tags/federated.md>), [linear-regression](<https://devfeed.tech/tags/linear-regression.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [mathematics](<https://devfeed.tech/tags/mathematics.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>), [statistics](<https://devfeed.tech/tags/statistics.md>)

### AI overview

In high-dimensional linear regression, self-distillation can attain optimal prediction among spectral shrinkage estimators under the paper's stated settings. With a covariance matrix having s distinct spikes, s distillation steps are optimal, and using fewer steps is strictly suboptimal. For general covariance, increasing the number of steps approaches the optimal limiting risk among continuous spectral shrinkage estimators. The authors also propose generalized cross-validation to select distillation parameters and establish an asymptotic oracle guarantee. In a federated setting, the best local rule also uses self-distillation, but differs from the optimal rule when data are centralized.

### Source excerpt

arXiv:2605.17778v2 Announce Type: replace-cross Abstract: Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in high-dimensional linear regression by introducing and analyzing a broad class of estimators, namely spectral shrinkage estimators. We consider both spiked and general covariance matrices. For the former, we establish that when the covariance has $s$ distinct spikes, $s$-step self-distillation achieves optimal prediction among spectral shrinkage estimators, outperforming well-known estimators in statistics and machine learning. Moreover, we show that $s$ steps are necessary for optimality: any $(s-k)$-step distilled estimator is strictly suboptimal for $1 \leq k \leq s$. For general covariance matrices, we show that self-distillation approaches the optimal limiting prediction risk among continuous spectral shrinkage estimators as the number of distillation steps increases. We further develop a generalized cross-validation procedure for selecting the distillation parameters from data and establish an asymptotic oracle guarantee for the resulting estimator. We also study a federated approach where multiple data centers share spectral shrinkage estimators and a common server seeks to aggregate them to achieve optimal performance. In this case, we find that the best local rule again takes the form of self-distillation, though it differs from the optimal rule when data are hosted centrally on a single server. Together, our results explain why self-distillation improves predictive performance and provide a broader statistical framework that connects it to classical shrinkage-based methods.

[Next page](<https://devfeed.tech/topics/linear-regression.md?cursor=WyIyMDI2LTExLTA5VDA1OjAwOjAwKzAwOjAwIiwgIjk4YWMyYjkyLWVkZTYtNDc5My1hMTE4LTg1N2MxOTVjYmRmMyJd>)