# Hyperparameter optimization

Hyperparameter optimization is the automated search for well-performing machine-learning hyperparameter configurations, using methods such as grid search, random search, evolutionary algorithms, Bayesian optimization, Hyperband, and racing.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Survey-aware Machine Learning: A Guideline for Valid Population Health Inference based on Scoping Review

DevFeed: [Survey-aware Machine Learning: A Guideline for Valid Population Health Inference based on Scoping Review](<https://devfeed.tech/articles/survey-aware-machine-learning-a-guideline-for-valid-population-health-inference-based-on-scoping-review-144893.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.08963>)

Author: YongKyung Oh, Henry W. Zheng, Jeffrey Feng, Alex A. T. Bui

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [design](<https://devfeed.tech/tags/design.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fairness](<https://devfeed.tech/tags/fairness.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [health](<https://devfeed.tech/tags/health.md>), [inference](<https://devfeed.tech/tags/inference.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [ml](<https://devfeed.tech/tags/ml.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [models](<https://devfeed.tech/tags/models.md>), [nutrition](<https://devfeed.tech/tags/nutrition.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [performance](<https://devfeed.tech/tags/performance.md>), [performance-evaluation](<https://devfeed.tech/tags/performance-evaluation.md>), [review](<https://devfeed.tech/tags/review.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [survey](<https://devfeed.tech/tags/survey.md>)

### AI overview

Standard ML evaluation on complex health surveys can produce biased estimates, understate uncertainty, and miss population-level disparities when it ignores sampling units, stratification, and weights. The paper proposes Survey-aware Machine Learning (SaML), a nine-step guideline for incorporating survey design metadata across the ML lifecycle. Its scoping review of 16 methodological papers summarizes weighted training, design-based cross-validation, and survey-adjusted performance evaluation, while identifying gaps in hyperparameter tuning and deployment. The guideline offers task-specific direction on which steps are needed for different analytical objectives.

### Source excerpt

arXiv:2605.08963v2 Announce Type: replace-cross Abstract: Machine Learning (ML) models trained on complex health surveys such as the National Health and Nutrition Examination Survey (NHANES) often ignore primary sampling units, stratification variables, and sampling weights. This practice violates the independence assumptions of standard evaluation methods. As a result, estimates become biased, uncertainty is underestimated, and fairness assessments fail to reflect population-level disparities. We propose Survey-aware Machine Learning (SaML), a nine-step guideline that incorporates survey design metadata across the ML lifecycle. Through a scoping review of 16 methodological papers, we summarize existing work on weighted model training, design-based cross-validation, and survey-adjusted performance evaluation. We also identify gaps in hyperparameter tuning and deployment. We provide task-specific guidance that clarifies which steps are required for different analytical objectives. SaML provides a checklist for valid population inference from survey data.

## HapBO: Harnessed $\\alpha$-$\\pi$ Bayesian Optimization via Expert Natural Language Priors

DevFeed: [HapBO: Harnessed $\\alpha$-$\\pi$ Bayesian Optimization via Expert Natural Language Priors](<https://devfeed.tech/articles/hapbo-harnessed-alpha-pi-bayesian-optimization-via-expert-natural-language-priors-145008.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25100>)

Author: Jean-S\\'ebastien Giroux, Rose Guay Hottin, Marco Bonizzato

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [applications](<https://devfeed.tech/tags/applications.md>), [based](<https://devfeed.tech/tags/based.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [family](<https://devfeed.tech/tags/family.md>), [function](<https://devfeed.tech/tags/function.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [probability](<https://devfeed.tech/tags/probability.md>), [queries](<https://devfeed.tech/tags/queries.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [solar](<https://devfeed.tech/tags/solar.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [token-cost](<https://devfeed.tech/tags/token-cost.md>), [token-costs](<https://devfeed.tech/tags/token-costs.md>)

### AI overview

HapBO turns natural-language expert or LLM knowledge into a structured, reusable belief function, elicited once and used as a bounded positive multiplier on nonnegative Bayesian optimization acquisitions. Its expressive representation is not limited to Gaussian priors, and optimization requires no further LLM calls. For Expected Improvement, the authors prove the standard simple-regret rate is preserved up to a bounded factor, even with adversarial beliefs. Across 40 HPOBench tasks, a synthetic suite in 6-15 dimensions, and applications in cortical neurostimulation and constrained solar design, HapBO reports the lowest aggregate HPOBench regret and strong real-world performance. The abstract says it improves real-world outcomes at budgets of 12-20 queries, with lower wall-clock and token costs than other LLM-based methods.

### Source excerpt

arXiv:2611.25100v1 Announce Type: new Abstract: Bayesian Optimization (BO) can be substantially more sample-efficient when informed by prior knowledge about the problem. In practice, however, such knowledge is often available only in natural language, e.g., expert intuitions or documentation, rather than as the explicit probability distribution landscape that prior-guided BO methods may require. Leveraging it effectively demands both an expressive mapping from language to priors and robustness to information that may be misleading. We propose HapBO (Harnessed $\alpha$-$\pi$ Bayesian Optimization), which elicits beliefs once from a human expert, a large language model (LLM), or both, and deterministically compiles a structured specification into a reusable belief function ($\pi: \mathcal{X} \to[0,1]$), with a rich representation not restricted to a Gaussian family. Building on $\alpha$-$\pi$BO, HapBO applies this belief function through a bounded positive multiplier on any nonnegative acquisition, preserving positive acquisition support and requiring no LLM calls during optimization. We prove that, for the Expected Improvement acquisition function (EI), HapBO preserves the standard simple-regret rate up to a bounded factor, even under adversarial beliefs. We evaluate HapBO on 40 HPOBench tasks, a synthetic suite in 6-15 dimensions and two real-world applications: cortical neurostimulation and constrained solar design. HapBO achieves the lowest aggregate HPOBench regret and strong real-world task performance, with zero in-loop token cost. In real-world applications it improves the optimization outcome even at budgets of 12-20 queries, and at a small fraction of the wall-clock and token costs of other LLM-based methods.

## Explainable AI for Solar PV Power Forecasting: From Model Selection to Trustworthy, Reproducible Deployment

DevFeed: [Explainable AI for Solar PV Power Forecasting: From Model Selection to Trustworthy, Reproducible Deployment](<https://devfeed.tech/articles/explainable-ai-for-solar-pv-power-forecasting-from-model-selection-to-trustworthy-reproducible-deployment-145603.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25894>)

Author: Nuzhat Khan, Muhammad Paend Bakht, Ab Al-Hadi Ab Rahman, Fabiano Pallonetto, Amy Fahy, Mohamed Khalafalla Hassan, Paul Stynes, Muhammad Anique Aslam, Terence O Donnell, Mohammad Riadh

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Feature Engineering](<https://devfeed.tech/topics/feature-engineering.md>), [tinyml](<https://devfeed.tech/topics/tinyml.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [climate](<https://devfeed.tech/tags/climate.md>), [cross-validation](<https://devfeed.tech/tags/cross-validation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [deep](<https://devfeed.tech/tags/deep.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [edge](<https://devfeed.tech/tags/edge.md>), [explainable-ai](<https://devfeed.tech/tags/explainable-ai.md>), [feature](<https://devfeed.tech/tags/feature.md>), [features](<https://devfeed.tech/tags/features.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model-selection](<https://devfeed.tech/tags/model-selection.md>), [prediction](<https://devfeed.tech/tags/prediction.md>), [pv](<https://devfeed.tech/tags/pv.md>), [shap](<https://devfeed.tech/tags/shap.md>), [solar](<https://devfeed.tech/tags/solar.md>)

### AI overview

On a one-year test of an openly available 16.32 kWp solar plant in Porto, Portugal, forecasting one day ahead from a 4 km weather grid, the best models achieved 11.6% nRMSE--about half the error of smart persistence. Input-feature choice affected accuracy more than the choice among strong algorithms, and shuffled cross-validation was mildly optimistic. Conformal calibration raised coverage of nominal 80% prediction intervals from 64.5% to 81.7%. SHAP rankings were stable across retraining periods but shifted when correlated inputs were removed, so they describe model behavior rather than the plant itself. The paper presents a reproducible workflow combining automated model search and explainable AI; the supplied text is an abstract and does not provide implementation details.

### Source excerpt

arXiv:2611.25894v1 Announce Type: new Abstract: Machine learning (ML) is now a standard tool for forecasting the output of solar photovoltaic (PV) plants, yet published accuracy figures are often hard to compare and harder to trust. Many studies validate with random splits, report no reference forecast, and present feature attributions without saying what they can and cannot show. This paper reviews the main families of ML methods for PV power forecasting and resource assessment, from tree ensembles to deep and foundation models, and sets out a reproducible workflow that combines automated model search with explainable AI. The workflow is then tested on an openly available dataset: a 16.32 kWp plant in Porto, Portugal, forecast one day ahead from a 4 km numerical weather prediction grid. Over a full test year, the best models reach a normalised root mean square error (nRMSE) of 11.6% of installed capacity, about half the error of smart persistence. The choice of input features changed accuracy far more than the choice among strong algorithms, and shuffled cross-validation was mildly optimistic. After conformal calibration, the nominal 80% prediction intervals covered 81.7% of observations, against 64.5% before. SHAP rankings stayed stable when the model was retrained on different periods but shifted when correlated inputs were removed, so attributions should be read as properties of a model rather than of the plant. The paper closes with practical lessons and open problems, including deployment on edge hardware and the effect of a changing climate.

## Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models

DevFeed: [Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models](<https://devfeed.tech/articles/spherical-boltzmann-machines-a-solvable-theory-of-learning-and-generation-in-energy-based-models-144894.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.09031>)

Author: Thomas Tulinski, Simona Cocco, R\\'emi Monasson, Jorge Fernandez-De-Cossio-Diaz

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [machine learning overfitting](<https://devfeed.tech/topics/machine-learning-overfitting.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [all](<https://devfeed.tech/tags/all.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [coupling](<https://devfeed.tech/tags/coupling.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [energy](<https://devfeed.tech/tags/energy.md>), [generation](<https://devfeed.tech/tags/generation.md>), [generative](<https://devfeed.tech/tags/generative.md>), [learning](<https://devfeed.tech/tags/learning.md>), [limit](<https://devfeed.tech/tags/limit.md>), [machine](<https://devfeed.tech/tags/machine.md>), [machines](<https://devfeed.tech/tags/machines.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [partition](<https://devfeed.tech/tags/partition.md>), [physics](<https://devfeed.tech/tags/physics.md>), [theory](<https://devfeed.tech/tags/theory.md>)

### AI overview

The paper analyzes the spherical Boltzmann machine (SBM) in the high-dimensional limit, deriving closed equations for its training dynamics and computing asymptotic Bayesian evidence as a measure of global properties of the trained model. It finds cascades of phase transitions during training and across hyperparameters, associated with successive alignment and condensation of the coupling matrix's top modes to the data. In a teacher-student setting, these transitions connect to sampling-time effects including temperature tuning, double descent with regularization, tempered posteriors, and training-time disequilibrium that biases the model. The abstract reports numerical evidence that these phenomena also occur in standard generative architectures beyond the SBM.

### Source excerpt

arXiv:2605.09031v2 Announce Type: replace Abstract: Energy-based models (EBMs) are flexible generative architectures inspired by statistical physics, but their learning and generative properties remain poorly understood. Here, we analyze a solvable EBM in the high-dimensional limit: the spherical Boltzmann machine (SBM). Combining tools from random matrix theory and dynamical mean-field theory, we: derive and solve closed equations describing the training dynamics of the SBM in the high-dimensional limit; compute the asymptotic Bayesian evidence, which acts as a partition function in parameter space and encodes global properties of the trained model; and uncover cascades of phase transitions that occur both during training and as a function of hyperparameters, related to successive alignment and condensation of the top modes of the coupling matrix to the data. We connect these transitions to sampling-time generative phenomena in a teacher-student scenario, including: sampling temperature tuning, double descent as a function of regularization strength, tempered posterior effects, and out-of-equilibrium effects during training that induce biases in the trained model. We provide numerical evidence showing that all these phenomena appear in standard generative architectures, beyond the SBM.

## Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization

DevFeed: [Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization](<https://devfeed.tech/articles/multi-objective-structured-pruning-of-llms-for-latency-and-model-size-optimization-145356.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.22583>)

Author: Muhammad Junaid Ali, Smail Niar, El-Ghazali Talbi

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Post-training optimization](<https://devfeed.tech/topics/post-training-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llms](<https://devfeed.tech/tags/llms.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pruning](<https://devfeed.tech/tags/pruning.md>), [size](<https://devfeed.tech/tags/size.md>), [structured](<https://devfeed.tech/tags/structured.md>)

### AI overview

The paper proposes a hardware-aware, two-stage structured pruning framework to balance LLM latency and model size for edge deployment. First, multi-objective depth pruning removes entire attention and MLP blocks; then Parallel Bayesian Optimization selects layer-wise pruning ratios under latency constraints, while importance-based strategies choose components within each layer's budget. The abstract reports that across multiple LLMs pruned by 37.5% and 50%, the approach outperformed existing methods on commonsense reasoning while reducing inference cost, with minimal reported impact on commonsense reasoning and zero-shot performance. The excerpt gives no specific model names, latency measurements, or detailed evaluation results.

### Source excerpt

arXiv:2607.22583v4 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved widespread adoption because of their strong reasoning and query-response capabilities. However, deploying them in embedded and edge computing environments remains challenging because of strict latency, memory, and energy constraints. Their large parameter counts and computational demands hinder efficient execution on resource-constrained platforms. Although model pruning has emerged as a viable solution for reducing scale while preserving performance, jointly optimizing layers, attention heads, and Multi-Layer Perceptron (MLP) dimensions remains highly complex. Exhaustively exploring this combined design space is computationally expensive and often leads to local optima or unstable configurations. To address these limitations, we propose a hardware-aware, multi-objective structured pruning framework. The proposed two-stage method explicitly targets latency and model size for efficient deployment on edge devices. In the coarse-grained stage, multi-objective depth pruning removes entire attention and MLP blocks to reduce computational load and memory usage. In the subsequent fine-grained stage, Parallel Bayesian Optimization (PBO) searches for the optimal layer-wise pruning ratios for pruning under latency constraints, while importance-based strategies rank the specific components to be pruned within each layer's allocated budget. Experimental results show that our approach reduces model complexity with minimal impact on commonsense reasoning tasks and zero-shot performance. Our method achieves a favorable trade-off among accuracy, latency, and model size, making it suitable for edge deployment. Across multiple LLMs at 37.5% and 50% pruning ratios, the proposed approach achieves better performance on commonsense reasoning tasks than existing methods while significantly reducing inference cost.

## What Survival Benchmarks Don't Tell You: Impact of Model Selection and Dataset Regimes

DevFeed: [What Survival Benchmarks Don't Tell You: Impact of Model Selection and Dataset Regimes](<https://devfeed.tech/articles/what-survival-benchmarks-don-t-tell-you-impact-of-model-selection-and-dataset-regimes-143568.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24837>)

Author: Sandra Dening, Emre Kavak, Christian Wachinger, George H. Chen

Published: 2026-11-19T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>)

Tags: [bayesian](<https://devfeed.tech/tags/bayesian.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [brier-score](<https://devfeed.tech/tags/brier-score.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [model](<https://devfeed.tech/tags/model.md>), [model-selection](<https://devfeed.tech/tags/model-selection.md>), [performance](<https://devfeed.tech/tags/performance.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [scale](<https://devfeed.tech/tags/scale.md>), [spaces](<https://devfeed.tech/tags/spaces.md>), [survival](<https://devfeed.tech/tags/survival.md>), [utility](<https://devfeed.tech/tags/utility.md>), [variable](<https://devfeed.tech/tags/variable.md>)

### AI overview

A benchmark of 10 classical, tree-based, and deep learning survival methods across 49 datasets finds that evaluation choices and dataset regimes shape performance conclusions. For complex deep learning and tree-based models, tuning for Integrated Brier Score (IBS) improves probabilistic accuracy and calibration over tuning for C-index, with little loss in ranking. Deep learning benefits from larger samples but performs worse in high-dimensional feature spaces. Heavy censoring can artificially inflate absolute calibration metrics and produce overly conservative tests. The authors argue for reporting evaluation setups transparently instead of relying on global single-number summaries.

### Source excerpt

arXiv:2611.24837v1 Announce Type: new Abstract: Despite the proliferation of survival analysis methods, current benchmarks often operate under narrow conditions: they heavily rely on the concordance index (C-index) for evaluation and model selection, and frequently either exclude deep learning (DL) methods or underrepresent challenging dataset regimes. To address these limitations, we conduct a large-scale, neutral benchmark comparing 10 classical, tree-based, and DL methods across 49 diverse datasets. Crucially, we treat the model selection criterion (optimizing for C-index, Integrated Brier Score (IBS), or Mean Absolute Error (MAE-PO)) as an independent experimental variable. Using Bayesian mixed-effects modeling, we demonstrate that evaluation design heavily dictates conclusions. For complex DL and tree-based architectures, selecting hyperparameters to optimize for IBS instead of C-index yields substantial improvements in probabilistic accuracy and calibration at a negligible cost to discriminative ranking. Furthermore, method performance (C-index, IBS, MAE-PO, and X-CAL) is fundamentally governed by dataset characteristics. While DL models scale exceptionally well with larger sample sizes, they suffer notable performance drops in high-dimensional feature spaces. Additionally, we expose a structural vulnerability where heavy censoring artificially inflates absolute calibration metrics, creating overly conservative tests. Ultimately, our findings challenge the utility of global single-number performance summaries and advocate for survival analysis results that transparently report the evaluation setup. Our code is available at: https://github.com/sdening/surv-bench-study

## Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization

DevFeed: [Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization](<https://devfeed.tech/articles/prior-knowledge-or-search-a-study-of-llm-agents-in-hardware-aware-code-optimization-142112.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.19782>)

Author: Dmitry Redko (Applied AI Institute), Albert Fazlyev (AI Talent Hub, ITMO University), Konstantin Sozykin (Applied AI Institute), Maria Ivanova (YSDA, Applied AI Institute), Evgeny Burnaev (Applied AI

Published: 2026-11-19T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [black-box](<https://devfeed.tech/tags/black-box.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [cuda](<https://devfeed.tech/tags/cuda.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [ir](<https://devfeed.tech/tags/ir.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [search](<https://devfeed.tech/tags/search.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

Across three controlled experiments, the authors find that LLM code optimization relies more on pretrained priors than on feedback or agent structure. In black-box optimization, models behave as greedy optimizers. For zero-shot kernel generation, explicit input-size information does not measurably change the generated parameters, while performance on uncommon kernel sizes drops sharply regardless of language. Under iterative feedback, CUDA kernels improve monotonically, whereas TVM IR kernels degrade. The abstract attributes this difference to optimization being less effective with low-density language.

### Source excerpt

arXiv:2605.19782v2 Announce Type: replace-cross Abstract: LLM discovery and optimization systems are increasingly applied across domains, implementing a common propose-evaluate-revise loop. Such optimization or discovery progresses via context conditioning on received feedback from an environment. However, as modern LLM agents are increasingly complex in their structure, it is difficult to evaluate which components contribute the most, and when and how this exploration may fail. We answer these questions through three controlled experiments. Our findings: (1) In pure black-box optimization, LLMs act as greedy optimizers. (2) In zero-shot kernel generation, providing explicit input-size information has no measurable effect, models converge to the same kernel parameters regardless of size or temperature, as though the size instruction were invisible. Moreover, when tasked to perform kernel optimization for uncommon kernel sizes, performance sharply degrades regardless of the language used. (3) In feedback-loop kernel optimization, CUDA improves monotonically under iterative feedback, while TVM IR actively degrades, which demonstrates that kernel optimization degrades when models operate with low-density language. Our results conclude that LLMs in code optimization tasks highly depend on pretrained priors rather than provided feedback or agentic structure.

## IQS-BO: In-Context Query Selection for Bayesian Optimisation

DevFeed: [IQS-BO: In-Context Query Selection for Bayesian Optimisation](<https://devfeed.tech/articles/iqs-bo-in-context-query-selection-for-bayesian-optimisation-143318.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.01269>)

Author: Luca Geminiani, Nadja Klein

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [batch](<https://devfeed.tech/tags/batch.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [best](<https://devfeed.tech/tags/best.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [gps](<https://devfeed.tech/tags/gps.md>), [kl-divergence](<https://devfeed.tech/tags/kl-divergence.md>), [optimisation](<https://devfeed.tech/tags/optimisation.md>), [probabilistic](<https://devfeed.tech/tags/probabilistic.md>), [query](<https://devfeed.tech/tags/query.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [selection](<https://devfeed.tech/tags/selection.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [supervised-learning](<https://devfeed.tech/tags/supervised-learning.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

IQS-BO uses a Prior-data Fitted Network to choose Bayesian optimisation queries by predicting, in one forward pass, each candidate's probability of being best in the set. It learns this decision with supervised training on synthetic priors, targeting the posterior probability of that event. It can operate without a surrogate for fully in-context optimisation, or use predictions from a fixed probabilistic surrogate to amortise only query selection. This differs from PFNS4BO, which still numerically maximises an acquisition function, and FIBO, which samples locations from a learned density and has no surrogate. The authors report lower query-proposal cost than acquisition-based methods and performance matching or exceeding Gaussian-process BO and available in-context methods across sequential and batch settings and low and high dimensions. A proposed mixture prior includes GP samples and functions with warped inputs, narrow optima, or plateaus; the abstract reports improved performance and broader generalisation from this prior.

### Source excerpt

arXiv:2610.01269v2 Announce Type: replace Abstract: Bayesian Optimisation (BO) is a powerful framework for optimising expensive black-box functions, but typically requires refitting a surrogate and maximising an acquisition function at every evaluation. In-context approaches based on Prior-data Fitted Networks (PFNs) amortise part of this cost by pre-training transformers on functions drawn from synthetic priors. PFNS4BO amortises the surrogate but still relies on a numerically maximised acquisition function, while FIBO performs BO fully in-context by sampling optimiser locations from a learned density, which fixes the decision rule and admits no surrogate. Learned acquisition functions score a finite candidate set with a network, but, lacking a label for the query, learn the score by reinforcement learning. We propose IQS-BO, a PFN that learns the query decision by supervised learning on synthetic priors. In a single forward pass, IQS-BO predicts the probability that each candidate is the best in the set, and we show that training minimises its expected KL divergence to the posterior probability of this event. The model can be pre-trained without a surrogate for fully in-context BO, or take the predictions of a fixed probabilistic surrogate as additional input, amortising only the decision step. Our method proposes queries at a fraction of the cost of acquisition-based methods, while matching or outperforming BO with Gaussian processes (GPs) and available in-context methods in sequential and batch optimisation, in low and high dimensions. Finally, we propose a mixture prior for pre-training PFNs which combines samples from GPs with functions exhibiting warped inputs, narrow optima, or plateaus that are poorly modelled by stationary GP kernels. We show that pre-training on this prior can improve performance and help the network generalise to a broader range of optimisation problems.

## Pragmatic Curiosity: A Unified Framework for Hybrid Learning and Optimization via Active Inference

DevFeed: [Pragmatic Curiosity: A Unified Framework for Hybrid Learning and Optimization via Active Inference](<https://devfeed.tech/articles/pragmatic-curiosity-a-unified-framework-for-hybrid-learning-and-optimization-via-active-inference-143231.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.06104>)

Author: Yingke Li, Anjali Parashar, Enlu Zhou, Chuchu Fan

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Model Development](<https://devfeed.tech/topics/model-development.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [bayesian](<https://devfeed.tech/tags/bayesian.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [curiosity](<https://devfeed.tech/tags/curiosity.md>), [decision](<https://devfeed.tech/tags/decision.md>), [experimental-design](<https://devfeed.tech/tags/experimental-design.md>), [framework](<https://devfeed.tech/tags/framework.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [inference](<https://devfeed.tech/tags/inference.md>), [information](<https://devfeed.tech/tags/information.md>), [learning](<https://devfeed.tech/tags/learning.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [risk](<https://devfeed.tech/tags/risk.md>), [search](<https://devfeed.tech/tags/search.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

Pragmatic Curiosity (PraC) unifies Bayesian optimization and Bayesian experimental design for sequential workflows where evaluations are costly and learning is coupled to improving task performance. It scores candidate queries by balancing information gain about a task-relevant latent symbol against expected regret, with design choices around what to clarify, how to encode task value, and the balance between information and regret. The abstract describes three settings: monitoring with fixed global symbols and known losses; active search with induced local symbols and evolving coverage goals; and composite Bayesian optimization that learns hierarchical regret under unknown preferences. It reports reduced downstream decision risk, better coverage of critical outcomes, and joint learning of predictive and preference structures without task-specific staging rules.

### Source excerpt

arXiv:2602.06104v3 Announce Type: replace Abstract: Many engineering and scientific workflows rely on expensive black-box evaluations, requiring sequential decisions that must both improve task performance and reduce uncertainty. Bayesian optimization (BO) and Bayesian experimental design (BED) provide powerful but largely separate treatments of goal-directed optimization and information-seeking experimentation, leaving limited guidance for hybrid settings in which learning and optimization are intrinsically coupled. We propose Pragmatic Curiosity (PraC), a unified framework for hybrid learning and optimization via active inference. PraC evaluates candidate queries by trading information gain about a task-relevant latent symbol against an expected regret-based potential over outcomes. This formulation foregrounds three operational design choices: which latent quantity should be clarified, how task value is encoded as regret, and how strongly information gain should be exchanged against pragmatic regret. We instantiate PraC across three regimes of increasing complexity: decision-oriented monitoring with fixed global symbols and known downstream losses, targeted active search with induced local symbols and evolving coverage goals, and composite Bayesian optimization with hierarchical regret learning under unknown preferences. Across these regimes, PraC reduces downstream decision risk, improves coverage of critical outcome regions, and jointly learns predictive and preference structures without relying on task-specific staging rules.

## LLM-Assisted Multi-Fidelity Bayesian Optimization: Fusing Expert Knowledge with Numerical Data

DevFeed: [LLM-Assisted Multi-Fidelity Bayesian Optimization: Fusing Expert Knowledge with Numerical Data](<https://devfeed.tech/articles/llm-assisted-multi-fidelity-bayesian-optimization-fusing-expert-knowledge-with-numerical-data-142787.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24377>)

Author: Dongjun Han, Mingyu Lee, Ikjin Lee

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [knowledge-engineering](<https://devfeed.tech/topics/knowledge-engineering.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [battery](<https://devfeed.tech/tags/battery.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ce](<https://devfeed.tech/tags/cs-ce.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [lithium-ion](<https://devfeed.tech/tags/lithium-ion.md>), [lithium-ion-battery](<https://devfeed.tech/tags/lithium-ion-battery.md>), [llm-assisted](<https://devfeed.tech/tags/llm-assisted.md>), [llms](<https://devfeed.tech/tags/llms.md>), [optimization](<https://devfeed.tech/tags/optimization.md>)

### AI overview

LLM-Assisted MFBO uses expert descriptions of physics omitted from a low-fidelity model to help estimate its design-dependent discrepancy when high-fidelity data are scarce. An LLM compares pairs of designs by which needs the larger signed residual correction; a pairwise Gaussian process turns those judgments into a knowledge basis. High-fidelity data calibrate the basis's contribution and sign, so expert-derived judgments are not used without numerical correction. On a lithium-ion battery design problem, the method reduced the regret area under the curve by 61% versus conventional MFBO. A controlled study found that knowledge quality, rather than LLM choice, determined whether the search improved; an ablation measured the contribution of individual knowledge statements.

### Source excerpt

arXiv:2611.24377v1 Announce Type: cross Abstract: Multi-fidelity Bayesian optimization (MFBO) reduces engineering design cost by combining inexpensive low-fidelity (LF) models with a few accurate high-fidelity (HF) evaluations and sequentially selecting designs to evaluate. However, when HF data are scarce, the LF-HF discrepancy is difficult to learn from numerical data alone, especially when physics omitted in the LF model makes it design-dependent. Yet even without sufficient data, domain experts often know which physics the LF model omits and under which conditions its effect grows. Existing multi-fidelity (MF) models cannot use such qualitative knowledge directly, but large language models (LLMs) can interpret it and compare designs accordingly. The proposed LLM-Assisted MFBO adds a knowledge basis to the conventional MF model: from an expert description of the omitted physics, an LLM judges which of two designs requires the larger signed residual correction, and a pairwise Gaussian process converts these judgments into the basis. HF data calibrate its contribution and sign, so the knowledge is not trusted blindly. On a lithium-ion battery design problem, the proposed method reduced the regret area under the curve of conventional MFBO by 61%. Knowledge quality, rather than the choice of LLM, determined whether the search was accelerated, as shown by a controlled study that varied the two independently. A knowledge-level ablation further quantified the contribution of each knowledge statement. These results show that LLM-interpreted expert knowledge can complement numerical data as an information source in MFBO.

## ICICLE: In-Context Inference for Curve-Learning Extrapolation

DevFeed: [ICICLE: In-Context Inference for Curve-Learning Extrapolation](<https://devfeed.tech/articles/icicle-in-context-inference-for-curve-learning-extrapolation-143527.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24469>)

Author: Laurits Fredsgaard, Michael Riis Andersen, Mikkel N. Schmidt

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [quantization](<https://devfeed.tech/topics/quantization.md>), [onnx](<https://devfeed.tech/topics/onnx.md>)

Tags: [bayesian](<https://devfeed.tech/tags/bayesian.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [budget](<https://devfeed.tech/tags/budget.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [encoder](<https://devfeed.tech/tags/encoder.md>), [extrapolation](<https://devfeed.tech/tags/extrapolation.md>), [freeze](<https://devfeed.tech/tags/freeze.md>), [inference](<https://devfeed.tech/tags/inference.md>), [jax](<https://devfeed.tech/tags/jax.md>), [learning](<https://devfeed.tech/tags/learning.md>), [meta](<https://devfeed.tech/tags/meta.md>), [model](<https://devfeed.tech/tags/model.md>), [noise](<https://devfeed.tech/tags/noise.md>), [number](<https://devfeed.tech/tags/number.md>), [onnx](<https://devfeed.tech/tags/onnx.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [quantization](<https://devfeed.tech/tags/quantization.md>), [ray](<https://devfeed.tech/tags/ray.md>), [release](<https://devfeed.tech/tags/release.md>)

### AI overview

ICICLE is an in-context surrogate for freeze-thaw Bayesian optimization that accepts a variable number of hyperparameters through a shared set encoder. Trained on synthetic tasks covering typed hyperparameters, heavy-tailed noise, quantization, and divergence, it uses 1.28 million parameters (8.7% of FT-PFN's count) and produces similar mean-regret trajectories to FT-PFN on LCBench, PD1, and TaskSet. On Meta-Album's 86 tasks with 33 hyperparameters, it achieves lower mean normalized regret than Quick-Tune overall and on extended, with no clear difference on mini or micro. ICICLE was not pre-trained or meta-trained on Meta-Album tasks, while Quick-Tune was meta-trained on related tasks. The release samples synthetic tasks on the fly and includes a Ray Tune freeze-thaw scheduler and an ONNX inference model that does not require JAX or PyTorch.

### Source excerpt

arXiv:2611.24469v1 Announce Type: new Abstract: Freeze-thaw Bayesian optimization allocates training budget by predicting how partially observed learning curves will continue. LC-PFN first applied prior-data fitted networks (PFNs) to learning-curve extrapolation, and FT-PFN extended this approach to configuration-aware freeze-thaw optimization with a ten-slot hyperparameter encoder. We introduce In-Context Inference for Curve-Learning Extrapolation (ICICLE), an in-context surrogate that accepts a variable number of hyperparameters through a shared set encoder and is trained on a synthetic prior covering typed hyperparameters, heavy-tailed noise, quantization, and divergence. ICICLE has 1.28 million parameters (8.7% of FT-PFN's count) and produces similar mean-regret trajectories to FT-PFN on LCBench, PD1, and TaskSet. On the 86-task Meta-Album benchmark with 33 hyperparameters, ICICLE reaches lower mean normalized regret than Quick-Tune overall and on extended, with no clear difference on mini or micro. It does so without pre-training or meta-training on Meta-Album tasks, whereas Quick-Tune is meta-trained on related Meta-Album tasks. Our release samples synthetic training tasks on the fly and provides a freeze-thaw scheduler for Ray Tune and an ONNX model for inference without JAX or PyTorch.

## Hyperparameter Optimization via Budget Allocation in Lipschitz Spaces

DevFeed: [Hyperparameter Optimization via Budget Allocation in Lipschitz Spaces](<https://devfeed.tech/articles/hyperparameter-optimization-via-budget-allocation-in-lipschitz-spaces-143220.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2510.10605>)

Author: MohammadHossein Bateni, Hossein Esfandiari, Samira HosseinGhorban, Alireza Mirrokni, Radin Shahdaei

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>)

Tags: [budget](<https://devfeed.tech/tags/budget.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [hyperparameter-optimization](<https://devfeed.tech/tags/hyperparameter-optimization.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [optimization](<https://devfeed.tech/tags/optimization.md>)

### AI overview

The paper studies how to allocate a limited evaluation budget across configurations whose performance changes with training or resources. It models the task as Unknown Value Probing, using similarity between configurations to relate their unknown performance curves. Its strategies combine exploration of the configuration space with greater budget allocation to regions supported by observed results. The authors report worst-case approximation guarantees that are near-optimal relative to matching hardness barriers, and experiments across more than 250 benchmark settings show gains over strong baselines in anytime behavior and final incumbent performance.

### Source excerpt

arXiv:2510.10605v3 Announce Type: replace Abstract: Hyperparameter optimization and model selection often require allocating a limited evaluation budget across many candidate configurations, where performance improves with additional training or resources. We study this setting when each configuration has an unknown performance curve over budget. Similarity between configurations induces similarity between their curves, capturing geometric structure in the search space. We formalize the resulting decision problem as Unknown Value Probing and propose budget allocation strategies that combine principled exploration of the configuration space with value-informed concentration on regions supported by observed evidence. We prove worst-case approximation guarantees for our approach and show these guarantees are near-optimal by matching corresponding hardness barriers. Extensive experiments on large benchmark suites spanning more than 250 experimental settings demonstrate consistent gains over strong baselines in both anytime behavior and final incumbent performance.

## Consecutive Preferential Bayesian Optimization

DevFeed: [Consecutive Preferential Bayesian Optimization](<https://devfeed.tech/articles/consecutive-preferential-bayesian-optimization-143223.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2511.05163>)

Author: Aras Erarslan, Carlos Sevilla Salcedo, Ville Tanskanen, Anni Nisov, Eero P\\"aiv\\"akumpu, Heikki Aisala, Kaisu Honkap\\"a\\"a, Arto Klami, Petrus Mikkola

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [food](<https://devfeed.tech/tags/food.md>), [information](<https://devfeed.tech/tags/information.md>), [model](<https://devfeed.tech/tags/model.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimum](<https://devfeed.tech/tags/optimum.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [practical](<https://devfeed.tech/tags/practical.md>), [probabilistic](<https://devfeed.tech/tags/probabilistic.md>), [production](<https://devfeed.tech/tags/production.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [small](<https://devfeed.tech/tags/small.md>), [solutions](<https://devfeed.tech/tags/solutions.md>)

### AI overview

Consecutive Preferential Bayesian Optimization (CPBO) addresses costly candidate production by reusing the previous candidate in each comparison. It models an evaluator's indifference to small utility differences with a Just-Noticeable Difference threshold, then adapts an information-theoretic acquisition strategy to select configurations informative about the unknown optimum. The abstract reports increased accuracy when production costs are high or feedback reflects indifference, and practical feasibility in a real-world food extrusion task; it provides no quantitative results or further experimental details.

### Source excerpt

arXiv:2511.05163v2 Announce Type: replace Abstract: Preferential Bayesian optimization allows optimization of objectives that are either expensive or difficult to measure directly, by relying on a minimal number of comparative evaluations done by a human expert. Generating candidate solutions for evaluation is also often expensive, but existing preferential optimization methods do not account for this production cost. We introduce Consecutive Preferential Bayesian Optimization (CPBO) for the consecutive setting, where production cost is reduced by reusing the previous candidate in each comparison. We also account for the perceptual ambiguity of the oracle providing the feedback by incorporating a Just-Noticeable Difference threshold into a probabilistic preference model to capture indifference to small utility differences. We adapt an information-theoretic acquisition strategy to this setting, selecting new configurations that are most informative about the unknown optimum under a preference model accounting for the perceptual ambiguity. We empirically demonstrate a notable increase in accuracy in setups with high production costs or indifference feedback, and demonstrate the practical feasibility of CPBO in a real-world food extrusion task.

## Full Fine-Tuning (FFT) on SageMaker AI Serverless Model Customization

DevFeed: [Full Fine-Tuning (FFT) on SageMaker AI Serverless Model Customization](<https://devfeed.tech/articles/full-fine-tuning-fft-on-sagemaker-ai-serverless-model-customization-141875.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/full-fine-tuning-fft-on-sagemaker-ai-serverless-model-customization/>)

Author: Sandeep Raveesh-Babu

Published: 2026-11-19T00:15:58Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [lora](<https://devfeed.tech/topics/lora.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [peft](<https://devfeed.tech/topics/peft.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [customization](<https://devfeed.tech/tags/customization.md>), [fft](<https://devfeed.tech/tags/fft.md>), [fine](<https://devfeed.tech/tags/fine.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [full](<https://devfeed.tech/tags/full.md>), [gap](<https://devfeed.tech/tags/gap.md>), [guidelines](<https://devfeed.tech/tags/guidelines.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [key](<https://devfeed.tech/tags/key.md>), [lora](<https://devfeed.tech/tags/lora.md>), [matrices](<https://devfeed.tech/tags/matrices.md>), [model](<https://devfeed.tech/tags/model.md>), [post](<https://devfeed.tech/tags/post.md>), [practical](<https://devfeed.tech/tags/practical.md>), [serverless](<https://devfeed.tech/tags/serverless.md>)

### AI overview

The excerpt reports that Full Fine-Tuning (FFT) can outperform LoRA on complex code-generation tasks with large datasets, while results depend heavily on hyperparameters and FFT can cause catastrophic forgetting. In experiments, Qwen3-4B was fine-tuned on 110,000 Magicoder coding pairs using SageMaker AI Serverless Model Customization. At 100K+ examples, FFT reached 90% LLM-as-Judge correctness; a baseline with 5% warmup, batch size 8, and learning rate 2e-5 best preserved MMLU knowledge. Three epochs yielded the reported correctness peak, while training beyond that degraded tested variants. Aggressive FFT learning rates of 5e-5 or higher sharply reduced MATH and MMLU scores. LoRA slightly exceeded FFT on one LLM-as-Judge comparison (81.5% versus 79.5%), and both methods fell below the base model on MATH exact match. The service handles compute allocation and checkpoints; switching between FFT and LoRA requires changing the training type. These findings reflect the reported coding experiments and do not establish FFT as universally superior.

### Source excerpt

Amazon SageMaker AI Serverless Model Customization now supports Full Fine-Tuning (FFT). This post explores when FFT outperforms LoRA across key benchmarks, examines the spectral properties of model weight matrices behind the quality gap, shares practical hyperparameter tuning guidelines, and shows how to get started.

## In-Warehouse Machine Learning and the Modern Data Stack | Blog | Fivetran

DevFeed: [In-Warehouse Machine Learning and the Modern Data Stack | Blog | Fivetran](<https://devfeed.tech/articles/in-warehouse-machine-learning-and-the-modern-data-stack-blog-fivetran-141332.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/in-warehouse-machine-learning-and-the-modern-data-stack>)

Author: Fivetran

Published: 2026-11-18T06:13:15Z

Content type: article

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [Tensorflow](<https://devfeed.tech/topics/tensorflow.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [automated-machine-learning](<https://devfeed.tech/tags/automated-machine-learning.md>), [aws](<https://devfeed.tech/tags/aws.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [data-lakehouse-architecture](<https://devfeed.tech/tags/data-lakehouse-architecture.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [data-warehouse](<https://devfeed.tech/tags/data-warehouse.md>), [data-warehousing](<https://devfeed.tech/tags/data-warehousing.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [modern-data-stack](<https://devfeed.tech/tags/modern-data-stack.md>)

### AI overview

A modern data science stack can bring analytics and machine learning workflows together around governed warehouse data, reducing duplicated processing and the risk of training-serving skew. BigQuery ML and Redshift ML extend SQL with CREATE MODEL workflows for automated or custom models; custom models must be TensorFlow models in BigQuery and deployed through SageMaker for Redshift ML. Both support invoking trained models in SELECT queries and storing predictions in warehouse tables. Snowflake describes an extensibility approach that integrates with tools such as SageMaker and Databricks. Delta Lake takes the reverse path, adding analytics features such as ACID transactions to data lakes as part of a lakehouse architecture. The excerpt presents these as different ways to consolidate data work and pipelines.

### Source excerpt

We evaluate various in-data warehouse machine learning services, and discuss each of their benefits and requirements.

## Beyond Success Rates: Trainability and Extractability for Offline GCRL

DevFeed: [Beyond Success Rates: Trainability and Extractability for Offline GCRL](<https://devfeed.tech/articles/beyond-success-rates-trainability-and-extractability-for-offline-gcrl-141540.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.05459>)

Author: Jan Malte T\\"opperwien, Aditya Mohan, Marius Lindauer

Published: 2026-11-18T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [crl](<https://devfeed.tech/tags/crl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cube](<https://devfeed.tech/tags/cube.md>), [diagnostics](<https://devfeed.tech/tags/diagnostics.md>), [goals](<https://devfeed.tech/tags/goals.md>), [it](<https://devfeed.tech/tags/it.md>), [learning](<https://devfeed.tech/tags/learning.md>), [offline](<https://devfeed.tech/tags/offline.md>), [policy](<https://devfeed.tech/tags/policy.md>), [random](<https://devfeed.tech/tags/random.md>), [range](<https://devfeed.tech/tags/range.md>), [rank](<https://devfeed.tech/tags/rank.md>), [regression](<https://devfeed.tech/tags/regression.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [results](<https://devfeed.tech/tags/results.md>), [settings](<https://devfeed.tech/tags/settings.md>), [signal](<https://devfeed.tech/tags/signal.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

Offline GCRL success rates alone conceal how sensitive methods are to value-learning and policy-extraction hyperparameters. Under a shared extractor, the study maps learning-rate and extraction settings for GCIQL, GCIVL, QRL, and CRL. On AntMaze-M, 59% of QRL and 66% of CRL configurations reach at least 90% of each method's best sampled success, versus 3% for GCIQL and GCIVL; on Cube and Scene, GCIQL and GCIVL perform better across broader hyperparameter regions. The authors examine advantage rankings of trajectory goals versus random goals and how strongly AWR focuses updates on a subset of transitions to interpret these differences. They recommend evaluating both attainable success and the hyperparameter range that sustains it, alongside post-hoc diagnostics of the policy-training signal.

### Source excerpt

arXiv:2602.05459v3 Announce Type: replace Abstract: Offline goal-conditioned reinforcement learning (GCRL) methods are typically benchmarked by the success rates of optimized hyperparameter configurations. However, success rate by itself does not provide information about the sensitivity of algorithms to hyperparameters for value learning or policy extraction, since an algorithm could obtain the same success rate across a wide range of hyperparameters, or might succeed only on specific settings. We study this phenomenon in the offline GCRL algorithms GCIQL, GCIVL, QRL, and CRL under a shared policy extractor, by constructing landscapes of interaction between the learning rate and policy extraction hyperparameters such as the temperature of advantage-weighted regression (AWR). These landscapes reveal large, task-dependent differences in hyperparameter sensitivity. On AntMaze-M, 59% of QRL configurations and 66% of CRL configurations achieve at least 90% of the method's best sampled success rate, compared with 3% for both GCIQL and GCIVL. On Cube and Scene, GCIQL and GCIVL instead attain higher success over broader hyperparameter regions than QRL and CRL. To interpret these differences, we analyze how reliably learned advantages rank actual goals from the same trajectory above random goals, and how strongly AWR concentrates policy updates on a small subset of transitions. Our findings indicate strongly differing behavior of these algorithms at policy-extraction time, even under the same success rates. Given these results, we argue that offline GCRL methods should be evaluated by both attainable success and the needed hyperparameter range to sustain it, complemented by post-hoc diagnostics to understand the signal used to train the policy. The overview and code are available at https://luh-ai.github.io/GCRL-Landscapes/ .

## An Open-Source Training Dataset for Foundation Models for Hyperparameter Optimization

DevFeed: [An Open-Source Training Dataset for Foundation Models for Hyperparameter Optimization](<https://devfeed.tech/articles/an-open-source-training-dataset-for-foundation-models-for-hyperparameter-optimization-141568.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.23417>)

Author: Aaron Klein, Herilalaina Rakotoarison, Luca Thale-Bombien, David Salinas

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Open Source Models & Datasets](<https://devfeed.tech/topics/open-source-models-datasets.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [hyperparameter-optimization](<https://devfeed.tech/tags/hyperparameter-optimization.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [training](<https://devfeed.tech/tags/training.md>), [training-dataset](<https://devfeed.tech/tags/training-dataset.md>)

### AI overview

BBO-Pile is an open-source dataset of over 500,000 optimization trajectories across 3,095 hyperparameter optimization tasks and global optimization problems, addressing the shortage of large, public pre-training data. The authors train foundation models ranging from 2M to 80M parameters and 200M to 2B training tokens, and study scaling with compute. They report that large-scale pre-training can effectively imitate hyperparameter optimization methods. The supplied text does not detail specific performance comparisons or limitations.

### Source excerpt

arXiv:2605.23417v2 Announce Type: replace Abstract: Most hyperparameter optimization methods require extensive tuning of their own hyperparameters, often limiting their ability to generalize across different benchmark families. Foundation models for hyperparameter optimization that learn optimization principles from a large collection of optimization trajectories offer a promising alternative, with the potential to outperform manually designed methods across diverse problem classes. However, prior work as either relied on non-public datasets or on purely synthetic data, limiting reproducibility and generalization to real-world problems. As a result, progress in this area has been constrained by the lack of large-scale, real-world, publicly available pre-training data. We introduce BBO-Pile, the first open-source dataset comprising over 500K optimization trajectories evaluated across 3,095 different hyperparameter optimization tasks and global optimization problems for different optimizers, which represents by far the largest public dataset for this task. Using this dataset, we train a family of foundation models at multiple scales, ranging from 2M to 80M parameters and from 200M to 2B training tokens, and study their scaling behavior with respect to compute. Our results demonstrate that large-scale pre-training is a viable and effective approach to imitate hyperparameter optimization methods, paving the way for future research in this direction.

## Kernel-based guarantees for nonlinear parametric models in Bayesian optimization

DevFeed: [Kernel-based guarantees for nonlinear parametric models in Bayesian optimization](<https://devfeed.tech/articles/kernel-based-guarantees-for-nonlinear-parametric-models-in-bayesian-optimization-141559.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.13160>)

Author: Rafael Oliveira

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [confidence-bounds](<https://devfeed.tech/tags/confidence-bounds.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [framework](<https://devfeed.tech/tags/framework.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [networks](<https://devfeed.tech/tags/networks.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>), [thompson-sampling](<https://devfeed.tech/tags/thompson-sampling.md>)

### AI overview

The paper develops a kernel-based framework for obtaining confidence bounds for nonlinear parametric models trained on adaptively collected data. Kernels over parameter space induce reproducing kernel Hilbert space structures over the model class, and the bounds apply to models trained with a broad class of regularized losses. The authors state that these bounds can establish convergence guarantees for approximate Thompson sampling and support analysis of broader likelihood-free Bayesian optimization methods. The supplied text is the abstract, so it does not provide details of assumptions or guarantee strength.

### Source excerpt

arXiv:2605.13160v2 Announce Type: replace-cross Abstract: Modern Bayesian optimization and adaptive sampling methods increasingly rely on nonlinear parametric models, such as neural networks, yet theoretical guarantees for such models under adaptive data collection remain limited. Existing analyses largely focus on Gaussian processes, kernel machines, linear models, or linearized neural approximations, leaving a gap between theory and the nonlinear models used in practice. We develop a kernel-based framework for analyzing regularized parametric models trained on adaptively collected data. Our approach uses kernels over the parameter space to induce reproducing kernel Hilbert space structures over the corresponding model class, yielding confidence bounds for models trained with a broad class of regularized losses. We show how these bounds can establish convergence guarantees for approximate Thompson sampling and provide a foundation for analyzing broader likelihood-free Bayesian optimization methods. These results provide a unified route to analyzing nonlinear models in Bayesian optimization and related adaptive optimization settings.

## Hybrid Quantum Selection of Learning Rates in Language-Model Training: Termination Criteria and an Interaction-Guided Schedule Oracle

DevFeed: [Hybrid Quantum Selection of Learning Rates in Language-Model Training: Termination Criteria and an Interaction-Guided Schedule Oracle](<https://devfeed.tech/articles/hybrid-quantum-selection-of-learning-rates-in-language-model-training-termination-criteria-and-an-interaction-guided-schedule-oracle-141669.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21701>)

Author: Wei-Che Hung

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [groups](<https://devfeed.tech/tags/groups.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [layer](<https://devfeed.tech/tags/layer.md>), [learning](<https://devfeed.tech/tags/learning.md>), [lookup](<https://devfeed.tech/tags/lookup.md>), [measurements](<https://devfeed.tech/tags/measurements.md>), [model](<https://devfeed.tech/tags/model.md>), [model-training](<https://devfeed.tech/tags/model-training.md>), [next](<https://devfeed.tech/tags/next.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [profile](<https://devfeed.tech/tags/profile.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [schedule](<https://devfeed.tech/tags/schedule.md>), [selection](<https://devfeed.tech/tags/selection.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The paper explores quantum-assisted selection of layer-group learning rates during language-model fine-tuning. One approach combines Durr-Hoyer minimum finding with BBHT search over a lookup oracle of measured profile scores, comparing fixed-budget and confidence-based stopping. Its Interaction-Guided Schedule Oracle instead scores joint schedules in superposition using classically measured main, pairwise, and three-way interaction effects; measured candidates are checked classically before acceptance. In closed-loop fine-tuning of a 1.5-billion-parameter model, each selected schedule guides 200 subsequent classical training steps. Benchmark accuracy stays within the range across five uniform-rate seeds. The initial baseline is uniform-rate training; using schedules from classical adaptive methods as refinement baselines is left for future work. The supplied excerpt does not detail the remaining quantum limitations.

### Source excerpt

arXiv:2611.21701v1 Announce Type: new Abstract: Learning-rate choices strongly affect language-model fine-tuning, but the best joint profile across layer groups is generally unknown. This paper studies quantum minimum finding for layer-group learning-rate selection within a hybrid classical-quantum training framework. It combines Durr-Hoyer minimum finding with BBHT search and compares fixed-budget and confidence-based stopping policies using a lookup oracle built from measured profile scores at fixed training checkpoints. The paper then introduces an Interaction-Guided Schedule Oracle that scores and marks joint schedules in superposition without a precomputed score table. Classical one-step loss measurements supply the main, pairwise and three-way interaction effects between layer groups. The circuit adds the effects for each schedule, compares the resulting score with the current baseline score, and marks improving schedules for amplitude amplification. Measured candidates are checked classically before acceptance. Uniform-rate training supplies the initial baseline; using schedules from classical adaptive methods as baselines for joint refinement is left for future work. In closed-loop fine-tuning of a 1.5-billion-parameter language model, each selected schedule drives the next 200 classical training steps. Benchmark accuracies lie within the range observed across five uniform-rate seeds. The results give a working example of how quantum superposition can support adaptive learning-rate selection during training; the remaining quantum limitations are discussed in the paper.

## AlignedHPO: Efficient Surrogate-Guided Loss Hyperparameter Optimization

DevFeed: [AlignedHPO: Efficient Surrogate-Guided Loss Hyperparameter Optimization](<https://devfeed.tech/articles/alignedhpo-efficient-surrogate-guided-loss-hyperparameter-optimization-140062.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22658>)

Author: Ivan Karpukhin, Andrey Savchenko

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [face-recognition](<https://devfeed.tech/tags/face-recognition.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [hyperparameter-optimization](<https://devfeed.tech/tags/hyperparameter-optimization.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [validation](<https://devfeed.tech/tags/validation.md>), [vector](<https://devfeed.tech/tags/vector.md>)

### AI overview

AlignedHPO tunes differentiable loss hyperparameters by aligning training-loss gradients with differentiable surrogates of validation metrics. It uses Jacobian-vector products to reduce hypergradient time and memory, and normalized alignment plus gradient smoothing to address gradient-scale ambiguity and stochastic instability. In experiments with generalized Focal Loss for long-tailed classification and ArcFace for open-set face recognition, it achieved competitive or better performance than implicit differentiation while computing hypergradients nearly 12 times faster.

### Source excerpt

arXiv:2611.22658v1 Announce Type: cross Abstract: Modern deep learning losses involve multiple hyperparameters, such as weights, scales, and margins, typically tuned manually or through expensive black-box optimization despite their differentiability. We propose AlignedHPO, an efficient framework for tuning differentiable loss hyperparameters using gradient alignment and surrogate validation objectives. Our method employs Jacobian-vector products to reduce hypergradient computation time and memory, while differentiable metric surrogates guide optimization toward task-specific evaluation criteria. We address gradient magnitude ambiguity and stochastic instability through normalized alignment and gradient smoothing. Experiments with generalized Focal Loss and ArcFace demonstrate effective adaptation across long-tailed classification and open-set face recognition. AlignedHPO achieves competitive or improved performance compared to implicit differentiation, with nearly 12x faster hypergradient computation. These results highlight the potential of efficient surrogate-guided loss hyperparameter optimization.

## Amortizing Scaling Law Construction Costs

DevFeed: [Amortizing Scaling Law Construction Costs](<https://devfeed.tech/articles/amortizing-scaling-law-construction-costs-139814.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.05016>)

Author: Abhash Kumar Jha, Diana Alexandra Onu\\c{t}u, Neeratyoy Mallik, Swagatam Haldar, Sam Laing, Niccol\\\`o Ajroldi, Shiwei Liu, Joaquin Vanschoren, Aaron Klein

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [budget](<https://devfeed.tech/tags/budget.md>), [collection](<https://devfeed.tech/tags/collection.md>), [compute](<https://devfeed.tech/tags/compute.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [construction](<https://devfeed.tech/tags/construction.md>), [costs](<https://devfeed.tech/tags/costs.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [exhaustive](<https://devfeed.tech/tags/exhaustive.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [framework](<https://devfeed.tech/tags/framework.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [full](<https://devfeed.tech/tags/full.md>), [law](<https://devfeed.tech/tags/law.md>), [scaling](<https://devfeed.tech/tags/scaling.md>)

### AI overview

The framework aims to reduce the cost of fitting scaling laws by collecting the best-loss frontier across compute scales instead of training an exhaustive grid. It formulates configuration selection as Bayesian optimization, progressively expands the acquisition compute budget, and uses surrogate-fantasized evaluations to reconstruct the broader grid. The authors report that these steps can closely match fits from a full dense grid while reducing compute by up to 10-100x.

### Source excerpt

arXiv:2609.05016v2 Announce Type: replace-cross Abstract: Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to $10\text{--}100\times$.

## Gradient Boosting vs. Random Forest vs. Neural Networks: A 2026 Performance Benchmark

DevFeed: [Gradient Boosting vs. Random Forest vs. Neural Networks: A 2026 Performance Benchmark](<https://devfeed.tech/articles/gradient-boosting-vs-random-forest-vs-neural-networks-a-2026-performance-benchmark-137767.md>)

Original publisher: [Read original article](<https://machinelearningmastery.com/gradient-boosting-vs-random-forest-vs-neural-networks-a-2026-performance-benchmark/>)

Author: Vinod Chugani

Published: 2026-11-17T11:00:00Z

Content type: comparison

Language: en

Sources: [MachineLearningMastery.com](<https://devfeed.tech/sources/machinelearningmastery-com.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Deep learning](<https://devfeed.tech/topics/deep-learning.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Transformer](<https://devfeed.tech/topics/transformer.md>), [catboost](<https://devfeed.tech/topics/catboost.md>), [mlp](<https://devfeed.tech/topics/mlp.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [audio](<https://devfeed.tech/tags/audio.md>), [backpropagation](<https://devfeed.tech/tags/backpropagation.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [catboost](<https://devfeed.tech/tags/catboost.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cnns](<https://devfeed.tech/tags/cnns.md>), [compare](<https://devfeed.tech/tags/compare.md>), [compute](<https://devfeed.tech/tags/compute.md>), [context-window](<https://devfeed.tech/tags/context-window.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [foundation-models](<https://devfeed.tech/tags/foundation-models.md>), [gradient-boosting](<https://devfeed.tech/tags/gradient-boosting.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [performance](<https://devfeed.tech/tags/performance.md>), [practical-machine-learning](<https://devfeed.tech/tags/practical-machine-learning.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [speed](<https://devfeed.tech/tags/speed.md>)

### AI overview

The excerpt describes two regimes for tabular data: tabular foundation models (TFMs) lead TabArena accuracy results on small-to-medium datasets, while gradient boosting remains stronger for large-scale data and regression. TabPFN-3.5 is reported to support up to 1 million rows and 20,000 features, but TFM limits vary and inference requires GPU memory to hold data in context. XGBoost, LightGBM, and CatBoost offer CPU-based training and low-latency inference, making them practical under compute or latency constraints. Random forest is a forgiving, quickly tuned baseline, especially with noisy data or limited tuning resources. Specialized networks have narrower advantages: TabR can outperform XGBoost on some classification tasks, while TabM and RealMLP reach parity with boosted trees on large datasets when carefully tuned and ensembled. For images, text, and audio, transformer architectures lead.

### Source excerpt

In this article, you will learn how gradient boosting, random forest, neural networks, and tabular foundation models compare on accuracy, speed, and data requirements in...

## The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search

DevFeed: [The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search](<https://devfeed.tech/articles/the-critic-can-hurt-on-the-safety-of-critics-in-llm-guided-automl-search-137381.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19037>)

Author: Gaoyuan Du, Xiaoyang Liu, Rex Zhou, Dengdu Jiang, Rong Fu, Meng Ding, Defu Cao

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [audit](<https://devfeed.tech/tags/audit.md>), [automl](<https://devfeed.tech/tags/automl.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [budget](<https://devfeed.tech/tags/budget.md>), [capability](<https://devfeed.tech/tags/capability.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [errors](<https://devfeed.tech/tags/errors.md>), [fault-injection](<https://devfeed.tech/tags/fault-injection.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [gate](<https://devfeed.tech/tags/gate.md>), [llm](<https://devfeed.tech/tags/llm.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [safety](<https://devfeed.tech/tags/safety.md>), [search](<https://devfeed.tech/tags/search.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

A naive 8B LLM critic pushed search below the no-critic random-search floor on 7 of 8 tabular datasets, with losses up to 0.245 AUC; the impact varied by dataset, search frontier and budget. Controlled fault injection suggests non-directional mistakes are nearly costless under best-so-far search, while directional mistakes consume finite search opportunities. Stronger critics reduced losses on the evaluated suite, but the study establishes no universal capability threshold. A simple validation gate and a learned detector also reduced realized loss, though neither was guaranteed to preserve the full-budget no-critic baseline. The authors propose auditing critics and gating feedback when capability or verification is insufficient, and announce CRITFAULT, a benchmark for detecting harmful critic feedback in AutoML.

### Source excerpt

arXiv:2611.19037v1 Announce Type: cross Abstract: LLM critics increasingly steer iterative search in agentic AutoML, prompt optimization, and self-refinement--evaluated for whether they help, but rarely for whether they can hurt. We isolate the critic's marginal effect through a four-arm audit and characterize the loop's tolerance through controlled fault injection. We show a naive 8B critic drives search below the no-critic random-search floor on 7 of 8 tabular datasets, losing up to 0.245 AUC in the worst case; the magnitude varies strongly with dataset, frontier, and budget. When: stronger critics attenuate the loss on the evaluated suite, without establishing a universal capability threshold. Why: controlled fault injection shows that non-directional errors are nearly costless under best-so-far search, whereas directional errors consume finite search opportunities; an idealized effective-sample calculation yields a necessary condition, $q(m-1) > \beta$, not a deployment certificate. How: a simple observable validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline. We will release CRITFAULT, to our knowledge the first benchmark for detecting harmful critic feedback in AutoML. A critic is not a free addition: audit before trusting, and gate when capability or verification is insufficient.

## Privacy-Preserving Clustering Algorithm and Parameter Selection for Collaborative Population Stratification Using Distributed PCA

DevFeed: [Privacy-Preserving Clustering Algorithm and Parameter Selection for Collaborative Population Stratification Using Distributed PCA](<https://devfeed.tech/articles/privacy-preserving-clustering-algorithm-and-parameter-selection-for-collaborative-population-stratification-using-distributed-pca-138805.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18925>)

Author: Maryam Ghasemian, Lynette Hammond Gerido, Erman Ayday

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [clustering](<https://devfeed.tech/topics/clustering.md>), [Differential Privacy](<https://devfeed.tech/topics/differential-privacy.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [clustering](<https://devfeed.tech/tags/clustering.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [selection](<https://devfeed.tech/tags/selection.md>)

### AI overview

The framework selects clustering methods and hyperparameters from privacy-protected data for collaborative population stratification. Each party computes local top-r principal-component information; a server aggregates low-rank summaries to form a global basis without a public reference basis. It supports three locally differentially private pipelines: PCA then Noise, Noise then PCA, and Noise-Only. The server scores configurations with internal validation metrics on perturbed data, a post-processing step. Experiments on openSNP and 1000 Genomes report that PCA-based pipelines outperform Noise-Only in clustering utility, communication cost, and runtime, and have lower membership-inference attack power at small privacy budgets. These findings apply to the reported experimental setting.

### Source excerpt

arXiv:2611.18925v1 Announce Type: cross Abstract: We propose a privacy-preserving selection layer for collaborative population stratification workflows, enabling an honest-but-curious server to recommend a strong clustering algorithm-hyperparameter configuration from privacy protected shares. Instead of using a fixed workflow such as PCA followed by K-Means with a preset number of clusters, the framework compares multiple clustering methods and parameter settings under privacy constraints. To avoid relying on a public reference basis, we construct a shared low-dimensional representation through a distributed-PCA (DPCA) procedure in which each party computes local top-$r$ principal-component information and the server aggregates only low-rank summaries to recover a global basis. Building on this representation, the framework supports three LDP-protected pipelines: PCA then Noise, Noise then PCA, and Noise-Only. The server evaluates clustering methods and parameter settings using internal validation metrics and recommends a top-performing configuration from the perturbed data; this step is pure post-processing. Experiments on openSNP and 1000 Genomes show that the server-side selection layer reliably identifies strong settings from LDP-protected data. Across both datasets, the PCA-based pipelines outperform Noise-Only in clustering utility, communication cost, and runtime, and show lower membership-inference attack (MIA) power at small privacy budgets. Overall, they provide the best utility--scalability trade-off in our setting.

[Next page](<https://devfeed.tech/topics/hyperparameter-optimization.md?cursor=WyIyMDI2LTExLTE3VDA1OjAwOjAwKzAwOjAwIiwgIjRlZjVlYTYzLTlkMDItNDNmNi04NWMxLTg4OTRlZjJjMTkyMCJd>)