# thompson sampling

Thompson Sampling is a Bayesian algorithm for balancing exploration and exploitation in multi-armed bandit problems by selecting an action according to its probability of being optimal.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## DACE: Diversity-Driven Adversarial Co-Evolution for Robust LLM Safety Alignment

DevFeed: [DACE: Diversity-Driven Adversarial Co-Evolution for Robust LLM Safety Alignment](<https://devfeed.tech/articles/dace-diversity-driven-adversarial-co-evolution-for-robust-llm-safety-alignment-139998.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22278>)

Author: Peng Yu, Xiaoyu Wen, Zhida He, Ziyuan Zhou, Han Qi, Shao Zhang, Qiaosheng Zhang, Ying Wen, Chaochao Lu

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Adversarial attacks](<https://devfeed.tech/topics/adversarial-attacks.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [attack](<https://devfeed.tech/tags/attack.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [automated](<https://devfeed.tech/tags/automated.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bernoulli](<https://devfeed.tech/tags/bernoulli.md>), [beta](<https://devfeed.tech/tags/beta.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [defender](<https://devfeed.tech/tags/defender.md>), [defense](<https://devfeed.tech/tags/defense.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-safety](<https://devfeed.tech/tags/llm-safety.md>), [robust](<https://devfeed.tech/tags/robust.md>), [safety](<https://devfeed.tech/tags/safety.md>)

### AI overview

DACE targets attacker strategy collapse and defender forgetting during adversarial co-evolution for LLM safety alignment. It represents attacks across risk category and attack style, rewarding normalized marginal coverage gain to encourage strategy-level exploration. For defense, a Bayesian replay pool tracks threats with Beta-Bernoulli posteriors, refreshes them through time decay, and samples them using Thompson sampling. The abstract reports improved robustness to out-of-distribution attacks, broader attacker strategy coverage, and preserved general capabilities across safety, automated-attacker, and general-capability benchmarks; it provides no benchmark-specific results.

### Source excerpt

arXiv:2611.22278v1 Announce Type: new Abstract: Ensuring robust safety alignment of large language models (LLMs) is increasingly difficult as adversarial attacks evolve and outpace alignment pipelines built on pre-collected data. Existing co-evolutionary frameworks let the defender chase a moving attacker, yet their training dynamics expose two compounding pathologies: \emph{attack strategy collapse}, where the attacker overfits to a narrow set of high-reward rewrites, and \emph{defense adversarial forgetting}, where the defender loses competence against earlier attacks as the attack distribution drifts. Existing remedies are partial: attacker-side diversity rewards score textual novelty rather than strategy novelty, and defender-side replay relies on discrete judge scores that ignore estimation uncertainty, response stochasticity, and defender non-stationarity. We introduce \textbf{\method}, a diversity-driven adversarial co-evolution framework that targets both pathologies jointly. On the attacker side, \method couples an explicit two-dimensional strategy space (risk category $\times$ attack style) with a \emph{normalized marginal coverage gain} reward, providing a bounded, non-vanishing exploration signal at the strategy layer. On the defender side, \method maintains a unified \emph{Bayesian adversarial replay pool} whose Beta--Bernoulli threat posteriors are refreshed by time decay and sampled via Thompson sampling, anchoring defender training to an evolving threat landscape. Across standard safety, automated-attacker, and general-capability benchmarks, \method improves robustness to out-of-distribution attacks while preserving general capabilities, and yields broader attacker strategy coverage.

## Towards Optimism-Pessimism Trade-off in Model-based Offline-to-Online Reinforcement Learning

DevFeed: [Towards Optimism-Pessimism Trade-off in Model-based Offline-to-Online Reinforcement Learning](<https://devfeed.tech/articles/towards-optimism-pessimism-trade-off-in-model-based-offline-to-online-reinforcement-learning-141744.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22172>)

Author: Guochen Zhou, Yijun Yang, Qiqi Duan, Qing Su, Weiming Ou, Li Shen, Shihao Ji, Yuhui Shi, Zhisong Zhang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [bandit](<https://devfeed.tech/tags/bandit.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [model](<https://devfeed.tech/tags/model.md>), [offline](<https://devfeed.tech/tags/offline.md>), [online](<https://devfeed.tech/tags/online.md>), [pool](<https://devfeed.tech/tags/pool.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>)

### AI overview

The paper proposes HiP3 for adapting offline-trained model-based reinforcement learning policies to online tasks under distribution shift. It identifies a trade-off: optimistic policies adapted better to novel tasks in motivating experiments, while pessimistic ones performed better on the same tasks. HiP3 represents both objectives during offline training, using MOSAIC to build a diverse Pareto policy pool; a contextual bandit then hierarchically selects a policy at each online interaction step. The authors report the strongest aggregate performance across same and novel tasks and an advantage under stronger shifts. The supplied abstract does not provide numerical results.

### Source excerpt

arXiv:2611.22172v1 Announce Type: new Abstract: Model-based offline-to-online reinforcement learning (RL) enables sample-efficient adaptation by leveraging offline pre-training for online fine-tuning. However, the distribution shifts between offline and online stages often hinder fine-tuning performance. Many existing methods encode the optimism-pessimism trade-off through a single-objective formulation. Selecting its trade-off coefficient through online fine-tuning can require costly bi-level optimization. We identify the balance established during offline training as a key challenge: in our motivating experiments, optimistic policies adapt better to novel tasks, whereas pessimistic policies perform better on the same online tasks. To address this challenge, we propose a bi-objective formulation that captures this trade-off, yielding a Pareto policy pool during offline training. This policy pool enables adaptive selection during online fine-tuning across diverse online tasks. To generate the pool, we introduce Multi-Objective Soft Actor-critIC (MOSAIC), which solves bi-objective problems and constructs diverse Pareto policies. After offline training, a contextual bandit algorithm hierarchically selects the most suitable policy for fine-tuning at each online interaction step. Empirically, our pipeline, Hierarchical Pareto Policy Pool (HiP3), achieves the strongest aggregate performance across both same and novel online tasks and maintains its advantage under stronger distribution shifts. Comprehensive ablations further clarify the roles of the policy pool, MOSAIC-based pool construction, and the online selection mechanism.

## Kernel-based guarantees for nonlinear parametric models in Bayesian optimization

DevFeed: [Kernel-based guarantees for nonlinear parametric models in Bayesian optimization](<https://devfeed.tech/articles/kernel-based-guarantees-for-nonlinear-parametric-models-in-bayesian-optimization-141559.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.13160>)

Author: Rafael Oliveira

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [confidence-bounds](<https://devfeed.tech/tags/confidence-bounds.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [framework](<https://devfeed.tech/tags/framework.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [networks](<https://devfeed.tech/tags/networks.md>), [neural-networks](<https://devfeed.tech/tags/neural-networks.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [parameter](<https://devfeed.tech/tags/parameter.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>), [thompson-sampling](<https://devfeed.tech/tags/thompson-sampling.md>)

### AI overview

The paper develops a kernel-based framework for obtaining confidence bounds for nonlinear parametric models trained on adaptively collected data. Kernels over parameter space induce reproducing kernel Hilbert space structures over the model class, and the bounds apply to models trained with a broad class of regularized losses. The authors state that these bounds can establish convergence guarantees for approximate Thompson sampling and support analysis of broader likelihood-free Bayesian optimization methods. The supplied text is the abstract, so it does not provide details of assumptions or guarantee strength.

### Source excerpt

arXiv:2605.13160v2 Announce Type: replace-cross Abstract: Modern Bayesian optimization and adaptive sampling methods increasingly rely on nonlinear parametric models, such as neural networks, yet theoretical guarantees for such models under adaptive data collection remain limited. Existing analyses largely focus on Gaussian processes, kernel machines, linear models, or linearized neural approximations, leaving a gap between theory and the nonlinear models used in practice. We develop a kernel-based framework for analyzing regularized parametric models trained on adaptively collected data. Our approach uses kernels over the parameter space to induce reproducing kernel Hilbert space structures over the corresponding model class, yielding confidence bounds for models trained with a broad class of regularized losses. We show how these bounds can establish convergence guarantees for approximate Thompson sampling and provide a foundation for analyzing broader likelihood-free Bayesian optimization methods. These results provide a unified route to analyzing nonlinear models in Bayesian optimization and related adaptive optimization settings.

## RecHarness: A Bandit-Routed Agentic Harness for Self-Evolving Recommender Systems

DevFeed: [RecHarness: A Bandit-Routed Agentic Harness for Self-Evolving Recommender Systems](<https://devfeed.tech/articles/recharness-a-bandit-routed-agentic-harness-for-self-evolving-recommender-systems-139798.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.29241>)

Author: Haoran Ling, Yuecheng Li, Zeyu Song, Jing Yao, Shuwen Kang, Songcheng Ren, Chi Lu, Wenjin Wu, Peng Jiang

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [advertising](<https://devfeed.tech/tags/advertising.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [bandit](<https://devfeed.tech/tags/bandit.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [harness](<https://devfeed.tech/tags/harness.md>), [recommender-systems](<https://devfeed.tech/tags/recommender-systems.md>)

### AI overview

RecHarness separates recommender optimization into direction selection and implementation: a bandit router uses historical validation feedback to choose a modification direction, then an LLM proposes a hypothesis and executable code edit within it. When local edits stagnate, a jump-basin mechanism activates a structural-jump option. The abstract reports more consistent gains and better use of limited trials than LLM-directed search across multiple tasks, datasets, and model backbones. In a 7-day A/B test on a large-scale short-video advertising platform, the selected candidate increased ADVV by 2.084%, revenue by 0.534%, and exposure by 0.559%. These are abstract-level claims; no experimental details are provided here.

### Source excerpt

arXiv:2607.29241v2 Announce Type: replace-cross Abstract: Optimizing recommender models remains heavily reliant on manual iteration over architectures, objectives, and training strategies. While LLM-based agents can automate this trial-and-error process, relying on LLMs to both select modification directions and formulate concrete hypotheses can lead to unstable search under limited trial budgets. Inspired by the above challenge, we propose RecHarness, a Bandit-Routed Agentic Harness for automated recommender model optimization. RecHarness decouples the optimization process into two steps: a bandit router selects the next modification direction according to historical validation feedback, while the LLM generates a concrete optimization hypothesis and executable code edit within the selected direction. To sustain long-horizon exploration, RecHarness uses a jump-basin mechanism to activate a structural-jump arm when local edits stagnate. Across multiple recommendation tasks, datasets, and model backbones, RecHarness achieves more consistent performance improvements and uses limited trial budgets more effectively than LLM-directed search. During a 7-day online A/B test on a large-scale short-video advertising platform, the selected candidate improves ADVV by 2.084%, Revenue by 0.534%, and Exposure by 0.559%. Code is available at: https://github.com/6lyc/RecHarness.

## Certified Policy Optimisation for Nested Causal Bandits via PAC-Bayes Risk

DevFeed: [Certified Policy Optimisation for Nested Causal Bandits via PAC-Bayes Risk](<https://devfeed.tech/articles/certified-policy-optimisation-for-nested-causal-bandits-via-pac-bayes-risk-134987.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.29788>)

Author: Tim Woydt, Paul-David Zuercher

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [causality](<https://devfeed.tech/topics/causality.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bandit](<https://devfeed.tech/tags/bandit.md>), [bandits](<https://devfeed.tech/tags/bandits.md>), [bayes](<https://devfeed.tech/tags/bayes.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [certificate](<https://devfeed.tech/tags/certificate.md>), [context](<https://devfeed.tech/tags/context.md>), [controller](<https://devfeed.tech/tags/controller.md>), [coupling](<https://devfeed.tech/tags/coupling.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [decision](<https://devfeed.tech/tags/decision.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [nested](<https://devfeed.tech/tags/nested.md>), [optimisation](<https://devfeed.tech/tags/optimisation.md>), [policy](<https://devfeed.tech/tags/policy.md>), [regression](<https://devfeed.tech/tags/regression.md>), [risk](<https://devfeed.tech/tags/risk.md>)

### AI overview

The paper models nested strategic and tactical decisions as Nested Contextual Causal Bandits (NCCBs), where an action at one level changes the next level's context distribution. Its Nested Causal Thompson Sampling (NCTS) draws a mechanism-factorised belief once per episode and acts recursively. A causal PAC-Bayesian excess-risk bound can certify a candidate policy using historical data, off-policy and at any time. In hierarchical SCM experiments, the factorised posterior transferred better zero-shot under exogenous shifts than matched RFF-GP joint regression; recursive meta-to-inner commits outperformed joint commits in distribution, and certificates tightened as offline data grew. The authors use this to propose progressive certified handover, switching timescales independently from legacy controllers to NCTS when gains are certifiable.

### Source excerpt

arXiv:2605.29788v2 Announce Type: replace Abstract: Critical sequential decisions are rarely single-timescale: a strategic decision causally shapes the context in which every subsequent tactical choice is made; standard bandit and reinforcement-learning theory does not capture this causal coupling between timescales. We formalise the problem class as Nested Contextual Causal Bandits (NCCBs), a hierarchical SCM where each level's action sets the next level's context distribution, and propose Nested Causal Thompson Sampling (NCTS), which draws one mechanism-factorised belief per episode and acts recursively under it. Our main theoretical result is a causal PAC-Bayesian excess-risk bound that certifies any candidate deployment policy from historic data alone, off-policy and anytime, answering the deployment question: can we trust this agent here, and at what risk? Experiments on a hierarchical SCM show that, against a matched RFF-GP joint regression on the same function class, the factorised SCM-mechanism posterior transfers significantly better zero-shot under exogenous distribution shifts, the recursive meta-to-inner commit significantly dominates the joint-commit alternative in distribution, and the certificate significantly contracts as offline data accumulates. Combining these results, we establish progressive certified handover, a safe-deployment method: each timescale flips from a legacy controller to NCTS when gains can be certified, independently of the others.

## Ballad: Bandit-Based LLM Routing for Automated Heuristic Discovery

DevFeed: [Ballad: Bandit-Based LLM Routing for Automated Heuristic Discovery](<https://devfeed.tech/articles/ballad-bandit-based-llm-routing-for-automated-heuristic-discovery-132325.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17241>)

Author: Samidha Verma, Ankit Anand, Sayan Ranu

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Model Routing](<https://devfeed.tech/topics/model-routing.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [api](<https://devfeed.tech/tags/api.md>), [bandit](<https://devfeed.tech/tags/bandit.md>), [combinatorial-optimization](<https://devfeed.tech/tags/combinatorial-optimization.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [dag](<https://devfeed.tech/tags/dag.md>), [decision](<https://devfeed.tech/tags/decision.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [genealogy](<https://devfeed.tech/tags/genealogy.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [np-hard](<https://devfeed.tech/tags/np-hard.md>), [online](<https://devfeed.tech/tags/online.md>), [open-weight-models](<https://devfeed.tech/tags/open-weight-models.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [policy](<https://devfeed.tech/tags/policy.md>), [routing](<https://devfeed.tech/tags/routing.md>)

### AI overview

Ballad adaptively routes calls among four LLMs during automated heuristic discovery, addressing the fact that a model's usefulness can shift as outputs seed later search steps and rewards become entangled across the heuristic population. Its policy favors rare, high-scoring heuristics, propagates credit through the population's genealogy as a DAG, and discounts stale observations. Across five NP-hard optimization problems, the authors report that Ballad matched the best individual model in hindsight and sometimes surpassed every individual model, while cutting API costs by up to 65.6% compared with always using the strongest model. The abstract reports no consistent heuristic-quality advantage for larger, costlier models.

### Source excerpt

arXiv:2611.17241v1 Announce Type: new Abstract: Modern automated heuristic discovery (AHD) systems increasingly rival hand-crafted heuristics by leveraging Large Language Models (LLMs) to iteratively mutate and refine solutions. A natural assumption is that more powerful and more expensive models consistently yield better heuristics. We challenge this empirically: across multiple NP-hard combinatorial optimization problems, we find no consistent advantage for larger, costlier models. This raises a deeper question: given a pool of LLMs spanning frontier and freely hosted or open-weight models, how should one adaptively select which model to invoke at each step of the search? The challenge is non-trivial. The optimal LLM may shift over the course of a run, and because one LLM's output seeds the context for subsequent calls, each model's reward signal is non-stationary and entangled across the heuristic population, thus violating the independence assumptions of classical online decision-making settings. We address this through Ballad (Bandit-based LLM Routing for Automated Heuristic Discovery), an online bandit algorithm that learns to orchestrate a pool of LLMs into an ensemble that matches or exceeds what any soloist LLM achieves alone. This orchestration is powered by an LLM selection policy that favors rare, high-scoring heuristics over stable averages, a DAG-based credit propagation mechanism through the genealogy of the heuristic population, and discounting of stale observations to remain attuned to each model's current utility. Across five NP-hard combinatorial optimization problems and a pool of four models of varying capability and cost, Ballad not only matches the best individual LLM in hindsight but, in several cases, surpasses every individual model while reducing API cost by up to 65.6% relative to always using the strongest model.

## Square-Root Regret for Adversarial Multiplayer Bandits without Collision Information or Shared Randomness

DevFeed: [Square-Root Regret for Adversarial Multiplayer Bandits without Collision Information or Shared Randomness](<https://devfeed.tech/articles/square-root-regret-for-adversarial-multiplayer-bandits-without-collision-information-or-shared-randomness-132838.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.05688>)

Author: Chenyu Gan

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [bandits](<https://devfeed.tech/tags/bandits.md>), [communication](<https://devfeed.tech/tags/communication.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [multiplayer](<https://devfeed.tech/tags/multiplayer.md>), [randomness](<https://devfeed.tech/tags/randomness.md>)

### AI overview

The paper refutes the conjecture that two-player adversarial bandits without collision information require regret of order T^(2/3). For m labeled players and K>m arms, it gives expected regret O(m^(3/2)√(KT) log²(KmT)) against every oblivious adversary, optimal in K and T up to logarithmic factors. Players observe only their own rewards, with no shared randomness or communication channel. The strategy sends bits through deliberate collisions and charges delay-related regret to arms with positive reward; uses exponential weights over arm sets with a frequently resampled coordinator arm and a follower assignment that changes only O(√(mT/K)) times in expectation; and teaches followers a privately chosen row index through reward observations. The bound applies even when reward matrices are chosen with knowledge of the public table; a random table works with probability 1−O(1/(KmT)).

### Source excerpt

arXiv:2610.05688v3 Announce Type: replace Abstract: We study cooperative multiplayer bandits with adversarial rewards and no collision information. In each of $T$ rounds, each of $m$ labeled players picks one of $K>m$ arms. A player alone on its arm receives that arm's reward in $[0,1]$; players who share an arm receive $0$. Each player observes only its own reward, so it cannot distinguish a collision from an arm with zero reward, and there is neither shared randomness nor a communication channel. Bubeck, Li, Peres and Sellke (2020) obtained regret $\widetilde O(KT^{3/4})$ for two players and $\widetilde O(mK^{3/2}T^{1-1/(2m)})$ for $m$ players, and conjectured that regret $\Omega(T^{2/3})$ is unavoidable already for two players. We refute this conjecture. We give a strategy whose expected regret against every oblivious adversary is $O(m^{3/2}\sqrt{KT}\log^2(KmT))$, which is optimal in $K$ and $T$ up to logarithmic factors. Three ideas make this possible. (i) Bits are sent through deliberate collisions as random parity checks, and the regret spent while a message is delayed is charged to the number of arms with positive reward rather than to elapsed time: an adversary that jams the channel with zero rewards forfeits the regret it could inflict. (ii) An exponential-weights learner over sets of arms combines a frequently resampled coordinator arm with a follower assignment that changes only $O(\sqrt{mT/K})$ times in expectation; delayed updates are reconstructed exactly. (iii) We teach the followers a privately chosen row index in a fixed public table using reward observations alone, allowing them to coordinate without shared randomness. The bound holds simultaneously for all reward matrices, even ones chosen with knowledge of the table, and a random table works with probability $1-O(1/(KmT))$.

## Thompson Sampling using Prior-fitted Diffusion Transformers

DevFeed: [Thompson Sampling using Prior-fitted Diffusion Transformers](<https://devfeed.tech/articles/thompson-sampling-using-prior-fitted-diffusion-transformers-132032.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.14656>)

Author: Sihwa Park, Jingsen Zhu, Vinamr Jain, Sheng-Yen Chou, Alexander Terenin

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [diffusion-transformers](<https://devfeed.tech/topics/diffusion-transformers.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [diffusion](<https://devfeed.tech/tags/diffusion.md>), [diffusion-transformers](<https://devfeed.tech/tags/diffusion-transformers.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [global](<https://devfeed.tech/tags/global.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scaling-laws](<https://devfeed.tech/tags/scaling-laws.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [thompson-sampling](<https://devfeed.tech/tags/thompson-sampling.md>), [training](<https://devfeed.tech/tags/training.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

Prior-fitted diffusion transformers perform Thompson sampling in context at inference time after training with the prior-fitted network approach. The work introduces practical non-Gaussian priors for black-box functions, including priors over unimodal functions that Gaussianity cannot provide. It also measures the tradeoff between model size and pre-training iterations, reporting scaling laws similar to those seen in transformers for other modalities. On global optimization benchmarks, the authors report improved performance over traditional Gaussian-process pipelines. These findings are limited to the benchmarks described in the abstract.

### Source excerpt

arXiv:2611.14656v1 Announce Type: cross Abstract: Thompson sampling is natural randomized strategy for optimizing unknown functions in a black-box manner. In continuous global optimization, Thompson sampling has largely been restricted to Gaussian process priors, as these are essentially the only class for which computing the posterior distribution has been tractable in practice. In this work, we introduce \emph{prior-fitted diffusion transformers}, which are diffusion models that perform Thompson sampling in-context at inference time, after being pre-trained using the prior-fitted network paradigm. To apply these models, we develop a set of practical non-Gaussian priors for general black-box functions---including those that have characteristics that cannot be achieved under Gaussianity, such as priors over unimodal functions. We also quantify the tradeoff between model size and the number of pre-training iterations, and show that these models follow scaling laws akin to those observed in transformers defined over other modalities. Across a set of global optimization benchmarks, we show that prior-fitted diffusion transformers with non-Gaussian priors can achieve improved performance compared to traditional pipelines built on Gaussian processes.

## High Probability Risk Control for Online Policy Learning

DevFeed: [High Probability Risk Control for Online Policy Learning](<https://devfeed.tech/articles/high-probability-risk-control-for-online-policy-learning-128662.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13163>)

Author: Yihong Guo, Drew Prinster, Suchi Saria, Anqi Liu

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [ablation](<https://devfeed.tech/tags/ablation.md>), [agent](<https://devfeed.tech/tags/agent.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [bandit](<https://devfeed.tech/tags/bandit.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bootstrap](<https://devfeed.tech/tags/bootstrap.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [confidence-bounds](<https://devfeed.tech/tags/confidence-bounds.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [data-collection](<https://devfeed.tech/tags/data-collection.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [history](<https://devfeed.tech/tags/history.md>), [learning](<https://devfeed.tech/tags/learning.md>), [online](<https://devfeed.tech/tags/online.md>), [policy](<https://devfeed.tech/tags/policy.md>), [probability](<https://devfeed.tech/tags/probability.md>), [risk](<https://devfeed.tech/tags/risk.md>)

### AI overview

Online policy updates create feedback covariate shift, making candidate risks harder to evaluate and safe policies harder to select. The paper proposes a permutation-based weighted risk evaluator whose weights are approximated with Sinkhorn, then selects policies only when Gaussian-bootstrap upper confidence bounds fall below the risk threshold. It proves the evaluator's variance upper bound is no larger than that of standard importance weighting, and establishes asymptotic simultaneous risk bounds and high-probability risk control for the selected policy. In evaluations on contextual bandits, safe question answering, and medical claim filtering, the method reportedly maintained target risk, lowered risk and estimation variance versus baselines, and supported reliable selection.

### Source excerpt

arXiv:2611.13163v1 Announce Type: new Abstract: We study risk control for online policy learning, where an agent adaptively collects data and aims to learn a new policy whose risk stays below a tolerance threshold at every round with high probability. Such policy updates and data collection induce feedback covariate shift (FCS): observed data depend on history through evolving policies and the data-collection process. FCS brings two substantive problems for risk control: 1) efficient risk evaluation for each candidate policy and 2) valid policy selection with high-probability risk control. To address these, we propose a permutation-based weighted risk evaluator with weights efficiently approximated by the Sinkhorn algorithm and select policies whose estimated risk upper confidence bounds (UCB) fall below the target threshold, with the UCB obtained through a proposed Gaussian bootstrap method. Theoretically, we prove that our risk evaluator has a no-larger variance upper bound than that of the standard importance-weighted evaluator. Our policy control method also yields an asymptotically valid simultaneous risk bound and provides asymptotic high-probability risk control for the selected policy. Empirically, we evaluate our method on the online contextual bandit problem, safe question answering benchmark, and Medical Claim Filtering tasks. Our method maintains the target risk level, achieves lower risk than baseline methods, has lower variance in risk estimation, and achieves reliable policy selection with high-probability risk control. Further ablation studies support our theoretical claims.

## Visible Tails: Characterizing Adaptation to Unknown Moments in Heavy-Tailed Bandits

DevFeed: [Visible Tails: Characterizing Adaptation to Unknown Moments in Heavy-Tailed Bandits](<https://devfeed.tech/articles/visible-tails-characterizing-adaptation-to-unknown-moments-in-heavy-tailed-bandits-125519.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12140>)

Author: Tuan Ngoc Do, Hai Duc Nguyen

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [Predictive AI](<https://devfeed.tech/topics/predictive-ai.md>)

Tags: [bandits](<https://devfeed.tech/tags/bandits.md>), [core](<https://devfeed.tech/tags/core.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [lower-bound](<https://devfeed.tech/tags/lower-bound.md>), [quantile](<https://devfeed.tech/tags/quantile.md>), [tails](<https://devfeed.tech/tags/tails.md>), [variance](<https://devfeed.tech/tags/variance.md>), [visibility](<https://devfeed.tech/tags/visibility.md>)

### AI overview

The paper identifies variance-core visibility as the condition under which heavy-tailed bandit learners can adapt to unknown reward moments at constant-factor cost: the mean hidden in any rare event must be at most a constant times the standard error of the rewards that remain. A quantile-trimmed UCB index, given the visibility constants, reaches the central-moment benchmark; the same index has logarithmic expected regret on instances with a visible optimal arm and finite-mean rewards. No single strategy adapts across all visibility levels. On an explicit family without higher moments, the regret coefficient and information lower bound scale as Δ⁻¹e^(A/Δ), showing exponential dependence on inverse gap even within one visibility class.

### Source excerpt

arXiv:2611.12140v1 Announce Type: new Abstract: Heavy-tailed bandit learners without moment parameters face an obstruction: the rewards that carry an arm's mean can stay unseen. We address the remaining part of a COLT open problem by characterizing the optimal-arm laws that permit adaptation to unknown moments at constant-factor cost. The condition, variance-core visibility, is one-sided and translation-invariant: the mean hidden in any rare event is at most a constant multiple of the standard error of the rewards that remain. A quantile-trimmed UCB index that knows the visibility constants attains the central-moment benchmark on this class, and no single strategy adapts across all visibility levels. The same index achieves logarithmic expected regret on every instance with a visible optimal arm and finite-mean rewards. On an explicit family with no higher moment, gap $\Delta$ and constant $A$, its logarithmic coefficient and the information lower bound are of order $\Delta^{-1}e^{A/\Delta}$, exponential in the inverse gap within one visibility class.

## UCB-min: Combining Bandits Under Variance Heterogeneity (Extended Version)

DevFeed: [UCB-min: Combining Bandits Under Variance Heterogeneity (Extended Version)](<https://devfeed.tech/articles/ucb-min-combining-bandits-under-variance-heterogeneity-extended-version-125552.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12427>)

Author: Lilia Izri, Benoit Groz, Silviu Maniu

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [bandits](<https://devfeed.tech/tags/bandits.md>), [clinical-trials](<https://devfeed.tech/tags/clinical-trials.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [educational](<https://devfeed.tech/tags/educational.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [framework](<https://devfeed.tech/tags/framework.md>), [independent](<https://devfeed.tech/tags/independent.md>), [objectives](<https://devfeed.tech/tags/objectives.md>), [online](<https://devfeed.tech/tags/online.md>), [online-advertising](<https://devfeed.tech/tags/online-advertising.md>), [performance](<https://devfeed.tech/tags/performance.md>), [recommender-systems](<https://devfeed.tech/tags/recommender-systems.md>), [reward](<https://devfeed.tech/tags/reward.md>), [robust](<https://devfeed.tech/tags/robust.md>), [sequential-decision](<https://devfeed.tech/tags/sequential-decision.md>), [strategies](<https://devfeed.tech/tags/strategies.md>), [support](<https://devfeed.tech/tags/support.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [variance](<https://devfeed.tech/tags/variance.md>)

### AI overview

UCB-min addresses arms with unknown, heterogeneous reward variances by adaptively choosing, for each arm, the tighter exploration bonus from a variance-independent method and a variance-aware method, without prior knowledge of the variance regime or reward normalization. The abstract identifies a tradeoff: variance-independent methods are robust but can be loose at low variance, while variance-aware methods depend on reliable variance estimates and support bounds. Using UCB-V as its variance-aware base, UCB-min's regret is bounded by the sum of the two base bounds at the standard exploration constant; with a larger constant, it is bounded up to a constant by their per-arm minimum. Tests on synthetic data, an educational tutoring task, and MovieLens and Steam group-recommendation benchmarks reportedly show it tracking the best base algorithm across the evaluated variance regimes and datasets.

### Source excerpt

arXiv:2611.12427v1 Announce Type: new Abstract: Multi-armed bandits are a natural framework for many real-world sequential decision problems, from recommender systems and clinical trials to educational test recommendation and online advertising. The performance of these algorithms depends on how they explore different arms, and exploration strategies range from variance-independent (e.g., UCB1) to variance-aware (e.g., UCB-V). In practice the reward variance of each arm is rarely known and varies widely across arms, and the reward support is often underspecified. In these cases no single rule is uniformly best: variance-independent methods are robust but loose when variances are small, while variance-aware methods are tight only when their variance estimates and the assumed support bound are reliable. Such heterogeneous, unknown variances arise in many settings, for instance when multiple objectives are scalarized into a single reward. We propose UCB-min, a per-arm adaptive meta-algorithm that selects, for each arm, the tighter of a variance-independent and a variance-aware exploration bonus, without prior knowledge of the variance regime and without reward normalization. We prove that, with UCB-V as variance-aware base, its regret is bounded by the sum of the two base bounds at the standard exploration constant, and by their per-arm minimum, up to a constant, for a larger one. We validate it on synthetic data, an educational tutoring task, and two group-recommendation benchmarks (MovieLens, Steam), showing that UCB-min consistently tracks the best base algorithm across diverse variance regimes and datasets.

## Off-Policy RL with Online Inference Tuning: My Solution to the CoRL 2026 RoboEval Challenge

DevFeed: [Off-Policy RL with Online Inference Tuning: My Solution to the CoRL 2026 RoboEval Challenge](<https://devfeed.tech/articles/off-policy-rl-with-online-inference-tuning-my-solution-to-the-corl-2026-roboeval-challenge-125385.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11464>)

Author: Ilia Larchenko

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [990](<https://devfeed.tech/tags/990.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [competition](<https://devfeed.tech/tags/competition.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [data](<https://devfeed.tech/tags/data.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [failures](<https://devfeed.tech/tags/failures.md>), [full](<https://devfeed.tech/tags/full.md>), [inference](<https://devfeed.tech/tags/inference.md>), [online](<https://devfeed.tech/tags/online.md>), [policy](<https://devfeed.tech/tags/policy.md>), [rds](<https://devfeed.tech/tags/rds.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [reward](<https://devfeed.tech/tags/reward.md>), [rl](<https://devfeed.tech/tags/rl.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [score](<https://devfeed.tech/tags/score.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [thompson-sampling](<https://devfeed.tech/tags/thompson-sampling.md>)

### AI overview

The solution combines off-policy reinforcement learning with targeted demonstrations and recovery data to improve performance on eight bimanual RoboEval tasks. Its generic method trains a π₀.₅-based policy that also serves as its value function, using AWR and RECAP on advantages derived from the official score. Each rollout also tests about 20 inference parameters through Thompson sampling. This reached 0.85 RDS on held-out seeds in two days, though training was far from converged. The competition-specific additions--generated demonstrations reaching full reward and recovery episodes beginning from the policy's own failures--addressed situations local exploration did not find. The submitted policy reached 0.990 RDS on held-out seeds and 0.987 on official seeds.

### Source excerpt

arXiv:2611.11464v1 Announce Type: cross Abstract: The CoRL 2026 RoboEval challenge has eight bimanual tasks, and it ranks a policy not only on whether it succeeds but on how: every stage of the task, done safely, efficiently and with both arms in coordination, scored against human demonstrations. My solution has two parts. The generic part, carried over from my LeHome challenge solution, is off-policy RL on a $\pi_{0.5}$-based policy that is also its own value function. It is trained with AWR and RECAP on advantages of the exact official score, in a loop where every rollout is at the same time a Thompson-sampling trial of about 20 inference parameters. Out of the box, it took the policy to 0.85 RDS on held-out seeds in two days (and it was still very far from convergence). The competition-specific part adds data that local exploration cannot find: generated demonstrations that reach the full reward, and recovery episodes that start from the policy's own failures. The submitted policy reaches 0.990 RDS on held-out seeds and 0.987 on the official seeds.

## Action-Centered Causal Bandits for Budget-Constrained Promotion Targeting

DevFeed: [Action-Centered Causal Bandits for Budget-Constrained Promotion Targeting](<https://devfeed.tech/articles/action-centered-causal-bandits-for-budget-constrained-promotion-targeting-123336.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08990>)

Author: Yifan Lin, Kaixu Yang, Di Mo

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [causality](<https://devfeed.tech/topics/causality.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [bandits](<https://devfeed.tech/tags/bandits.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bias](<https://devfeed.tech/tags/bias.md>), [budget](<https://devfeed.tech/tags/budget.md>), [contextual-bandits](<https://devfeed.tech/tags/contextual-bandits.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [effect](<https://devfeed.tech/tags/effect.md>), [offline](<https://devfeed.tech/tags/offline.md>), [online](<https://devfeed.tech/tags/online.md>), [promotion](<https://devfeed.tech/tags/promotion.md>)

### AI overview

DR-CATE-CBwK targets incremental outcomes under a global intervention budget by learning treatment contrasts directly, avoiding the added statistical complexity of modeling a nonlinear action-independent outcome baseline. It warm-starts with an offline doubly robust learner, updates contrasts online with an estimator that remains unbiased under any predictable baseline estimate without inverse-propensity weighting, and uses a primal-dual controller with an affordable-action rule to enforce the effective-cost budget pathwise. For linear CATEs in a stationary i.i.d. setting, the paper proves high-probability regret of Õ(Kd√T). On a semi-synthetic job-marketplace benchmark, its Thompson-restricted variant had the lowest regret among evaluated budget-feasible methods in the operating regime, reducing regret by about 20% versus the strongest reward-based baseline without exceeding the effective-cost budget. The abstract also reports that a specified separate-arm reward regression can have asymptotic contrast bias under action-dependent logging and shared-baseline misspecification.

### Source excerpt

arXiv:2611.08990v1 Announce Type: new Abstract: Web platforms select among costly interventions for each user to maximize incremental outcomes under a global budget. We formulate this problem as a budget-constrained contextual bandit with heterogeneous treatment effects. Although an action-independent outcome baseline cancels from treatment comparisons, modeling a nonlinear baseline can increase statistical complexity. We propose DR-CATE-CBwK, an action-centered causal bandit that directly learns treatment contrasts. Its online update remains valid under baseline-model misspecification without inverse-propensity weighting. It combines an offline doubly robust learner to warm-start treatment-effect estimates, an online action-centered estimator for sequential contrast learning, and a primal-dual controller with an affordable-action rule to enforce the effective-cost budget pathwise. For linear conditional average treatment effects (CATEs) in a stationary i.i.d. setting, we prove a high-probability (\widetilde{O}(Kd\sqrt{T})) regret bound, where K is the number of priced actions, d the context dimension, and T the horizon. We benchmark against the fluid linear program (LP) under the coverage-guaranteed action rule. We further show that the action-centered estimating moment remains unbiased under any predictable baseline estimate, whereas a specified separate-arm reward regression can exhibit asymptotic contrast bias under action-dependent logging and shared-baseline misspecification. On a semi-synthetic job-marketplace benchmark with real logged features and simulated outcomes, our practical Thompson-restricted variant achieves the lowest regret among evaluated budget-feasible methods in the operating regime. It reduces regret by approximately 20% versus the strongest reward-based baseline, outperforming raw-reward contextual bandits, BwK baselines, and a static offline policy, while never exceeding the effective-cost budget.

## Satisficing as competitive Bayesian weighting: a minimal model of goal-directed choice under an unattained target

DevFeed: [Satisficing as competitive Bayesian weighting: a minimal model of goal-directed choice under an unattained target](<https://devfeed.tech/articles/satisficing-as-competitive-bayesian-weighting-a-minimal-model-of-goal-directed-choice-under-an-unattained-target-120179.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07270>)

Author: Moto Shinriki, Yu Kono, Tatsuji Takahashi

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Estimator](<https://devfeed.tech/topics/estimator.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [attention](<https://devfeed.tech/tags/attention.md>), [bandit-learning](<https://devfeed.tech/tags/bandit-learning.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [choice](<https://devfeed.tech/tags/choice.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

The paper recasts Risk-sensitive Satisficing as Attention Competitive Bayes (ACB): each action's estimate is weighted between observed outcomes and a shared prior mean set by the aspiration level, with a finite trial budget controlling how much evidence can override that goal. When an action is no longer sampled, its estimate relaxes toward the aspiration level without discarding data, a form of "pseudo-forgetting" distinct from explicit forgetting-rate models. If the aspiration is unattainable, exploration continues; in Bernoulli bandit simulations, adjusting the aspiration level based on evidence reduced expected loss. The abstract presents ACB as a minimal, analyzable model and says it yields testable predictions about revaluing abandoned options.

### Source excerpt

arXiv:2611.07270v1 Announce Type: cross Abstract: Animals and humans often do not maximize; they act relative to goals (aspiration levels) that are frequently not yet met. How can a goal that is not (yet) realized organize choice? Deacon (2012) calls such absence-referenced organization \emph{ententional}, and Garc\'{i}a-Valdecasas and Deacon (2024) argue that ends are best understood as constraints rather than as particular states. We examine this idea in a minimal formal setting: Risk-sensitive Satisficing (RS), a bandit learning rule that values each action by its share of past trials multiplied by its surplus or shortfall relative to an aspiration level. We show that RS can be rewritten as a Bayesian-like estimator, Attention Competitive Bayes (ACB), in which the aspiration level is the prior mean of every action and the finite budget of trials sets how strongly evidence can override it. Two consequences follow. First, actions that are no longer sampled are partially overridden by the goal rather than by past observations: their estimates relax toward the aspiration level without any data being discarded. This ``pseudo-forgetting'' distinguishes ACB from models with an explicit forgetting rate. Second, when the aspiration is unattainable, exploration does not settle; in Bernoulli bandit simulations, adjusting the aspiration level from evidence reduced regret (expected loss). ACB offers an analyzable model in which an unattained goal and a limited attentional resource jointly constrain behavior, and it yields testable predictions about how organisms revalue options they have abandoned.

## Meta-Frieren: Beyond Single Level Online Learning

DevFeed: [Meta-Frieren: Beyond Single Level Online Learning](<https://devfeed.tech/articles/meta-frieren-beyond-single-level-online-learning-120569.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07169>)

Author: Vahan Arsenyan, Arnak Dalalyan

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [framework](<https://devfeed.tech/tags/framework.md>), [general](<https://devfeed.tech/tags/general.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [hyperparameter-optimization](<https://devfeed.tech/tags/hyperparameter-optimization.md>), [model](<https://devfeed.tech/tags/model.md>), [online](<https://devfeed.tech/tags/online.md>), [online-learning](<https://devfeed.tech/tags/online-learning.md>), [stage](<https://devfeed.tech/tags/stage.md>), [theoretical](<https://devfeed.tech/tags/theoretical.md>), [thompson-sampling](<https://devfeed.tech/tags/thompson-sampling.md>)

### AI overview

Meta-Frieren addresses online optimization where a meta-learner chooses an agent-environment pair before the selected agent interacts with the environment and receives a reward. It provides a UCB-style rule for meta-level selection, recovering prior guarantees as special cases while extending them to broader settings. The authors instantiate the framework with GP-UCB under Bayesian and frequentist assumptions, and report a high-probability regret bound for Feel-Good Thompson Sampling, specialized to finite and parametric function classes and integrated into Meta-Frieren. Experiments support the theory and demonstrate use in hyperparameter optimization across multiple model classes.

### Source excerpt

arXiv:2611.07169v1 Announce Type: new Abstract: We study a two-stage online optimization problem: at each round, a meta-learner first selects an agent-environment pair, then the chosen agent interacts with the environment, and receives a reward. We introduce Meta-Frieren, a general framework for maximizing cumulative reward in this setting. Our analysis yields a simple UCB-style criterion for meta-level selection and recovers prior guarantees as special cases while extending them to broader settings. We instantiate Meta-Frieren with GP-UCB under Bayesian and frequentist assumptions. We also derive, to our knowledge, the first high-probability regret bound for Feel-Good Thompson Sampling, specialize it to finite and parametric function classes, and integrate it into Meta-Frieren. Experiments support our theoretical results and demonstrate that Meta-Frieren can be successfully applied to hyperparameter optimization across multiple model classes.

## On the Feature Evolution of Temporal Difference Learning

DevFeed: [On the Feature Evolution of Temporal Difference Learning](<https://devfeed.tech/articles/on-the-feature-evolution-of-temporal-difference-learning-116908.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04158>)

Author: Shangzhe Li, Dechen Zhang, Difan Zou, Weitong Zhang

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Representation learning](<https://devfeed.tech/topics/representation-learning.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [bounded](<https://devfeed.tech/tags/bounded.md>), [budget](<https://devfeed.tech/tags/budget.md>), [commit](<https://devfeed.tech/tags/commit.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [continuation](<https://devfeed.tech/tags/continuation.md>), [copy](<https://devfeed.tech/tags/copy.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [d](<https://devfeed.tech/tags/d.md>), [error](<https://devfeed.tech/tags/error.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [exp](<https://devfeed.tech/tags/exp.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [family](<https://devfeed.tech/tags/family.md>), [feature](<https://devfeed.tech/tags/feature.md>), [features](<https://devfeed.tech/tags/features.md>), [global](<https://devfeed.tech/tags/global.md>), [learning](<https://devfeed.tech/tags/learning.md>), [temporal](<https://devfeed.tech/tags/temporal.md>)

### AI overview

For a constructed family of smooth stochastic horizon-H MDPs with unknown quadratic structure, the paper analyzes how finite-width two-layer ReLU critics acquire and propagate predictive features through TD updates. Simultaneously updating hidden weights and the output head gives strictly lower finite-sample prediction risk than matched head-only SGD, with a 2(d+1)-fold larger population quadratic response. The analysis also finds that an imperfect learned continuation can supervise an earlier zero-reward stage; coupled updates with one SGD step per target copy yield a prediction-error transient of exp[-Ω(K/(Hd))], subject to action-independent shifts and sampling and finite-width errors. On a bounded-reward signed-action MDP family, an exploration-then-commit algorithm achieves stated regret Õ(d³H^(1/3)T^(2/3)) using Õ(Hd) global updates.

### Source excerpt

arXiv:2611.04158v1 Announce Type: new Abstract: Temporal-difference (TD) learning trains neural representations using bootstrapped targets generated by evolving value estimates, yet how predictive features emerge and propagate under incomplete neural updates remains poorly understood. We investigate this problem using finite-width two-layer ReLU critics on a constructed family of smooth stochastic horizon-$H$ MDPs with unknown quadratic structure and $d$-dimensional observations. We first show that an informative TD update acquires the quadratic Bellman signal and yields a representation admitting a bounded predictive readout. The simultaneous hidden-weight and head update achieves a strict finite-sample prediction-risk improvement over matched head-only SGD, with a $2(d+1)$-fold larger population quadratic response. We then establish that an imperfect learned continuation can already provide informative supervision to an earlier zero-reward stage. By tracking coupled neural updates with one SGD step per target copy, without requiring intermediate Bellman regressions to be solved, we obtain a uniform prediction-error transient $\exp[-\Omega(K/(Hd))]$, up to action-independent shifts and explicit sampling and finite-width errors. On the constructed signed-action MDP family whose total reward is bounded by 1, we show that an exploration-then-commit algorithm can achieve $\widetilde{O}(d^3H^{1/3}T^{2/3})$ regret over $T$ episodes, using $K=\widetilde{O}(Hd)$ global updates. Our analysis connects bootstrapped feature acquisition, concurrent neural refinement, and finite-budget policy performance.

## Annealing chains that communicate: low-energy cluster structures across engines with one configuration

DevFeed: [Annealing chains that communicate: low-energy cluster structures across engines with one configuration](<https://devfeed.tech/articles/annealing-chains-that-communicate-low-energy-cluster-structures-across-engines-with-one-configuration-113453.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2302.02811>)

Author: Rohit Goswami, Ruhila Goswami, Amrita Goswami, Moritz Sallermann, Sonaly Goswami, Debabrata Goswami

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [bank](<https://devfeed.tech/tags/bank.md>), [call](<https://devfeed.tech/tags/call.md>), [cambridge](<https://devfeed.tech/tags/cambridge.md>), [charges](<https://devfeed.tech/tags/charges.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [core](<https://devfeed.tech/tags/core.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [energy](<https://devfeed.tech/tags/energy.md>), [engines](<https://devfeed.tech/tags/engines.md>), [flow](<https://devfeed.tech/tags/flow.md>), [gamma](<https://devfeed.tech/tags/gamma.md>), [global](<https://devfeed.tech/tags/global.md>), [good](<https://devfeed.tech/tags/good.md>), [hit](<https://devfeed.tech/tags/hit.md>), [low-energy](<https://devfeed.tech/tags/low-energy.md>), [one](<https://devfeed.tech/tags/one.md>), [physics-comp-ph](<https://devfeed.tech/tags/physics-comp-ph.md>), [physics-data-an](<https://devfeed.tech/tags/physics-data-an.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The paper presents anneal, a global optimizer that coordinates multiple search chains to find low-energy cluster structures. Chains choose moves using Thompson sampling over a Normal-Gamma model of quench depth; stalled chains explore symmetry-related moves, and chains retire based on a Good-Turing estimate of unexplored mass. An annealed-flow bank shares minima, while a ledger attributes force-call costs. Across 64 shared starts for LJ₃₈ and LJ₇₅, anneal and GMIN each found the target in all 64 runs; anneal had lower median cost, though a paired test did not distinguish the costs. On the Cambridge Morse table, anneal reached 145 of 244 targets versus GMIN's 131. These are the abstract's reported results; the supplied text does not provide further experimental details.

### Source excerpt

arXiv:2302.02811v4 Announce Type: replace Abstract: The global minimum of a cluster decides which structure a simulation predicts. We present anneal, a global optimizer of communicating chains. Each chain draws its move by Thompson sampling from a Normal-Gamma posterior on quench depth. A stalled chain completes the orbits of a symmetric core. A chain retires on a Good-Turing estimate of its unseen mass, a rule with a Lean derivation. The chains share minima through an annealed-flow bank, and a ledger charges each force call to the chain that spent it. On 64 shared starts of LJ$_{38}$ and LJ$_{75}$ the hop matches the hit rate of GMIN, 64 of 64, at a lower median cost a paired test does not separate. On the Cambridge Morse table it reaches 145 of 244 targets and GMIN 131.

## JailbreakOPT: Iterative Jailbreak Prompt Optimization with Macro-Action

DevFeed: [JailbreakOPT: Iterative Jailbreak Prompt Optimization with Macro-Action](<https://devfeed.tech/articles/jailbreakopt-iterative-jailbreak-prompt-optimization-with-macro-action-113996.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.11425>)

Author: Ge Shi, Jun Yin, Donglin Xie, Fangyi Liu, Yucan Li, Menglin Liu

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [attack](<https://devfeed.tech/tags/attack.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [failure](<https://devfeed.tech/tags/failure.md>), [jailbreak](<https://devfeed.tech/tags/jailbreak.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [policy](<https://devfeed.tech/tags/policy.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [reuse](<https://devfeed.tech/tags/reuse.md>), [safety](<https://devfeed.tech/tags/safety.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [search](<https://devfeed.tech/tags/search.md>), [vulnerabilities](<https://devfeed.tech/tags/vulnerabilities.md>)

### AI overview

JailbreakOPT seeks to improve jailbreak attack success while reducing the response burden of iterative prompt search. It composes reusable prompt transformations into macro-actions to generate standalone attack prompts, then uses contextual Thompson sampling and outcomes from earlier search paths to guide action selection across episodes. The abstract reports higher attack success rates across multiple LLMs and benchmarks than single-action and iterative baselines, but gives no numerical results or further experimental details. The authors suggest the approach can help probe model vulnerabilities and identify failure patterns that may inform defenses.

### Source excerpt

arXiv:2606.11425v2 Announce Type: replace Abstract: Jailbreak attacks expose persistent safety weaknesses in large language models (LLMs), but existing stateless single-turn methods face a trade-off: hand-crafted prompts are expressive but static, while iterative prompt optimization can adapt but often relies on low-level mutations that require many target-model responses. We propose JailbreakOPT, a method that maintains a library of reusable prompt transformations (actions) and composes them into macro-actions for within-episode search, generating standalone attack prompts. To reuse experience across episodes, JailbreakOPT learns an action-selection policy using contextual Thompson sampling and outcomes from prior search paths. Experiments across multiple LLMs and benchmarks show that JailbreakOPT improves attack success rate (ASR) over single-action and iterative baselines. More broadly, exploration and exploitation over human-designed attack strategies can help probe residual vulnerabilities and identify failure patterns that may inform defenses aimed at broader classes of attacks.

## Near-Optimal First-Order Regret for Bandits with Feedback Graphs

DevFeed: [Near-Optimal First-Order Regret for Bandits with Feedback Graphs](<https://devfeed.tech/articles/near-optimal-first-order-regret-for-bandits-with-feedback-graphs-111508.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.40861>)

Author: Mengxiao Zhang, Yuheng Zhang, Haipeng Luo, Paul Mineiro

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [graph theory](<https://devfeed.tech/topics/graph-theory.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [alpha](<https://devfeed.tech/tags/alpha.md>), [arm](<https://devfeed.tech/tags/arm.md>), [bandits](<https://devfeed.tech/tags/bandits.md>), [bridge](<https://devfeed.tech/tags/bridge.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [exp3](<https://devfeed.tech/tags/exp3.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [first](<https://devfeed.tech/tags/first.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [gap](<https://devfeed.tech/tags/gap.md>), [general](<https://devfeed.tech/tags/general.md>), [graph](<https://devfeed.tech/tags/graph.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [iteration](<https://devfeed.tech/tags/iteration.md>), [learner](<https://devfeed.tech/tags/learner.md>), [map](<https://devfeed.tech/tags/map.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>)

### AI overview

For adversarial bandits with a fixed, known strongly observable feedback graph, the paper gives an expected first-order regret bound of Õ(√(αL⋆) + α), where α is the graph's independence number and L⋆ is the best arm's cumulative loss. This narrows the stated gap between worst-case guarantees scaling with α and prior first-order bounds using the larger clique partition number. For graphs with a self-loop at every vertex, the algorithm combines EXP3's update with a sampling rule inspired by exploration by optimization; its sampling distribution comes from a fixed point controlling the one-round regret remainder. The paper states that a sufficiently accurate fixed point can be found in polynomial time by a geometrically convergent iteration. It extends the guarantee to general strongly observable graphs using two-level aggregation.

### Source excerpt

arXiv:2610.40861v1 Announce Type: new Abstract: We study first-order regret guarantees for adversarial bandits with a fixed, known directed feedback graph. For strongly observable graphs, [ACBDK15] establishes a minimax regret of $\widetilde{\Theta}(\sqrt{\alpha T})$, where $\alpha$ denotes the independence number of the graph and $T$ denotes the horizon. On the other hand, the best existing first-order bound scales as $\widetilde{O}(\sqrt{(\kappa+1) L^\star})$ [LLZ20], where $\kappa$ denotes the clique partition number, satisfying $\kappa+1\geq\alpha$, and $L^\star$ denotes the cumulative loss of the best arm, leaving a gap between the graph dependence in worst-case and first-order guarantees. In this paper, we bridge this gap by obtaining an expected regret bound of $\widetilde{O}(\sqrt{\alpha L^\star}+\alpha)$. Our first result is an algorithm for the special case of self-aware graphs, in which every vertex has a self-loop. The algorithm combines the update of EXP3 [ACBFS02] with a sampling rule inspired by exploration by optimization [LS20]. More specifically, we construct a map whose fixed points yield sampling distributions that control the one-round regret remainder in terms of the learner's loss and the graph's independence number, which is crucial in deriving optimal first-order guarantees. We further show that a sufficiently accurate fixed point can be computed in polynomial time by a geometrically convergent iteration. Finally, we extend the algorithm to general strongly observable graphs and achieve the same $\widetilde{O}(\sqrt{\alpha L^\star}+\alpha)$ guarantee by adapting the two-level aggregation approach of [LLZ20].

## Toward Optimal Switching Regret for Multi-Armed Bandits with Oblivious Adversary

DevFeed: [Toward Optimal Switching Regret for Multi-Armed Bandits with Oblivious Adversary](<https://devfeed.tech/articles/toward-optimal-switching-regret-for-multi-armed-bandits-with-oblivious-adversary-111440.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.13547>)

Author: Mengxiao Zhang

Published: 2026-11-02T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [adaptive](<https://devfeed.tech/tags/adaptive.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [adversary](<https://devfeed.tech/tags/adversary.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [arm](<https://devfeed.tech/tags/arm.md>), [bandits](<https://devfeed.tech/tags/bandits.md>), [changes](<https://devfeed.tech/tags/changes.md>), [combines](<https://devfeed.tech/tags/combines.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [learner](<https://devfeed.tech/tags/learner.md>), [learning](<https://devfeed.tech/tags/learning.md>), [local](<https://devfeed.tech/tags/local.md>), [number](<https://devfeed.tech/tags/number.md>), [open](<https://devfeed.tech/tags/open.md>), [sequence](<https://devfeed.tech/tags/sequence.md>), [share](<https://devfeed.tech/tags/share.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [switching](<https://devfeed.tech/tags/switching.md>)

### AI overview

A single algorithm achieves expected switching regret of Õ(√((S+1)KT)) for every number of comparator switches S against an oblivious adversary, even when S is unknown. It combines a fixed-share learner, initialized with a small learning rate, with dyadic-interval subroutines that use randomized learning rates and implicit exploration to find local improvements. A non-uniform prior favors the main learner to limit the cost of maintaining subroutines; when they show sufficient improvement, the main learner's learning rate doubles. The guarantee addresses the unknown-S setting for oblivious adversaries; the abstract notes that the same guarantee is impossible for unknown S under an adaptive adversary.

### Source excerpt

arXiv:2609.13547v2 Announce Type: replace Abstract: We study switching regret in adversarial multi-armed bandits, where the learner competes with an arm sequence that changes at most $S$ times. When $S$ is known, an optimal expected regret of $\widetilde{\mathcal{O}}(\sqrt{(S+1)KT})$ is obtainable [Auer et al., 2002]. However, when $S$ is unknown, Marinov and Zimmert [2021] show that this guarantee is impossible under an adaptive adversary. In this paper, we show that a single algorithm achieves $\widetilde{\mathcal{O}}(\sqrt{(S+1)KT})$ expected regret for every $S$ against an oblivious adversary, resolving an open problem of Auer et al. [2019b]. Our algorithm combines a fixed-share learner initialized with a small learning rate and dyadic-interval subroutines that search for local improvements using randomized learning rates and implicit exploration. Importantly, a non-uniform prior favors following the main learner, keeping the cost of maintaining many subroutines small. When the subroutines accumulate sufficient improvement over the main learner, its learning rate doubles, allowing adaptation to the unknown number of comparator switches $S$.

## Planned Restlessness: Delayed Meta Bandits with Budgets

DevFeed: [Planned Restlessness: Delayed Meta Bandits with Budgets](<https://devfeed.tech/articles/planned-restlessness-delayed-meta-bandits-with-budgets-108742.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.38955>)

Author: Sudipto Guha

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>)

Tags: [approximation](<https://devfeed.tech/tags/approximation.md>), [bandits](<https://devfeed.tech/tags/bandits.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [cs-ds](<https://devfeed.tech/tags/cs-ds.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [explore](<https://devfeed.tech/tags/explore.md>), [future](<https://devfeed.tech/tags/future.md>), [information](<https://devfeed.tech/tags/information.md>), [lag](<https://devfeed.tech/tags/lag.md>), [lower-bound](<https://devfeed.tech/tags/lower-bound.md>), [lp-relaxation](<https://devfeed.tech/tags/lp-relaxation.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [reward](<https://devfeed.tech/tags/reward.md>), [risk](<https://devfeed.tech/tags/risk.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [thompson-sampling](<https://devfeed.tech/tags/thompson-sampling.md>)

### AI overview

The paper develops a planning framework for episodic Bayesian bandits where arm qualities are redrawn from unknown recurring priors, observations may be delayed, and plays face hard caps and budgets. Its LP relaxation values experiments by immediate reward, resource use, and the future decisions enabled by the resulting learning state; dual prices yield per-arm programs and policy certificates. The framework includes hierarchical Thompson sampling, risk-sensitive objectives, purchased observations, and buyout options, while execution accounts for preserving or losing planned learning transitions under constraints. With stationary priors, finite reward alphabets, bounded report lag, and nonbinding budgets, posterior sampling over priors has an O(√N) bound on the cost of prior uncertainty across N episodes (other parameters fixed). In the separate undelayed single-episode setting with each arm capped at T/X plays, the stated LP-relative approximation is 1+(2πX)^−1/2+O(X^−1), with a matching lower bound on the leading term.

### Source excerpt

arXiv:2610.38955v1 Announce Type: new Abstract: We study episodic finite-horizon Bayesian bandits, or meta-bandits, with independent arms. Each episode draws fresh arm qualities from unknown recurring priors. Observations arrive after known delays, and plays obey hard per-episode caps and total budgets. A play earns reward and informs both the current quality and the recurring prior: the learner must explore, use, and generalize. We formulate a common planning optimization over learning trajectories. Its LP relaxation values experiments through current reward, resource consumption and continuation: the value of future decisions from the learning state produced. Dual prices separate the arms into local programs and certify every admissible policy, including hierarchical Thompson sampling. Execution enforces the hard constraints while preserving the promised learning transitions or accounting for their loss. The framework also represents risk-sensitive objectives and prices purchased observations and buyout options. Delayed reports create planned restlessness: an arm's information changes while it is not played, through reports of its own earlier plays arriving on a known calendar. For stationary priors, finite reward alphabets, bounded report lag and nonbinding budgets, posterior sampling over the recurring priors bounds the price of not knowing them by $O(\sqrt{N})$ over $N$ episodes, with other parameters fixed. For one undelayed $T$-step episode with arms capped at $T/X$ plays each, we obtain the LP-relative approximation factor $1+(2\pi X)^{-1/2}+O(X^{-1})$, improving previous factors, with a matching lower bound for the leading term.

## Variable-Length Feedback Coding With Universal Decoding and Bandit Channel Selection

DevFeed: [Variable-Length Feedback Coding With Universal Decoding and Bandit Channel Selection](<https://devfeed.tech/articles/variable-length-feedback-coding-with-universal-decoding-and-bandit-channel-selection-104006.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36818>)

Author: Recep Can Yavas

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Decoding](<https://devfeed.tech/topics/decoding.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [avoid](<https://devfeed.tech/tags/avoid.md>), [bandit](<https://devfeed.tech/tags/bandit.md>), [based](<https://devfeed.tech/tags/based.md>), [bits](<https://devfeed.tech/tags/bits.md>), [but](<https://devfeed.tech/tags/but.md>), [call](<https://devfeed.tech/tags/call.md>), [channel](<https://devfeed.tech/tags/channel.md>), [coding](<https://devfeed.tech/tags/coding.md>), [combines](<https://devfeed.tech/tags/combines.md>), [commit](<https://devfeed.tech/tags/commit.md>), [cs-it](<https://devfeed.tech/tags/cs-it.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [decoding](<https://devfeed.tech/tags/decoding.md>), [develop](<https://devfeed.tech/tags/develop.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [math-it](<https://devfeed.tech/tags/math-it.md>), [selection](<https://devfeed.tech/tags/selection.md>), [variable](<https://devfeed.tech/tags/variable.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The paper studies variable-length feedback coding when both terminals lack the noise variances of K AWGN channels and the receiver must learn channel quality while decoding. It presents two first-order optimal schemes with second-order achievability bounds. VLSF-Bandit uses a Gaussian random codebook, a likelihood-mixture score, and kl-UCB++ on received power. VLF-Bandit sends message packets by Schalkwijk-Kailath transmission, learns variances from normalized outputs, and carries message bits in exploration packets. Its bound has a smaller log N coefficient magnitude than VLSF-Bandit, with a universal-decoding term replacing the latter's baseline and a smaller channel-exploration term. The authors state that it improves on the universal VLF bound for K=1 and on an explore-then-commit bound for K>=2.

### Source excerpt

arXiv:2610.36818v1 Announce Type: cross Abstract: We study variable-length feedback (VLF) coding over $K$ additive white Gaussian noise (AWGN) channels whose noise variances are unknown to both terminals but lie in a known interval. The receiver selects one channel at each time, conveys the selection over noiseless feedback, and must learn the channel qualities while decoding, to avoid overusing suboptimal channels. This setting, which we call the VLF-Bandit problem, combines VLF coding, universal decoding, and bandit channel selection. We develop two joint coding and channel-selection schemes. The VLSF-Bandit scheme is a single-phase variable-length stop-feedback (VLSF) scheme based on a Gaussian random codebook, a score formed from a ratio of likelihood mixtures, and the asymptotically optimal $\text{kl-UCB}^{++}$ bandit algorithm applied to the received power. The VLF-Bandit scheme sends the message in packets by Schalkwijk--Kailath transmission, uses its normalized outputs as noise samples to learn the variances, and lets its exploration packets carry message bits. We prove second-order achievability bounds for both schemes, which are first-order optimal. In the logarithm of the achievable codebook size, the magnitude of the $\log N$ coefficient, with $N$ the average decoding time, is a baseline of one plus a channel-exploration term for VLSF-Bandit. For VLF-Bandit, this magnitude is smaller, with a universal-decoding term replacing the baseline and a smaller channel-exploration term. The VLF-Bandit bound improves on the universal VLF bound of Yavas and Tan (2025) for $K=1$ and on the bound of an explore-then-commit strategy whenever $K\geq2$.

## On the Optimal Sample Complexity of Offline Multi-Armed Bandits with KL Regularization

DevFeed: [On the Optimal Sample Complexity of Offline Multi-Armed Bandits with KL Regularization](<https://devfeed.tech/articles/on-the-optimal-sample-complexity-of-offline-multi-armed-bandits-with-kl-regularization-102953.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.02141>)

Author: Kaixuan Ji, Qiwei Di, Heyang Zhao, Qingyue Zhao, Quanquan Gu

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [bandits](<https://devfeed.tech/tags/bandits.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [kl-regularization](<https://devfeed.tech/tags/kl-regularization.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [math-st](<https://devfeed.tech/tags/math-st.md>), [offline](<https://devfeed.tech/tags/offline.md>), [sample](<https://devfeed.tech/tags/sample.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [stat-th](<https://devfeed.tech/tags/stat-th.md>)

### AI overview

The paper characterizes sample complexity for offline multi-armed bandits evaluated with KL-regularized performance. Its analysis of KL-PCB gives a bound of Õ(ηSAC^{π*}/ε) in the stated large-regularization regime and identifies a lower-bound scale of Õ(SAC^{π*}/ε²) in the stated small-regularization regime. The authors also give two sharper lower bounds that they say match the upper bounds across regularization strengths. Here S is the number of contexts, A the number of arms, C^{π*} the optimal policy's coverage coefficient, ε the target suboptimality, and η the regularization parameter; the tilde notation hides polylogarithmic factors. The abstract describes the result as a nearly complete characterization.

### Source excerpt

arXiv:2605.02141v2 Announce Type: replace-cross Abstract: Kullback-Leibler (KL) regularization is widely used in offline decision-making and offers several benefits, motivating recent work on the sample complexity of offline learning with respect to KL-regularized performance metrics. Nevertheless, the exact sample complexity of KL-regularized offline learning remains largely from fully characterized. In this paper, we study this question in the setting of multi-armed bandits (MABs). We provide a sharp analysis of KL-PCB (Zhao et al., 2026), showing that it achieves a sample complexity of $\tilde{O}(\eta SAC^{\pi^*}/\epsilon)$ under large regularization $\eta = \tilde{O}(\epsilon^{-1})$, and a sample complexity of $\tilde{\Omega}(SAC^{\pi^*}/\epsilon^2)$ under small regularization $\eta = \tilde{\Omega}(\epsilon^{-1})$, where $\eta$ is the regularization parameter, $S$ is the number of contexts, $A$ is the number of arms, $C^{\pi^*}$ policy coverage coefficient at the optimal policy $\pi^*$, $\epsilon$ is the desired sub-optimality, and $\tilde{O}$ and $\tilde{\Omega}$ hide all poly-logarithmic factors. We further provide a pair of sharper sample complexity lower bounds, which matches the upper bounds over the entire range of regularization strengths. Overall, our results provide a nearly complete characterization of offline multi-armed bandits with KL regularization.

## Exploratory creative pre-ranking: Hierarchical empirical-Bayes Thompson sampling for real-time creative optimization

DevFeed: [Exploratory creative pre-ranking: Hierarchical empirical-Bayes Thompson sampling for real-time creative optimization](<https://devfeed.tech/articles/exploratory-creative-pre-ranking-hierarchical-empirical-bayes-thompson-sampling-for-real-time-creative-optimization-103906.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.36020>)

Author: Jiale Han, Ajith Moparthi, Dan Xie, Xiaowu Dai

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>)

Tags: [ads](<https://devfeed.tech/tags/ads.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-generated-content](<https://devfeed.tech/tags/ai-generated-content.md>), [campaign](<https://devfeed.tech/tags/campaign.md>), [creative](<https://devfeed.tech/tags/creative.md>), [cs-gt](<https://devfeed.tech/tags/cs-gt.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [exploitation](<https://devfeed.tech/tags/exploitation.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [hierarchy](<https://devfeed.tech/tags/hierarchy.md>), [latency](<https://devfeed.tech/tags/latency.md>), [live](<https://devfeed.tech/tags/live.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [production](<https://devfeed.tech/tags/production.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [roblox](<https://devfeed.tech/tags/roblox.md>), [thompson-sampling](<https://devfeed.tech/tags/thompson-sampling.md>)

### AI overview

The pre-ranker uses hierarchical empirical-Bayes estimates and a Thompson draw to shortlist ad creatives before per-user ranking. Sparse data for a new creative can draw on results from its audience segment, advertised item, and campaign; borrowing weights are set from historical logs with one parameter per level and no tuning. To meet latency constraints, it predicts at the audience-segment level using per-attribute counters. In a replay of Roblox Sponsored-ads traffic, the shortlist increased predicted play-through rate by 8.14%, versus 2.80% for an equally sized random shortlist, forwarded over 40% fewer creatives than the full pool, and added 0.2% to serving time. The abstract says the method entered Roblox production after a live-traffic A/B test.

### Source excerpt

arXiv:2610.36020v1 Announce Type: new Abstract: Dynamic creative optimization decides which creative of an ad to render for each request. Recent work places that choice before ad ranking, but relies on deterministic point predictors with no calibrated uncertainty. AI-generated content has made creatives nearly free, while feedback per creative remains sparse and selection faces a tight latency budget. We introduce a creative pre-ranker that scores every recalled creative and shortlists each campaign's pool before the per-user ranker runs. Creatives sit in a hierarchy of audience segment, advertised item, campaign and creative, so a cold creative falls back on what its parent levels have earned, and how much it borrows from its parents is set in closed form from historical logs, with one parameter per level and nothing tuned. To stay within the latency budget, the pre-ranker predicts for an audience segment rather than an individual user, from per-attribute counters. It selects by a single Thompson draw per creative, balancing exploration and exploitation so that a new creative still has a chance to be learned. Replaying Roblox Sponsored-ads traffic over campaigns with competing creatives, the shortlist lifts predicted play-through rate by 8.14% against 2.80% for an equally sized random shortlist, forwards more than 40% fewer creatives than the full pool, and adds only 0.2% to serving time. After an A/B test on live traffic, the method now serves in production at Roblox.

[Next page](<https://devfeed.tech/topics/thompson-sampling.md?cursor=WyIyMDI2LTEwLTI4VDA0OjAwOjAwKzAwOjAwIiwgIjI4MDI1NGY0LTBiMzYtNDZlZS1iMmRkLTVlMDc3MjE4OTMzMyJd>)