# Prompt optimization

Published articles for Prompt optimization.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Risk-certified prompt optimization for LLM-as-a-judge guardrails

DevFeed: [Risk-certified prompt optimization for LLM-as-a-judge guardrails](<https://devfeed.tech/articles/risk-certified-prompt-optimization-for-llm-as-a-judge-guardrails-138999.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20156>)

Author: Hussein Jawad, Alexandre Landeau, Guy Fleury Iradukunda, Paul Beaumont, Th\\'eodore Rezel, Sofiane Schaack, Nicolas J-B. Brunel

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [prompt](<https://devfeed.tech/topics/prompt.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [certification](<https://devfeed.tech/tags/certification.md>), [confidence-bounds](<https://devfeed.tech/tags/confidence-bounds.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [disclosure](<https://devfeed.tech/tags/disclosure.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [guardrails](<https://devfeed.tech/tags/guardrails.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-injection](<https://devfeed.tech/tags/prompt-injection.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [risk](<https://devfeed.tech/tags/risk.md>), [safety](<https://devfeed.tech/tags/safety.md>), [system-prompt-leakage](<https://devfeed.tech/tags/system-prompt-leakage.md>)

### AI overview

The framework selects system prompts for LLM-as-a-judge guardrails by maximizing benign pass rate subject to a user-set limit on missed attacks, with finite-sample risk certification. CPS generates a fixed prompt pool and uses simultaneous upper confidence bounds on a held-out set to choose the highest-utility certified prompt. CRISP uses risk feedback, hard-example-guided rewriting, and a reusable-holdout interface to adapt its search toward feasible candidates; it is particularly useful when a fixed pool yields no certifiable high-utility option. Across four datasets on prompt injection, system-prompt leakage, and sensitive-information disclosure, the authors report that certified selection respects the target missed-attack level on held-out tests, while utility rises as the allowed risk increases.

### Source excerpt

arXiv:2611.20156v1 Announce Type: cross Abstract: LLM-as-a-judge guardrails are often controlled primarily through system prompts, yet standard prompt optimization methods target average performance rather than explicit safety constraints. This paper studies system-prompt selection as a constrained optimization problem: maximize benign pass rate while keeping the missed-attack rate below a user-specified level {\alpha} with finite-sample statistical control. The framework combines LLM-based judging, automatic prompt generation, and distribution-free risk certification. This work instantiates it in two procedures: CPS and CRISP. CPS generates a fixed pool of prompts from training data, then certifies candidates on a held-out evaluation set using simultaneous upper confidence bounds and selects the certified prompt with the highest benign utility. CRISP instead integrates risk feedback into an adaptive search, using hard example-guided rewriting and a reusable-holdout interface to steer generation toward the feasible utility frontier. Experiments on four datasets covering prompt injection, system-prompt leakage, and sensitive-information disclosure show that risk-certified selection consistently respects the target missed-attack level on heldout tests, while utility improves as the permitted risk is relaxed. CRISP is especially useful when fixed-pool search fails to produce certifiable high-utility candidates.

## The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search

DevFeed: [The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search](<https://devfeed.tech/articles/the-critic-can-hurt-on-the-safety-of-critics-in-llm-guided-automl-search-137381.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19037>)

Author: Gaoyuan Du, Xiaoyang Liu, Rex Zhou, Dengdu Jiang, Rong Fu, Meng Ding, Defu Cao

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [audit](<https://devfeed.tech/tags/audit.md>), [automl](<https://devfeed.tech/tags/automl.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [budget](<https://devfeed.tech/tags/budget.md>), [capability](<https://devfeed.tech/tags/capability.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [errors](<https://devfeed.tech/tags/errors.md>), [fault-injection](<https://devfeed.tech/tags/fault-injection.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [gate](<https://devfeed.tech/tags/gate.md>), [llm](<https://devfeed.tech/tags/llm.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [safety](<https://devfeed.tech/tags/safety.md>), [search](<https://devfeed.tech/tags/search.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

A naive 8B LLM critic pushed search below the no-critic random-search floor on 7 of 8 tabular datasets, with losses up to 0.245 AUC; the impact varied by dataset, search frontier and budget. Controlled fault injection suggests non-directional mistakes are nearly costless under best-so-far search, while directional mistakes consume finite search opportunities. Stronger critics reduced losses on the evaluated suite, but the study establishes no universal capability threshold. A simple validation gate and a learned detector also reduced realized loss, though neither was guaranteed to preserve the full-budget no-critic baseline. The authors propose auditing critics and gating feedback when capability or verification is insufficient, and announce CRITFAULT, a benchmark for detecting harmful critic feedback in AutoML.

### Source excerpt

arXiv:2611.19037v1 Announce Type: cross Abstract: LLM critics increasingly steer iterative search in agentic AutoML, prompt optimization, and self-refinement--evaluated for whether they help, but rarely for whether they can hurt. We isolate the critic's marginal effect through a four-arm audit and characterize the loop's tolerance through controlled fault injection. We show a naive 8B critic drives search below the no-critic random-search floor on 7 of 8 tabular datasets, losing up to 0.245 AUC in the worst case; the magnitude varies strongly with dataset, frontier, and budget. When: stronger critics attenuate the loss on the evaluated suite, without establishing a universal capability threshold. Why: controlled fault injection shows that non-directional errors are nearly costless under best-so-far search, whereas directional errors consume finite search opportunities; an idealized effective-sample calculation yields a necessary condition, $q(m-1) > \beta$, not a deployment certificate. How: a simple observable validation gate and a learned detector can reduce realized loss, but neither is guaranteed to preserve the full-budget no-critic baseline. We will release CRITFAULT, to our knowledge the first benchmark for detecting harmful critic feedback in AutoML. A critic is not a free addition: audit before trusting, and gate when capability or verification is insufficient.

## Small Language Models for Inference-Capability Routing: Effects of Model Capacity and Prompt Optimization on Downstream Quality and Efficiency

DevFeed: [Small Language Models for Inference-Capability Routing: Effects of Model Capacity and Prompt Optimization on Downstream Quality and Efficiency](<https://devfeed.tech/articles/small-language-models-for-inference-capability-routing-effects-of-model-capacity-and-prompt-optimization-on-downstream-quality-and-efficiency-132231.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16414>)

Author: Sailendra Kolluru, Maheshkumar Duvvarapu

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [prompt](<https://devfeed.tech/topics/prompt.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [capacity](<https://devfeed.tech/tags/capacity.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [quality](<https://devfeed.tech/tags/quality.md>), [routing](<https://devfeed.tech/tags/routing.md>), [small-language-models](<https://devfeed.tech/tags/small-language-models.md>), [token](<https://devfeed.tech/tags/token.md>)

### AI overview

The study tests prompt-driven Qwen2.5 routers (1.5B-14B parameters) that independently select reasoning, retrieval, web search, and tool use, allowing multiple capabilities per query. Across five prompting strategies, added capacity does not consistently improve classification under minimal prompts. Prompt optimization narrows the capacity gap for reasoning and calculator routing, while RAG and web search retain larger benefits from higher capacity. On 127 questions using GPT-OSS-120B, the 14B router's mean BERTScore F1 (0.8408) is close to enabling all capabilities (0.8418); it cuts average downstream tokens by 68.7% and median downstream latency by 80.5%. These savings exclude router overhead, which was measured separately in a different inference environment and increases with router capacity. The abstract cautions that similar classification F1 can mask different error costs: false positives waste computation, while missed capabilities can lower answer quality.

### Source excerpt

arXiv:2611.16414v1 Announce Type: new Abstract: Optional reasoning, retrieval, web search, and tool use can improve large language model responses but incur unnecessary computation when activated indiscriminately. This study investigates whether prompt optimization can compensate for limited router capacity and whether classification gains translate into downstream efficiency. The proposed framework uses prompt-driven small language models to make four independent capability decisions, allowing multiple capabilities to be selected per query. Qwen2.5 routers ranging from 1.5B to 14B parameters are compared across five prompting strategies, followed by an evaluation of capability-specific optimized configurations using a fixed GPT-OSS-120B downstream model on 127 questions. Under minimal prompting, increasing router capacity does not consistently improve classification performance. Prompt optimization narrows the capacity gap for reasoning and calculator routing, whereas RAG and web search retain larger benefits from higher capacity. Similar classification F1 scores do not necessarily imply equivalent downstream behavior: false-positive activations increase unnecessary computation, while missed capabilities can reduce answer quality. The 14B router achieves a mean BERTScore F1 of 0.8408, compared with 0.8418 when all capabilities are enabled, while reducing average downstream token consumption by 68.7\% (8,837 to 2,764 tokens) and median downstream latency by 80.5\% (9.79 to 1.91 seconds). These savings exclude router overhead, which is measured separately in a different inference environment and increases with router capacity. The findings demonstrate that the relationship between router capacity, prompt design, and downstream efficiency is capability-dependent. Consequently, router selection must consider error profiles and routing overhead alongside classification performance.

## Chatter Attack: Resource Consumption Attack for Large Language Models

DevFeed: [Chatter Attack: Resource Consumption Attack for Large Language Models](<https://devfeed.tech/articles/chatter-attack-resource-consumption-attack-for-large-language-models-125349.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11261>)

Author: Yibo Miao, Yichuan Cao, Xiao-Shan Gao, Yinpeng Dong

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [attack](<https://devfeed.tech/tags/attack.md>), [autoregressive](<https://devfeed.tech/tags/autoregressive.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-inference](<https://devfeed.tech/tags/llm-inference.md>), [llms](<https://devfeed.tech/tags/llms.md>), [malicious](<https://devfeed.tech/tags/malicious.md>), [math-oc](<https://devfeed.tech/tags/math-oc.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [practical](<https://devfeed.tech/tags/practical.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [resource](<https://devfeed.tech/tags/resource.md>), [responses](<https://devfeed.tech/tags/responses.md>), [search](<https://devfeed.tech/tags/search.md>)

### AI overview

Chatter Attack aims to increase LLM inference cost by making models generate unusually long responses. Instead of optimizing for long outputs directly, it prompts models toward short doubt-inducing phrases, then uses inference attention loss and global entropy loss to sustain self-doubt during generation. For black-box settings, the authors propose Bayesian Prompt Optimization, which uses a discrete token kernel and prior information to explore the prompt search space. They report outperforming baselines across multiple models and datasets, with response lengths increasing by up to 31.5x. The supplied abstract describes these results but does not provide experimental details.

### Source excerpt

arXiv:2611.11261v1 Announce Type: new Abstract: LLMs are powerful but incur substantial inference cost, which can be exploited by malicious users to induce overly long outputs, increasing latency and operational expense. Prior resource consumption attacks either coerce repetition or suppress the end-of-sequence token by lowering its logit value. However, with autoregressive, sampling-based generation, the search space grows exponentially with length and small deviations or unstable trajectories can derail optimization, limiting attack effectiveness and stability. To address this issue, we introduce Chatter Attack, a novel class of resource consumption attacks that optimize adversarial prompts to induce predefined short target phrases, thereby avoiding search-space explosion while still eliciting long responses. Specifically, Chatter Attack triggers self-doubt using short doubt-inducing phrases as the target, and sustains this state throughout generation via inference attention loss and global entropy loss. To extend to black-box settings, we further propose a novel Bayesian Prompt Optimization (BPO) method, which models the objective function globally by constructing a discrete token kernel, efficiently exploring the entire search space using prior information. Extensive experiments across multiple models and datasets show that our method outperforms baselines, with maximum response length increases to 31.5$\times$, and revealing practical resource-depletion risks for real-world LLM inference services.

## Promptim: an experimental library for prompt optimization

DevFeed: [Promptim: an experimental library for prompt optimization](<https://devfeed.tech/articles/promptim-an-experimental-library-for-prompt-optimization-123895.md>)

Original publisher: [Read original article](<https://www.langchain.com/blog/promptim>)

Author: LangChain Accounts

Published: 2026-11-09T18:30:44Z

Content type: article

Language: en

Sources: [LangChain Blog](<https://devfeed.tech/sources/langchain-blog.md>)

Topics: [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [prompt](<https://devfeed.tech/topics/prompt.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-performance](<https://devfeed.tech/tags/ai-performance.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [automate](<https://devfeed.tech/tags/automate.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [experimental](<https://devfeed.tech/tags/experimental.md>), [faster](<https://devfeed.tech/tags/faster.md>), [library](<https://devfeed.tech/tags/library.md>), [models](<https://devfeed.tech/tags/models.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-engineering](<https://devfeed.tech/tags/prompt-engineering.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [save](<https://devfeed.tech/tags/save.md>), [switch](<https://devfeed.tech/tags/switch.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

Promptim automates optimization of a single prompt using a LangSmith dataset, locally defined evaluators, and an iterative metaprompt loop. It scores an initial prompt on a dev split, uses training examples and their scores to propose revisions, and keeps a revision only when dev metrics improve; train/dev/test splits are optional. Human feedback through LangSmith Annotation Queues is also optional, and the authors recommend human review of the final prompt. Compared with DSPy, Promptim focuses on rewriting one prompt and retaining human oversight, while DSPy targets whole compound AI systems and offers options such as fine-tuning and few-shot prompting. Promptim is described as experimental, and its optimization is not presented as a substitute for human sanity checks.

### Source excerpt

Automate prompt engineering with Promptim's optimization library. Save time, improve AI performance, and switch models faster.

## AgentGrad: Intervention-guided Prompt Optimization for Multi-Agent Systems

DevFeed: [AgentGrad: Intervention-guided Prompt Optimization for Multi-Agent Systems](<https://devfeed.tech/articles/agentgrad-intervention-guided-prompt-optimization-for-multi-agent-systems-124475.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.08572>)

Author: Jaewon Chu, Jinwoo Seo, Jaewon Cho, Jeehye Na, Yunyang Xiong, Youngdae Kim, Hyunwoo J. Kim

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [prompt](<https://devfeed.tech/topics/prompt.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [multi-agent-systems](<https://devfeed.tech/tags/multi-agent-systems.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [semantic](<https://devfeed.tech/tags/semantic.md>)

### AI overview

AgentGrad improves multi-agent prompt optimization by targeting the agent responsible for each failure and grouping feedback by meaning. It injects a hint into one agent at a time; when that agent's modified output resolves the failure, the output supervises extraction of a fine-grained textual gradient. It then clusters semantically similar gradients and abstracts each cluster into a shared corrective pattern, avoiding aggregation of unrelated failures. The abstract reports state-of-the-art results on five multi-agent benchmarks, with 2.5x less wall-clock optimization time and 21.8% lower average optimization cost than the next-best baselines.

### Source excerpt

arXiv:2609.08572v3 Announce Type: replace Abstract: Large language model (LLM)-based multi-agent systems (MAS) achieve strong performance by employing specialized multiple agents, yet their performance depends on the prompt design of each agent. For MAS prompt optimization, textual gradient methods that guide prompt updates using natural-language feedback have emerged as a leading paradigm. In this paper, we identify limitations in two stages of existing textual gradient approaches: gradient extraction and gradient aggregation. In gradient extraction, previous works select a prompt to update without verifying whether modifying it resolves the failure, and derive gradients without agent-level supervision over the corresponding agent's intermediate output. In gradient aggregation, individual gradients are randomly grouped and concatenated, often mixing unrelated failure modes and producing prompts that fail to generalize. To address these limitations, we propose AgentGrad, a prompt optimization framework for multi-agent systems based on sequential intervention and semantic textual gradient abstraction. For each failure, sequential intervention injects a hint into one agent at a time to identify the target agent whose modification resolves the failure. The modified output of the target agent then serves as agent-level supervision for extracting a fine-grained gradient. Semantic textual gradient abstraction clusters semantically similar gradients to prevent mixing unrelated failure modes, and abstracts each cluster into a generalized gradient that captures the shared corrective pattern. Experimental results show that AgentGrad achieves state-of-the-art performance across five MAS benchmarks while reducing wall-clock optimization time by 2.5 times and optimization cost by 21.8% on average compared to the next-best baselines. Our code is available at https://github.com/mlvlab/AgentGrad.

## FinEvolveKG: Controlled and Attributable Prompt Evolution for Financial Knowledge Graph Construction

DevFeed: [FinEvolveKG: Controlled and Attributable Prompt Evolution for Financial Knowledge Graph Construction](<https://devfeed.tech/articles/finevolvekg-controlled-and-attributable-prompt-evolution-for-financial-knowledge-graph-construction-124539.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.08643>)

Author: Abhinav Arun, Jayita Bhattacharyya, Fabrizio Dimino, Bhaskarjit Sarmah, Stefano Pasquali

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [база знаний](<https://devfeed.tech/topics/tag-c51cc4bf76e5.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Finance](<https://devfeed.tech/topics/finance.md>)

Tags: [arxiv](<https://devfeed.tech/tags/arxiv.md>), [audit](<https://devfeed.tech/tags/audit.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [financial](<https://devfeed.tech/tags/financial.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [guidance](<https://devfeed.tech/tags/guidance.md>), [harness](<https://devfeed.tech/tags/harness.md>), [investment](<https://devfeed.tech/tags/investment.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [knowledge-graphs](<https://devfeed.tech/tags/knowledge-graphs.md>), [management](<https://devfeed.tech/tags/management.md>), [mutation](<https://devfeed.tech/tags/mutation.md>), [needs](<https://devfeed.tech/tags/needs.md>), [ood](<https://devfeed.tech/tags/ood.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>)

### AI overview

FinEvolveKG separates reusable general rules from persona-specific policy when constructing financial knowledge graphs from SEC 10-Ks. It mechanically confines GEPA prompt edits to the designated, separately versioned persona block, keeping the approved general block byte-stable, and optimizes for both generic temporal-KG quality and persona-relevant facts, motifs, and connectivity. Across three personas, the complete two-stage workflow (v4) improved 19.4-30.9% over a frozen-prompt baseline and 2.2-6.4% over the tested unconstrained workflow (v5) on three-seed means; it led v5 on both objective components, won all nine paired-seed comparisons, and had the lowest seed variance per persona. It also exceeded v5 across evaluated out-of-distribution shifts without further optimization. SME audits supported the triplets as ground truth and favored v4 in 70.1% of contrast-enriched judgments. The authors caution that these results do not establish the block boundary as the cause of the gains.

### Source excerpt

arXiv:2611.08643v1 Announce Type: cross Abstract: Financial institutions construct knowledge graphs (KGs) from shared SEC 10-Ks to serve risk analysis, portfolio management, and investment research, whose analytical needs differ. Extraction must adapt without fragmenting shared grounding, typing, and temporal rules. Hand-authored persona prompts and fine-tuning add cost. Weight-free prompt optimization offers an alternative, but unconstrained evolution of a monolithic prompt can rewrite shared rules alongside persona guidance, obscuring changes and complicating review. We introduce FinEvolveKG, a KG-construction harness separating reusable general and persona-policy blocks and mechanically confining GEPA, a reflective prompt optimizer, to the designated block. During persona adaptation, the approved general block remains byte-stable and every optimizer edit is localized to a separately versioned persona policy. A dual objective balances generic temporal-KG quality with persona-relevant facts, motifs, and multi-hop connectivity. We evaluate five mutation scopes on ticker-disjoint 10-K splits, with three seeds for the persona-conditioned arms (v3-v5) across three personas. On the three-seed mean, the complete two-stage block-constrained workflow (v4) improves over the frozen prompt baseline (v1) by 19.4-30.9% and over the tested unconstrained full-prompt workflow (v5) by 2.2-6.4%, leading v5 on both objective components and winning all nine paired-seed comparisons with the lowest seed variance in every persona. Without further optimization, v4 also exceeds v5 across all evaluated out-of-distribution (OOD) shifts. An SME audit supports using the generated triplets as ground truth; a blinded two-SME audit favors v4 over v5 in 70.1% of contrast-enriched judgments. These results show that mechanically attributable evolution can coexist with strong empirical performance, without attributing the gains causally to the boundary.

## PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems

DevFeed: [PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems](<https://devfeed.tech/articles/promptops-iterative-prompt-optimization-for-llm-as-a-judge-systems-120163.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07123>)

Author: Sven Kleinkop, Tam\\'as Janusko, Ricardo Bochnia, Gunnar Hempel, Steffen Tomschke, J\\"urgen Anke, Maik Thiele

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [prompt](<https://devfeed.tech/topics/prompt.md>), [Human-AI evaluation](<https://devfeed.tech/topics/human-ai-evaluation.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>)

Tags: [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [framework](<https://devfeed.tech/tags/framework.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [maintenance](<https://devfeed.tech/tags/maintenance.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>)

### AI overview

PromptOps treats maintaining LLM-as-a-Judge systems as an iterative response to judge drift: it separates decisions about when to optimize, which method and feedback records to use, and how to update the prompt. The framework is instantiated with GEPA, SIMBA, Rubric Synthesis, and exemplar selection using simulated annealing or a Tree-structured Parzen Estimator. Experiments report that iterative optimization can recalibrate judges and recover performance after controlled shifts in reference-score distributions. The best method and update timing vary by setting, so neither generalizes universally. The supplied text is an abstract and does not provide further experimental details.

### Source excerpt

arXiv:2611.07123v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems. In deployment, however, their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change. Existing automatic prompt optimization methods are primarily studied as one-shot procedures on static development sets and therefore provide limited guidance for maintaining LLM judges over time. We frame this problem as judge drift and introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge (LLMaaJ) systems. PromptOps separates four operational decisions: when to optimize, which optimization procedure to apply, which feedback records to use, and how to update the judge prompt. We instantiate the framework with five prompt optimization methods: GEPA, SIMBA, Rubric Synthesis, and exemplar selection using either simulated annealing or a Tree-structured Parzen Estimator. Our experiments demonstrate that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the referencescore distribution. However, effective maintenance is inherently contextdependent: neither optimization algorithm choice nor update timing generalizes universally across settings. These findings establish prompt optimization for deployed LLM judges as an inherently iterative maintenance problem and demonstrate that PromptOps provides a practical abstraction for studying and implementing this process.

## IntentGEO: Intent-aware Generative Engine Optimization for Recommendation

DevFeed: [IntentGEO: Intent-aware Generative Engine Optimization for Recommendation](<https://devfeed.tech/articles/intentgeo-intent-aware-generative-engine-optimization-for-recommendation-117443.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04225>)

Author: Runlin Lei, Shiqi Zhang, Xiaokui Xiao, Zhewei Wei

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [recommendation systems](<https://devfeed.tech/topics/recommendation-systems.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [chatbots](<https://devfeed.tech/tags/chatbots.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [doubao](<https://devfeed.tech/tags/doubao.md>), [e-commerce](<https://devfeed.tech/tags/e-commerce.md>), [generative-engine-optimization](<https://devfeed.tech/tags/generative-engine-optimization.md>), [geo](<https://devfeed.tech/tags/geo.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llms](<https://devfeed.tech/tags/llms.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-drift](<https://devfeed.tech/tags/prompt-drift.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [recommendation](<https://devfeed.tech/tags/recommendation.md>), [recommendations](<https://devfeed.tech/tags/recommendations.md>), [ugc](<https://devfeed.tech/tags/ugc.md>), [user-generated-content-ugc](<https://devfeed.tech/tags/user-generated-content-ugc.md>), [user-intent](<https://devfeed.tech/tags/user-intent.md>), [visibility](<https://devfeed.tech/tags/visibility.md>)

### AI overview

IntentGEO evaluates how item visibility changes across user needs and decision stages, finding that existing GEO methods' gains vary by intent and that learning-based prompt optimization can underperform its handwritten seed. The authors trace this to prompt drift from recommendation instructions toward generic promotion. Their AMO method keeps recommendation instructions in a fixed core and learns appended strategies from citation feedback. Across four domains and ten intent states, AMO raises average citation-rank gain 35% over the strongest prior learning-based baseline and leads on nine states. The abstract also reports top gains in an agentic recommendation pipeline and, after separate adaptation, on Doubao. These results suggest evaluating GEO by user intent and anchoring prompt adaptation to recommendation goals.

### Source excerpt

arXiv:2611.04225v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly shape recommendations across chatbots, e-commerce, and user-generated content (UGC) platforms, creating demand for Generative Engine Optimization (GEO) to improve item visibility in generated responses. Existing GEO studies are mainly content-centric, optimizing visibility under general-purpose queries. However, a shared item description usually serves users with different needs and decision goals, resulting in its visibility gains being dependent on the request. To investigate how changing intent affects visibility, we formulate IntentGEO to evaluate shared rewrites across need specificity and decision progression. We revisit existing GEO methods and find that their visibility gains vary across user intents. More critically, learning-based prompt optimization can underperform its handwritten seed on average. Tracing these updates reveals prompt drift from explicit recommendation instructions toward generic promotional tactics. To address this drift, we propose AMO (Anchored Meta-Optimization), which preserves recommendation instructions in a fixed core while learning appended strategies from citation feedback. Across four domains and ten intent states, AMO improves average citation-rank gain by 35% over the strongest prior learning-based baseline and leads on nine states. Ablations show that learned additions improve on the fixed core alone and that AMO outperforms full-prompt optimization initialized from that core. It also achieves the highest gains in an agentic recommendation pipeline and, after separate adaptation, on Doubao, a commercially deployed, closed-source online assistant. These findings highlight user intent as a key dimension of GEO evaluation and support anchored adaptation across recommendation needs and decision stages.

## JailbreakOPT: Iterative Jailbreak Prompt Optimization with Macro-Action

DevFeed: [JailbreakOPT: Iterative Jailbreak Prompt Optimization with Macro-Action](<https://devfeed.tech/articles/jailbreakopt-iterative-jailbreak-prompt-optimization-with-macro-action-113996.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.11425>)

Author: Ge Shi, Jun Yin, Donglin Xie, Fangyi Liu, Yucan Li, Menglin Liu

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [thompson sampling](<https://devfeed.tech/topics/thompson-sampling.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [attack](<https://devfeed.tech/tags/attack.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [failure](<https://devfeed.tech/tags/failure.md>), [jailbreak](<https://devfeed.tech/tags/jailbreak.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [policy](<https://devfeed.tech/tags/policy.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [reuse](<https://devfeed.tech/tags/reuse.md>), [safety](<https://devfeed.tech/tags/safety.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [search](<https://devfeed.tech/tags/search.md>), [vulnerabilities](<https://devfeed.tech/tags/vulnerabilities.md>)

### AI overview

JailbreakOPT seeks to improve jailbreak attack success while reducing the response burden of iterative prompt search. It composes reusable prompt transformations into macro-actions to generate standalone attack prompts, then uses contextual Thompson sampling and outcomes from earlier search paths to guide action selection across episodes. The abstract reports higher attack success rates across multiple LLMs and benchmarks than single-action and iterative baselines, but gives no numerical results or further experimental details. The authors suggest the approach can help probe model vulnerabilities and identify failure patterns that may inform defenses.

### Source excerpt

arXiv:2606.11425v2 Announce Type: replace Abstract: Jailbreak attacks expose persistent safety weaknesses in large language models (LLMs), but existing stateless single-turn methods face a trade-off: hand-crafted prompts are expressive but static, while iterative prompt optimization can adapt but often relies on low-level mutations that require many target-model responses. We propose JailbreakOPT, a method that maintains a library of reusable prompt transformations (actions) and composes them into macro-actions for within-episode search, generating standalone attack prompts. To reuse experience across episodes, JailbreakOPT learns an action-selection policy using contextual Thompson sampling and outcomes from prior search paths. Experiments across multiple LLMs and benchmarks show that JailbreakOPT improves attack success rate (ASR) over single-action and iterative baselines. More broadly, exploration and exploitation over human-designed attack strategies can help probe residual vulnerabilities and identify failure patterns that may inform defenses aimed at broader classes of attacks.

## StateGate: A Behavioral-Fidelity Prompt Optimization Method for Long-Chain LLM Agents

DevFeed: [StateGate: A Behavioral-Fidelity Prompt Optimization Method for Long-Chain LLM Agents](<https://devfeed.tech/articles/stategate-a-behavioral-fidelity-prompt-optimization-method-for-long-chain-llm-agents-114053.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01585>)

Author: Shicheng Huang, Cai Fu, Jianqiang Lyu, Yifan Zhang, Xianmao Ji

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [prompt](<https://devfeed.tech/topics/prompt.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [security](<https://devfeed.tech/tags/security.md>), [size](<https://devfeed.tech/tags/size.md>), [vulnerability-detection](<https://devfeed.tech/tags/vulnerability-detection.md>), [vulnerability-discovery](<https://devfeed.tech/tags/vulnerability-discovery.md>)

### AI overview

StateGate aims to preserve long-chain agents' adherence to constraints by turning semantic requirements into symbolic gates enforced by an external deterministic runtime. Its proposed design uses four prompt-compression strategies--deduplication, generalization, externalization, and text-to-gating--and separates execution from judgment through a state file, validator, and orchestrator. The paper says it formally proves symbolic-constraint compliance is independent of context size and reports 100% execution closure and accurate vulnerability detection in a security vulnerability-discovery task. The supplied text is an abstract and does not provide evaluation details or evidence beyond that reported claim.

### Source excerpt

arXiv:2611.01585v1 Announce Type: new Abstract: Currently, long-chain LLM agents commonly suffer from behavioral degradation in complex tasks, where prompts are written in full but steps are not executed in full. To address this issue, existing approaches face three major challenges. First, constraints are strongly bound to their textual carriers; once natural-language constraint sentences are removed, their binding force vanishes. Second, the execution process lacks independent verification, as the evidence for judgment is often supplied by the executing party itself, leading to the failure of self-referential validation. Third, existing remediation schemes are difficult to generalize, and hard-coded mechanisms cannot be transferred across tasks. In this paper, we propose StateGate, a novel prompt-optimization scheme that transforms semantic constraints into symbolic gates enforced by an external deterministic runtime. By decoupling the content of a constraint from its carrier of efficacy, StateGate achieves behavioral fidelity in long-chain tasks without losing the binding force of the prompt. Specifically, we first propose four independent compression strategies (deduplication, generalization, externalization, and text-to-gating) together with their corresponding decision chains, enabling principled prompt symbolization. Second, we design a three-component architecture consisting of a state file, a validator, and an orchestrator, which thoroughly separates judgment authority from execution authority. Third, we formally prove that the compliance of symbolic constraints is independent of context size. Finally, we validate the framework on a vulnerability-discovery task in the security domain; the results show that StateGate achieves 100% execution closure and accurate vulnerability detection.

## Compile as Hypothesis: A Headroom Typology for Prompt Optimization on Real Reasoning Models

DevFeed: [Compile as Hypothesis: A Headroom Typology for Prompt Optimization on Real Reasoning Models](<https://devfeed.tech/articles/compile-as-hypothesis-a-headroom-typology-for-prompt-optimization-on-real-reasoning-models-115130.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01135>)

Author: Zhengshuai Yang, Changyun Li

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [headroom](<https://devfeed.tech/tags/headroom.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [reasoning-models](<https://devfeed.tech/tags/reasoning-models.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

Across four prompt optimizer families and eight tasks, structured search produced no statistically significant accuracy gains on strong reasoning models, though a discrete structural genome prevented seed-collapse seen in unconstrained search in 3 of 5 runs. The study's headroom typology, based on 190-problem evaluations, finds that prompt optimization can fix protocol and formatting errors but cannot overcome intrinsic knowledge gaps or physical token truncation. On one gpt-6 task, GEPA gained 0.14 while structured search was flat; the authors attribute about 0.035 to judge-rubric reconstruction and 0.105 to instruction refinement. A pre-registered selection gate matched oracle choices in 5 of 7 groups and rescued seed-collapse arms with zero regret. The excerpt also reports silent backend mixing during commercial API outages and automated per-call audits to check evaluation fidelity.

### Source excerpt

arXiv:2611.01135v1 Announce Type: new Abstract: Automated prompt compilers search for or rewrite prompts under the implicit assumption that optimization consistently improves downstream task performance. We evaluate this assumption on frontier reasoning models through rigorous paired, same-day measurements. Across four optimizer families (genome search, profile-guided mutation, GEPA, and ESPO) and eight tasks, structured search yields no statistically significant accuracy gains on strong reasoners. However, a discrete structural genome provides essential architectural stability, preventing catastrophic bimodal seed-collapse (where unconstrained search collapses toward the judge floor in 3/5 runs). Through 190-problem evaluations on knowledge-intensive benchmarks, we establish a three-dimensional headroom typology: prompt optimization cannot compensate for intrinsic knowledge deficits or physical token truncation, but successfully resolves protocol and formatting errors. On gpt-6, while free-text reflective search (GEPA) yields a +.14 gain on a task where structured search remains flat, qualitative ablation reveals that ~.035 stems from judge-rubric reconstruction while ~.105 reflects genuine instruction refinement. Our pre-registered AutoAPC-Select deployment gate matches the oracle policy in 5/7 groups and rescues seed-collapse arms with zero regret. Finally, we expose silent backend-mixing in commercial APIs during outage windows and provide automated per-call audit verification tools to ensure empirical evaluation fidelity.

## Prompt-agnostic Language Models

DevFeed: [Prompt-agnostic Language Models](<https://devfeed.tech/articles/prompt-agnostic-language-models-107942.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.37683>)

Author: Frederic Sadrieh

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [prompt](<https://devfeed.tech/topics/prompt.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [construction](<https://devfeed.tech/tags/construction.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [inference](<https://devfeed.tech/tags/inference.md>), [interaction](<https://devfeed.tech/tags/interaction.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [operators](<https://devfeed.tech/tags/operators.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [robustness](<https://devfeed.tech/tags/robustness.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [training](<https://devfeed.tech/tags/training.md>), [variance](<https://devfeed.tech/tags/variance.md>)

### AI overview

Prompt-agnostic Language Models addresses prompt sensitivity during instruction tuning rather than relying on manual inference-time prompt optimization. The authors report that gradients from paraphrased prompts conflict in 57% of trainable parameters, while naive averaging discards about one fifth of gradient magnitude. Their methods, Prompt-TIES and Prompt-DARE, apply gradient-merging techniques to reduce this interference. Across four models from two families at two scales, prompt phrasing and example interactions account for 54% of per-example variability on average, exceeding 70% on ANLI and WiC. Gradient merging nearly halves per-example variance on fully fine-tuned models with little cost to overall accuracy; the other evaluated train-time approaches improve raw performance but do not reliably reduce that variance. Evidence is from the supplied abstract.

### Source excerpt

arXiv:2610.37683v1 Announce Type: cross Abstract: Despite their strong performance, large language models remain highly sensitive to prompt formulation. This issue is typically addressed at inference-time through manual prompt optimization, which introduces overhead and lacks scalability. Conversely, we focus on train-time approaches that build prompt robustness into the model during instruction tuning. A common strategy is to mix multiple paraphrases of each prompt into training. We demonstrate that across trainable parameters, the per-paraphrasing gradients interfere, with 57% exhibiting sign conflicts and the naive gradient average discarding approximately a fifth of the magnitude. Therefore, we develop two gradient-merging methods, Prompt-TIES and Prompt-DARE, that address this interference by leveraging interference-resolution operators from model and gradient merging and applying them to prompt gradients. In this study, we compare the efficacy of our methods to increase model robustness to train-time robustness methods spanning all three established classes: data construction, consistency regularization, and contrastive alignment. The evaluation is conducted across four models from two families at two scales. We observe the interaction between prompt phrasing and the specific examples accounts for 54% of per-example variability on average, and exceeds 70% on tasks such as ANLI and WiC. The gradient-merging methods directly improve this local robustness and nearly halve per-example variance on fully fine-tuned models at little cost to overall accuracy. The other train-time methods enhance raw performance, but fail to improve per-example variance reliably. Code: https://github.com/FSadrieh/prompt-agnostic-llms. Model checkpoints: https://huggingface.co/prompt-agnostic-language-models.

## Zero-Shot Multimodal Event Spotting in Soccer Broadcasts

DevFeed: [Zero-Shot Multimodal Event Spotting in Soccer Broadcasts](<https://devfeed.tech/articles/zero-shot-multimodal-event-spotting-in-soccer-broadcasts-104106.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.37583>)

Author: Sadia Kamal, Mahmudul Hasan, Tony Braskich

Published: 2026-10-28T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Sports](<https://devfeed.tech/topics/sports.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>), [Temporal data](<https://devfeed.tech/topics/temporal-data.md>), [Image](<https://devfeed.tech/topics/image.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-mm](<https://devfeed.tech/tags/cs-mm.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generation](<https://devfeed.tech/tags/generation.md>), [indexing](<https://devfeed.tech/tags/indexing.md>), [inference](<https://devfeed.tech/tags/inference.md>), [input](<https://devfeed.tech/tags/input.md>), [language](<https://devfeed.tech/tags/language.md>), [model](<https://devfeed.tech/tags/model.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [selection](<https://devfeed.tech/tags/selection.md>), [sports](<https://devfeed.tech/tags/sports.md>), [storyboard](<https://devfeed.tech/tags/storyboard.md>), [training](<https://devfeed.tech/tags/training.md>), [transcript](<https://devfeed.tech/tags/transcript.md>), [video](<https://devfeed.tech/tags/video.md>), [zero-shot](<https://devfeed.tech/tags/zero-shot.md>)

### AI overview

The framework spots soccer events in full-length broadcasts without task-specific training by processing overlapping clips as compact visual storyboards paired with aligned commentary transcripts. On three manually annotated matches with 130 events, its best configuration achieved 72.0% F1@5s and outperformed SoccerNet-trained CALF and NetVLAD++ in a cross-dataset evaluation. The storyboard representation reduced inference cost by 72% compared with using individual frames. The work also describes an evidence-grounded excitement-scoring stage that ranks detected events and supplies natural-language explanations for highlight selection.

### Source excerpt

arXiv:2610.37583v1 Announce Type: cross Abstract: Automatic soccer event spotting supports highlight generation, video indexing, and sports analytics, but existing methods depend on large annotated datasets and soccer-specific training. We present a zero-shot framework that detects, classifies, and temporally localizes events in full-length soccer broadcasts without task-specific parameter training. Each match is segmented into overlapping clips represented by a compact visual storyboard and the aligned commentary transcript, which a general-purpose multimodal model processes jointly. On three manually annotated matches containing 130 events, the best configuration achieves 72.0% F1@5s and outperforms SoccerNet-trained CALF and NetVLAD++ in a cross-dataset setting, while the storyboard representation reduces inference cost by 72% compared with individual frames. We further analyze the effects of model family, input representation, and prompt optimization, and introduce an evidence-grounded excitement-scoring stage that ranks detected events with natural-language explanations for highlight selection.

## Fix evaluation bias to unlock better models on Amazon Bedrock

DevFeed: [Fix evaluation bias to unlock better models on Amazon Bedrock](<https://devfeed.tech/articles/fix-evaluation-bias-to-unlock-better-models-on-amazon-bedrock-102789.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/fix-evaluation-bias-to-unlock-better-models-on-amazon-bedrock/>)

Author: Khaled Thabit

Published: 2026-10-27T16:53:02Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [vlm](<https://devfeed.tech/topics/vlm.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-evaluation](<https://devfeed.tech/tags/ai-evaluation.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [bias](<https://devfeed.tech/tags/bias.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [image](<https://devfeed.tech/tags/image.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [property](<https://devfeed.tech/tags/property.md>), [quality](<https://devfeed.tech/tags/quality.md>), [vision](<https://devfeed.tech/tags/vision.md>)

### AI overview

Property Finder found that 96.8% of its evaluation labels came from its incumbent model, biasing comparisons against alternatives. It replaced those labels with independent expert annotations for 356 UAE images, used multi-label classification for overlapping room types, and tuned prompts through error analysis and iteration. On an evaluation set that included the 256 images used for tuning, Llama 4 Maverick 17B Instruct scored Micro F1 0.80 versus 0.56 for the incumbent; the article cautions these results are optimistic. The selected model matched or exceeded the incumbent on all 11 attributes, with equal photo-classification performance. The reported 86% cost reduction is an estimate against the previous setup's cost analysis, and the reported 40-60% latency reduction comes from Property Finder's production monitoring.

### Source excerpt

Property Finder found that 96.8% of their vision-AI evaluation labels were inherited from the incumbent model, hiding its true accuracy. See how they built unbiased ground truth, applied evolutionary prompt optimization, and selected a more accurate, cost-effective model on Amazon Bedrock, cutting cost by 86% across 11 image-quality attributes.

## GFlowPO: Generative Flow Network as a Language Model Prompt Optimizer

DevFeed: [GFlowPO: Generative Flow Network as a Language Model Prompt Optimizer](<https://devfeed.tech/articles/gflowpo-generative-flow-network-as-a-language-model-prompt-optimizer-96814.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2602.03358>)

Author: Junmo Cho, Suhan Kim, Sangjune An, Minsu Kim, Dong Bok Lee, Heejun Lee, Sung Ju Hwang, Hae Beom Lee

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [prompt](<https://devfeed.tech/topics/prompt.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [few-shot](<https://devfeed.tech/topics/few-shot.md>), [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [buffer](<https://devfeed.tech/tags/buffer.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [framework](<https://devfeed.tech/tags/framework.md>), [inference](<https://devfeed.tech/tags/inference.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [lightweight](<https://devfeed.tech/tags/lightweight.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [memory](<https://devfeed.tech/tags/memory.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [probabilistic](<https://devfeed.tech/tags/probabilistic.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [question](<https://devfeed.tech/tags/question.md>), [rl](<https://devfeed.tech/tags/rl.md>)

### AI overview

GFlowPO treats prompt search as posterior inference over latent prompts, regularized by a reference language model guided by a meta-prompt. It fine-tunes a lightweight prompt model with an off-policy Generative Flow Network objective and reuses past evaluations through replay-based training to improve sample-efficient exploration. A second, training-free component, Dynamic Memory Update, refreshes the meta-prompt using diverse replay-buffer prompts and top-performing prompts from a priority queue, progressively steering search toward high-reward regions. The abstract reports that GFlowPO outperforms recent discrete prompt optimization baselines on few-shot text classification, instruction induction, and question answering; it provides no numerical results or further evaluation details.

### Source excerpt

arXiv:2602.03358v2 Announce Type: replace-cross Abstract: Finding effective prompts for language models (LMs) is critical yet notoriously difficult: the prompt space is combinatorially large, rewards are sparse due to expensive target-LM evaluation. Yet, existing RL-based prompt optimizers often rely on on-policy updates and a meta-prompt sampled from a fixed distribution, leading to poor sample efficiency. We propose GFlowPO, a probabilistic prompt optimization framework that casts prompt search as a posterior inference problem over latent prompts regularized by a meta-prompted reference-LM prior. In the first step, we fine-tune a lightweight prompt-LM with an off-policy Generative Flow Network (GFlowNet) objective, using a replay-based training policy that reuses past prompt evaluations to enable sample-efficient exploration. In the second step, we introduce Dynamic Memory Update (DMU), a training-free mechanism that updates the meta-prompt by injecting both (i) diverse prompts from a replay buffer and (ii) top-performing prompts from a small priority queue, thereby progressively concentrating the search process on high-reward regions. Across few-shot text classification, instruction induction benchmarks, and question answering tasks, GFlowPO consistently outperforms recent discrete prompt optimization baselines.

## Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles

DevFeed: [Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles](<https://devfeed.tech/articles/revealing-what-benchmark-scores-hide-jointly-measuring-the-breadth-and-reliability-of-llm-capabilities-using-accessibility-profiles-93394.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.24957>)

Author: Kazem Faghih, Yize Cheng, Shoumik Saha, Mobina Pournemat, Armin Gerami, Soheil Feizi

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [accessibility](<https://devfeed.tech/tags/accessibility.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [llm](<https://devfeed.tech/tags/llm.md>), [measuring](<https://devfeed.tech/tags/measuring.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [scaling](<https://devfeed.tech/tags/scaling.md>)

### AI overview

The paper proposes measuring an inquiry's accessibility: the probability that a model answers correctly across meaning-preserving formulations, rather than judging it from one wording. Accessibility profiles distinguish breadth--how many inquiries a model can sometimes answer--from reliability--how consistently it succeeds. Across six benchmarks and more than twenty models, the authors report that for about half of model pairs with nearly equal mean accessibility, rankings depend on the requirement: at least one success across five formulations (Access@5) or success on all five (Reliable@5). Some pairs differ by up to 10 points in opposite directions. The abstract also reports that scaling, reasoning, and prompt optimization can broaden the inquiries a model can answer while making success less repeatable.

### Source excerpt

arXiv:2610.24957v1 Announce Type: cross Abstract: Language model benchmarks assign each question a single outcome from one fixed wording, even though the same underlying request can be expressed in many equivalent ways. We argue that the quantity of interest should therefore not be correctness on one wording, but the accessibility of an inquiry, which we define as the probability that a model succeeds across meaning-preserving formulations. The distribution of accessibility across inquiries reveals a structure that accuracy and existing best-, worst-, or average-case prompt scores collapse: how broadly a model's capabilities extend across inquiries and how dependably those capabilities can be elicited. Two models can have the same average performance while one succeeds across more inquiries unreliably and the other succeeds across fewer inquiries but more consistently, leading to opposite preferences depending on whether the task allows multiple retries until one success or when repeated consistent success is required. Across six benchmarks and more than twenty models, this tradeoff is common: for about half of the model pairs with nearly equal mean accessibility, which model looks better depends on whether one success across five formulations is enough or all five must succeed, quantities we call Access@5 and Reliable@5, respectively. For some pairs the two measures diverge significantly, by up to 10 points in opposite directions. Accessibility also reveals that scaling, reasoning, and prompt optimization can redistribute capability across inquiries in ways aggregate accuracy obscures, sometimes broadening the set of inquiries a model can succeed on while making success on them less repeatable. Accessibility turns formulation sensitivity from a nuisance into a measurable property of capability, distinguishing what a model can sometimes succeed on from what users can actually depend on.

## How uniopen customized Amazon Nova to their retail moderation policies for production deployment

DevFeed: [How uniopen customized Amazon Nova to their retail moderation policies for production deployment](<https://devfeed.tech/articles/how-uniopen-customized-amazon-nova-to-their-retail-moderation-policies-for-production-deployment-63581.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/how-uniopen-customized-amazon-nova-to-their-retail-moderation-policies-for-production-deployment/>)

Author: Felix Chin, Jia-You Lin

Published: 2026-10-01T15:33:01Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Nova](<https://devfeed.tech/topics/nova.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [Deployment & Scaling](<https://devfeed.tech/topics/deployment-scaling.md>), [human review](<https://devfeed.tech/topics/human-review.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [amazon-bedrock-guardrails](<https://devfeed.tech/tags/amazon-bedrock-guardrails.md>), [amazon-nova](<https://devfeed.tech/tags/amazon-nova.md>), [amazon-sagemaker](<https://devfeed.tech/tags/amazon-sagemaker.md>), [amazon-sagemaker-ai](<https://devfeed.tech/tags/amazon-sagemaker-ai.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fine](<https://devfeed.tech/tags/fine.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [release](<https://devfeed.tech/tags/release.md>)

### AI overview

uniopen adapted Amazon Nova 2 Lite to its retail content moderation policy through supervised fine-tuning with LoRA in Amazon SageMaker AI, followed by prompt optimization. On a held-out set of 737 conversation windows, the final configuration exceeded production targets for both behavior and subject-type classification. Human-verified corrections, evaluation gates, and review for ambiguous cases are part of the deployment workflow.

### Source excerpt

See how uniopen, a retail platform from Taiwan's Uni-President Enterprises Group, adapted Amazon Nova 2 Lite to its content-moderation policies using supervised fine-tuning in Amazon SageMaker AI and prompt optimization. Business-relevant evaluation and release gates kept quality in check.

## Optimizing agent system prompts with Amazon Bedrock AgentCore

DevFeed: [Optimizing agent system prompts with Amazon Bedrock AgentCore](<https://devfeed.tech/articles/optimizing-agent-system-prompts-with-amazon-bedrock-agentcore-31522.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/optimizing-agent-system-prompts-with-amazon-bedrock-agentcore/>)

Author: Han Ding

Published: 2026-09-16T15:47:39Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [A/B Testing](<https://devfeed.tech/topics/a-b-testing.md>)

Tags: [a-b-testing](<https://devfeed.tech/tags/a-b-testing.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-agentcore](<https://devfeed.tech/tags/amazon-bedrock-agentcore.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [system-prompts](<https://devfeed.tech/tags/system-prompts.md>), [traces](<https://devfeed.tech/tags/traces.md>)

### AI overview

This technical companion explains how Amazon Bedrock AgentCore's system prompt optimizer uses production agent traces and reward signals to propose revised system prompts. It describes the reflector engine, recommendation explanations, offline batch evaluation, online A/B testing, and promotion workflow, and reports benchmark results for Single Agent Reflector and the experimental open source Sub-Agent Reflector.

### Source excerpt

AgentCore optimization turns production traces into proposed configuration changes, then validates them before promotion. This technical companion to the launch post explains how the system prompt optimizer's reflector engine works and shares benchmark results for the Single Agent and Sub-Agent Reflectors.

## PROOF-Gen: From Optimized Data to Better Distillation

DevFeed: [PROOF-Gen: From Optimized Data to Better Distillation](<https://devfeed.tech/articles/proof-gen-from-optimized-data-to-better-distillation-6731.md>)

Original publisher: [Read original article](<https://machinelearning.apple.com/research/proof-gen-optimized-distillation>)

Published: 2026-08-26T00:00:00Z

Content type: article

Language: en

Sources: [Apple Machine Learning Research](<https://devfeed.tech/sources/apple-machine-learning-research.md>)

Topics: [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [gemma](<https://devfeed.tech/topics/gemma.md>), [Programming](<https://devfeed.tech/topics/programming.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gemma](<https://devfeed.tech/tags/gemma.md>), [generate](<https://devfeed.tech/tags/generate.md>), [models](<https://devfeed.tech/tags/models.md>), [on-device](<https://devfeed.tech/tags/on-device.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>)

### AI overview

PROOF-Gen improves tool-calling model distillation by using per-scenario prompt optimization to recover successful trajectories from failed teacher attempts. The method strips corrective guidance before training, producing clean demonstrations and improving benchmark, deployed-pipeline, and on-device model performance.

### Source excerpt

Supervised fine-tuning on teacher-generated trajectories is the standard first stage for distilling tool-calling capabilities into deployable models. Post-training pipelines that drive shipped tool-calling agents re-run this stage on a daily or weekly cadence, paying the frontier-teacher cost each cycle, yet the mechanism is generate-and-filter (keep the teacher's passing trajectories, discard the rest) and each cycle leaves behind the same hard scenarios because failures supply no signal. On τ 2-bench, 57% of teacher trials fail, two-thirds of them near-misses (most tool calls correct, undone...

## Promptimus: Improving already good LLM prompts with zero manual engineering

DevFeed: [Promptimus: Improving already good LLM prompts with zero manual engineering](<https://devfeed.tech/articles/promptimus-improving-already-good-llm-prompts-with-zero-manual-engineering-7605.md>)

Original publisher: [Read original article](<https://www.amazon.science/blog/promptimus-improving-already-good-llm-prompts-with-zero-manual-engineering>)

Author: Zhengyuan Shen; Yunfei Bai; Sullam Jeoung; Shuai Wang

Published: 2026-05-14T13:47:45Z

Content type: article

Language: en

Sources: [Amazon Science homepage](<https://devfeed.tech/sources/amazon-science-homepage.md>)

Topics: [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [multimodal-ai](<https://devfeed.tech/topics/multimodal-ai.md>), [Code](<https://devfeed.tech/topics/code.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [automated-prompt-engineering](<https://devfeed.tech/tags/automated-prompt-engineering.md>), [code-generation](<https://devfeed.tech/tags/code-generation.md>), [conversational-ai](<https://devfeed.tech/tags/conversational-ai.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [llm](<https://devfeed.tech/tags/llm.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prompt-engineering](<https://devfeed.tech/tags/prompt-engineering.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>)

### AI overview

Promptimus is an automated prompt-engineering method for improving already strong prompts without manual engineering. It uses task data, user-defined performance metrics, failure analysis, debugging agents, sanitization, and targeted edit mode to refine prompts while preserving working business logic. The method is model agnostic and supports textual and multimodal LLM tasks, including classification, extraction, generation, summarization, code generation, and tool use.

### Source excerpt

By focusing on specific failure points and suggesting targeted solutions, a new automated prompt-engineering framework improves prompt performance without compromising existing functionality.

## AI-generated test cases from user stories: An experimental research study

DevFeed: [AI-generated test cases from user stories: An experimental research study](<https://devfeed.tech/articles/ai-generated-test-cases-from-user-stories-an-experimental-research-study-57177.md>)

Original publisher: [Read original article](<https://www.thoughtworks.com/insights/blog/generative-ai/AI-generated-test-cases-from-user-stories-an-experimental-research-study>)

Author: Yang Zhao, Vy Nguyen, Thuc Van Hoang

Published: 2025-07-30T00:00:00Z

Content type: article

Language: en

Sources: [Thoughtworks Insights](<https://devfeed.tech/sources/thoughtworks-insights.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [test-coverage](<https://devfeed.tech/topics/test-coverage.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [User Stories](<https://devfeed.tech/topics/user-stories.md>), [GitHub Copilot](<https://devfeed.tech/topics/github-copilot.md>), [ChatGPT](<https://devfeed.tech/topics/chatgpt.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-tool](<https://devfeed.tech/tags/ai-tool.md>), [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [github-copilot](<https://devfeed.tech/tags/github-copilot.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>), [research](<https://devfeed.tech/tags/research.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

This experimental research study examines whether AI tools can help QA professionals generate accurate and comprehensive test cases from user stories. It compares manually created cases with outputs from GitHub Copilot and ChatGPT, evaluating creation time, coverage, quality, consistency, and the effects of iterative prompt refinement.

### Source excerpt

AI-generated test cases from user stories: An experimental research study

## AI Engineering Primer

DevFeed: [AI Engineering Primer](<https://devfeed.tech/articles/ai-engineering-primer-33448.md>)

Original publisher: [Read original article](<https://timkellogg.me/blog/2024/12/19/ai-primer>)

Published: 2024-12-19T00:00:00Z

Content type: tutorial

Language: en

Sources: [Tim Kellogg](<https://devfeed.tech/sources/tim-kellogg.md>)

Topics: [AI Engineering](<https://devfeed.tech/topics/ai-engineering.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LangChain](<https://devfeed.tech/topics/langchain.md>), [DSPy](<https://devfeed.tech/topics/dspy.md>)

Tags: [advice](<https://devfeed.tech/tags/advice.md>), [ai-engineer](<https://devfeed.tech/tags/ai-engineer.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [dspy](<https://devfeed.tech/tags/dspy.md>), [langchain](<https://devfeed.tech/tags/langchain.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llms](<https://devfeed.tech/tags/llms.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>)

### AI overview

A primer on getting up to speed with AI engineering. It discusses the field's rapid evolution and advises readers to be cautious with LangChain tutorials, prompt influencers, and blindly applying traditional machine-learning advice. It mentions alternatives including hand-written code, LlamaIndex, DSPy, Gradio, and vendor SDKs.

### Source excerpt

How do you get up to speed with AI engineering? Unfortunately, I don't know of any good consolidated resources, so I'm going to attempt to make one here. My first attempt at this focused more on what an AI engineer is and made only a feeble attempt at providing resources to get started. Let's go!