# PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems

DevFeed: [PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems](<https://devfeed.tech/articles/promptops-iterative-prompt-optimization-for-llm-as-a-judge-systems-120163.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07123>)

Author: Sven Kleinkop, Tam\\'as Janusko, Ricardo Bochnia, Gunnar Hempel, Steffen Tomschke, J\\"urgen Anke, Maik Thiele

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [prompt](<https://devfeed.tech/topics/prompt.md>), [Human-AI evaluation](<https://devfeed.tech/topics/human-ai-evaluation.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>)

Tags: [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [framework](<https://devfeed.tech/tags/framework.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [maintenance](<https://devfeed.tech/tags/maintenance.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [performance](<https://devfeed.tech/tags/performance.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [prompt-optimization](<https://devfeed.tech/tags/prompt-optimization.md>)

## AI overview

PromptOps treats maintaining LLM-as-a-Judge systems as an iterative response to judge drift: it separates decisions about when to optimize, which method and feedback records to use, and how to update the prompt. The framework is instantiated with GEPA, SIMBA, Rubric Synthesis, and exemplar selection using simulated annealing or a Tree-structured Parzen Estimator. Experiments report that iterative optimization can recalibrate judges and recover performance after controlled shifts in reference-score distributions. The best method and update timing vary by setting, so neither generalizes universally. The supplied text is an abstract and does not provide further experimental details.

## Source excerpt

arXiv:2611.07123v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems. In deployment, however, their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change. Existing automatic prompt optimization methods are primarily studied as one-shot procedures on static development sets and therefore provide limited guidance for maintaining LLM judges over time. We frame this problem as judge drift and introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge (LLMaaJ) systems. PromptOps separates four operational decisions: when to optimize, which optimization procedure to apply, which feedback records to use, and how to update the judge prompt. We instantiate the framework with five prompt optimization methods: GEPA, SIMBA, Rubric Synthesis, and exemplar selection using either simulated annealing or a Tree-structured Parzen Estimator. Our experiments demonstrate that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the referencescore distribution. However, effective maintenance is inherently contextdependent: neither optimization algorithm choice nor update timing generalizes universally across settings. These findings establish prompt optimization for deployed LLM judges as an inherently iterative maintenance problem and demonstrate that PromptOps provides a practical abstraction for studying and implementing this process.