# responses Published articles for responses. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## HaloGuard 2.0: An Open-Weights Constitutional Output Guard for Multilingual Response Safety DevFeed: [HaloGuard 2.0: An Open-Weights Constitutional Output Guard for Multilingual Response Safety]() Original publisher: [Read original article]() Author: Ashmiya Lenin, Navaneeth Sangameswaran, Preetham S Published: 2026-11-03T05:00:00Z Content type: release Language: en Sources: [arXiv: Cryptography and Security]() Topics: [LLM security](), [Streaming](), [Open Source Models & Datasets]() Tags: [benchmarks](), [concurrent](), [costs](), [cs-cl](), [cs-cr](), [false-positives](), [input](), [jailbreak](), [languages](), [open-weights](), [operational](), [red-teaming](), [responses](), [safety](), [stop](), [stream](), [token](), [training]() ### AI overview HaloGuard 2.0 is an open-weight streaming output guard that scores each response token in one forward pass, allowing a deployment to stop generation when harmful content is confirmed. Its first training setup learned to judge prompts instead of responses, correctly handling only 13.9% of matched safe/unsafe pairs; a corrective phase on 106,043 counterfactual responses addressed this, and the released model averages the base and corrective models. On six English benchmarks, the 0.8B model reaches 83.6 macro F1, up from 72.8, with 38% fewer false positives; it is reported to match leading full-response guards with 5-10 times fewer parameters and score 2.6 points above Qwen3Guard-Stream-4B. The multilingual 0.8B variant scores 83.1 F1 across 17 languages, 5.0 points above the strongest baseline. At 64 concurrent streams on one H100, reported cost is 0.37 ms per token per stream. The excerpt also says both guards undergo weekly red-teaming. ### Source excerpt arXiv:2611.02134v1 Announce Type: cross Abstract: An input guard decides whether a request reaches the model, but never sees what the model says. Harm that emerges during generation, because a jailbreak passed the input layer or a benign request drifted into operational detail, is visible only on the output side and only as tokens arrive. We present HaloGuard 2.0, an open-weight streaming output guard that assigns a safe/unsafe probability to every response token in a single forward pass, so a deployment can stop the stream once harm is confirmed. The guard reuses the constitution of HaloGuard 1.0, our input guard, to organise a 478,532-response training corpus, and supervises each token from the point where the response becomes harmful. We find that a guard trained on conventionally shaped data learns a shortcut specific to streaming: it judges the prompt instead of the response, and handles only 13.9% of matched safe/unsafe response pairs correctly. We correct this with a second phase on 106,043 counterfactual responses built around the base guard's own failure modes, and release the equal-weight average of the base and corrective models. On six English response-safety benchmarks, HaloGuard 2.0-0.8B reaches a macro F1 of 83.6 (83.3 on held-out halves of the decision-rule search), up from 72.8, with 38% fewer false positives. It matches the strongest full-response guards with 5-10x fewer parameters and is 2.6 points above the strongest streaming guard, Qwen3Guard-Stream-4B. A multilingual variant, HaloGuard 2.0-Poly-0.8B, trained on PolyGuardMix, reaches 83.1 F1 on the response labels of PolyGuardPrompts across 17 languages, 5.0 points above the strongest baseline. On one H100 the guard costs 0.37 ms per token per stream when moderating 64 concurrent responses. Both guards are hardened continuously through the weekly red-teaming protocol of HaloGuard 1.0, extended here to the output layer. ## Prevented is Not Protected: What Attack Success Rate Hides about RAG DevFeed: [Prevented is Not Protected: What Attack Success Rate Hides about RAG]() Original publisher: [Read original article]() Author: Chahana Dahal, Zuobin Xiong Published: 2026-11-02T05:00:00Z Content type: article Language: en Sources: [arXiv: Cryptography and Security]() Topics: [Retrieval Augmented Generation (RAG)](), [Security]() Tags: [asr](), [assessment](), [attack](), [composition](), [cs-cl](), [cs-cr](), [cs-ir](), [defense](), [evaluation](), [experiments](), [llm](), [protocol](), [rag](), [rag-poisoning](), [rankings](), [responses]() ### AI overview Attack success rate can hide failures introduced by RAG poisoning defenses, so the paper proposes Aligned RAG Evaluation, which compares matched clean and poisoned queries with and without defense. It combines matched-condition analysis, LLM-judge endorsement, and retrieval-dependence controls to distinguish prevented from defense-induced failures, target mentions from endorsed answers, and single-passage influence from evidence composition. Experiments on HotpotQA and Natural Questions found that defenses could prevent some attacks yet also produce unexpected target-positive answers, and that evaluation criteria could change defense rankings. ### Source excerpt arXiv:2610.40742v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) is vulnerable to knowledge poisoning, where adversaries manipulate retrieved evidence to influence model outputs. Existing defenses are primarily evaluated using attack success rate (ASR), which obscures defense-induced failures and the mechanisms behind apparent attack prevention. To address this issue, this work introduces \textbf{Aligned RAG Evaluation}, a matched-query protocol that jointly examines clean and poisoned responses, with and without defense. By combining matched conditions analysis, LLM-judge endorsement, and retrieval dependence controls, the proposed framework distinguishes prevented from introduced failures, target mentions from LLM endorsement, and single-passage influence from evidence composition. Experiments on HotpotQA and Natural Questions reveal that existing defenses can introduce unexpected target-positive answers, although they can prevent some attacks, while evaluation criteria can also change defense rankings. Our findings expose critical blind spots in ASR-based evaluation and motivate a more comprehensive assessment of RAG poisoning defenses. The code and the evaluation artifact are available at https://github.com/chxhana/aligned-rag-eval. ## Build an AI Code Reviewer with OpenRouter DevFeed: [Build an AI Code Reviewer with OpenRouter]() Original publisher: [Read original article]() Author: OpenRouter Published: 2026-11-01T00:00:00Z Content type: tutorial Language: en Sources: [OpenRouter]() Topics: [ai code review](), [Code review](), [CI/CD](), [Test-driven development]() Tags: [ai](), [ai-code](), [ai-code-review](), [api](), [apis](), [bug](), [build](), [code](), [costs](), [files](), [git](), [guide](), [linux](), [openrouter](), [patches](), [python](), [responses](), [review](), [sandbox](), [uploads]() ### AI overview The guide builds a local Python code reviewer that uses OpenRouter's Responses API to upload changed files, context files, and a Git diff to a hosted Linux sandbox. The model can run checks to confirm suspected bugs and propose V4A patches; findings are marked confirmed with command output or unconfirmed when they cannot be reproduced. Patches are applied to a temporary copy first, and reach the working tree only with `--apply`. The script requires Python 3.11+, Git, and an OpenRouter API key; the Files, Shell, and Apply Patch tools require the global endpoint. In three-run tests on a five-bug change, Claude Haiku 5.5 averaged $0.0153 per review, while Claude Sonnet 5.5 averaged $0.0447 and fixed all five bugs in every run. The sandbox adds cost over diff-only review, and the guide reports model misses and broken patches, so it recommends reading patches before applying them. Changed code is sent to OpenRouter and the provider; uploads are deleted when the script finishes. The proposed GitHub Actions workflow was not tested in GitHub Actions. ### Source excerpt Code reviewers that only read a Git diff have to guess whether a change breaks anything, and their fixes come back as prose you apply by hand. This guide builds a Python reviewer on the OpenRouter Responses API that uploads your changed files into a Linux sandbox, runs checks to confirm each bug, and returns validated patches. A review takes one to four minutes and costs $0.015 to $0.045. ## CVE-2026-62979 - jack: upstream error responses expose backend and peer credentials DevFeed: [CVE-2026-62979 - jack: upstream error responses expose backend and peer credentials]() Original publisher: [Read original article]() Author: CVEFEED.io; github.com Published: 2026-10-30T21:17:15Z Content type: news Language: en Sources: [Latest High/Critical Vulnerabilitiy Feed]() Topics: [Secrets Management](), [P2P](), [Error Handling]() Tags: [2026](), [backend](), [credentials](), [cve](), [cve-2026-62979](), [error](), [peer](), [responses](), [upstream]() ### AI overview Before jack 0.1.0, raw FetchError objects were serialized into HTTP error responses with outbound request headers still exposed in an enumerable `extras` field. A client with the shared peer API key can trigger errors that reveal Radarr/Sonarr `X-Api-Key` values; depending on deployment, responses may also expose Cloudflare Access secrets or another peer's credentials. The issue affects `GET /peer/items/:id`, `GET /servers/health`, and `GET /torznab/download/:id`; `/peer/search` and `/items` catch these errors and are not affected. The disclosed backend keys can grant full administrative control of the corresponding media server. Log and span redaction does not cover this response path. The issue is fixed in version 0.1.0. ### Source excerpt CVE ID :CVE-2026-62979 Published : Oct. 30, 2026, 9:17 p.m. | 37 minutes ago Description :jack lets users share media libraries through their Radarr/Sonarr stack, peer-to-peer. Prior to 0.1.0, `middleware/handle-error.ts` serializes raw `FetchError` objects into HTTP responses. `lib/servers/base.ts` attaches outbound request headers on upstream non-success responses and schema-validation failures; the public `extras` field in `lib/errors/FetchError.ts` remains enumerable and is emitted by JSON serialization. These headers include the internal Radarr/Sonarr `X-Api-Key` and custom connector headers such as `CF-Access-Client-Secret`. `lib/redact.ts` redacts logs and spans but not this response path. Any client with the shared peer API key, plus Cloudflare Access credentials if required by the deployment, can force an error without a race or user interaction. `GET /peer/items/:id` and `GET /servers/health` can expose the internal media-server key, while `GET /torznab/download/:id` can expose another peer's key and Cloudflare Access secrets. The /peer/search and /items routes catch these errors and are not affected. Disclosed internal keys grant full administrative control of the corresponding Radarr/Sonarr instance, crossing the intended boundary between friends' shared peer access and private backend credentials. This issue is fixed in version 0.1.0. Severity: 8.4 | HIGH Visit the link for more details, such as CVSS details, affected products, timeline, and more... ## Red-Teaming Large Language Models in Low-Resource African Languages: A Case Study of Yoruba and Nigerian Pidgin DevFeed: [Red-Teaming Large Language Models in Low-Resource African Languages: A Case Study of Yoruba and Nigerian Pidgin]() Original publisher: [Read original article]() Author: Paula Chikaodinaka Anyanwu, Valdini Douglace Lemofouet, Blessing Ngozi Uzor, Ifeoluwa Dorcas Awotunde, Danielle Blanche Kapsa, Tassallah Abdullahi, Clemencia Siro, Idris Abdulmumin, Seid Muhie Yimam, Published: 2026-10-30T04:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [LLM evaluation / benchmarking](), [LLM security](), [Large Language Model](), [Localization (l10n)]() Tags: [adversarial](), [africa](), [alignment](), [asr](), [attack](), [cs-ai](), [cs-cl](), [english](), [evaluations](), [gemini](), [large-language-models](), [llms](), [native](), [red-teaming](), [responses](), [safety](), [understand]() ### AI overview The evaluation of six LLMs on 400 adversarial prompts found that English safety alignment often did not carry over to Yoruba and Nigerian Pidgin. Low Yoruba comprehension could make models appear to reject fewer prompts without showing greater safety; when models engaged, harmful responses remained common. Attack success rose for most models in the Nigerian languages, with GPT-4o Mini showing the largest reported cross-language discrepancy in Yoruba (+0.23). Gemini 2.0 Flash was the most consistent across languages, while Aya 101 had high English attack success (0.60) and low Yoruba comprehension (0.28), despite explicit African-language training. Cultural stereotypes and discrimination were especially difficult to detect in Nigerian Pidgin (ASR 0.44). The findings support language-specific safety evaluation; the supplied text does not describe the study's full methods or limitations. ### Source excerpt arXiv:2610.39590v1 Announce Type: cross Abstract: As large language models (LLMs) become more widespread in Africa, their safety in low-resource languages remains largely untested since most red-teaming work focuses on English. This study evaluates six state-of-the-art LLMs using 400 explicit adversarial prompts in Yoruba and Nigerian Pidgin, categorised into four harm categories. The responses were scored by native-speaker annotators and reliability was assessed using Cohen's $\kappa$. While models that rejected up to 69% of English prompts rejected as few as 0% in Yoruba, this was not because they were safer, but because they failed to understand the prompts. Attack success rates increased for most models in the Nigerian languages, with GPT-4o Mini demonstrating the most significant cross-lingual safety discrepancy (+0.23 in Yoruba). When models did engage with adversarial prompts, many of the responses were harmful, with rates of harmful content reaching 0.63 in English and 0.50 in Nigerian Pidgin. The most difficult harm category to detect was cultural stereotypes and discrimination (ASR = 0.44 in Nigerian Pidgin). Comprehension in Yoruba was low across most models, and this directly suppressed apparent attack success without indicating actual safety. Gemini 2.0 Flash was the most consistent model across languages. Despite its explicit training in African languages, Aya 101 had the highest attack success rate in English (0.60) and one of the lowest Yoruba comprehension rates (0.28). These results demonstrate that English-centric safety alignment does not transfer to Nigerian languages, highlighting the urgent need for language-specific safety evaluations. ## The Indian Financial Automated Advice Bias (IFAAB) Dataset : Auditing Socioeconomic and Demographic Disparities in LLM-Generated Financial Advice in India DevFeed: [The Indian Financial Automated Advice Bias (IFAAB) Dataset : Auditing Socioeconomic and Demographic Disparities in LLM-Generated Financial Advice in India]() Original publisher: [Read original article]() Author: Kriti Mahajan Published: 2026-10-30T04:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [Large Language Model](), [Finance](), [audit](), [Data analysis]() Tags: [advice](), [auditing](), [benchmark](), [bias](), [consumer-protection](), [cs-ai](), [cs-cy](), [dataset](), [evaluation](), [fairness](), [financial](), [gender](), [gpt](), [indian](), [investment](), [large-language-models](), [llm](), [residence](), [responses](), [saving](), [semantic](), [similarity]() ### AI overview The IFAAB benchmark audits 31,734 GPT-4o financial-advice responses for synthetic Indian personas while holding requests fixed and varying social and demographic attributes. In the gender-only audit, male-labelled profiles received 60.1% higher initial and 46.8% higher post-advice total resources than comparable female-labelled profiles; adding district information reduced the aggregate post-advice gap to 7.9%. Standalone caste comparisons found lower post-advice resources for OBC, SC, and ST profiles than for General-category profiles. Paired gendered responses had average TF-IDF cosine similarity of 0.44. Qualitative analysis found women more often directed toward cautious saving, community support, and financial literacy, while men more often received advice about investment, entrepreneurship, and asset accumulation. The abstract presents IFAAB as a reusable fairness and suitability audit benchmark; these findings describe the reported audits, not necessarily all financial-advice systems. ### Source excerpt arXiv:2610.38239v1 Announce Type: cross Abstract: Large language models are increasingly used for financial guidance, yet little is known about whether their advice changes across social identities in Global South contexts. We introduce the Indian Financial Automated Advice Bias (IFAAB) benchmark, a counterfactual audit dataset of 31,734 GPT-4o-generated financial-advice responses for synthetic Indian personas varying by gender, caste, education, employment, residence, age, and district. Holding the underlying request fixed, we evaluate disparities in LLM-implied wealth, income, savings, projected gains, textual similarity, advice topics, and qualitative bias judgments. In the gender-only audit, male-labelled profiles receive 60.1\% higher initial total resources and 46.8\% higher post-advice total resources than comparable female-labelled profiles; in the caste-augmented audit, the post-advice total-resource gap remains 49.4\%. Adding district information reduces the aggregate post-advice total-resource gender gap to 7.9\%, revealing strong geographic conditioning. Standalone caste comparisons also show systematic stratification: relative to General-category profiles, OBC, SC, and ST profiles receive significantly lower post-advice total resources, respectively. Textual audits show low semantic similarity between paired gendered responses, with an average TF--IDF cosine similarity score of 0.44. Qualitative analysis shows that women are more often steered toward cautious saving, community support, and basic financial literacy, while men receive more growth-oriented advice involving investment, entrepreneurship, and asset accumulation. IFAAB provides a reusable benchmark for evaluating fairness, suitability, and consumer-protection risks in LLM-mediated financial advice, and highlights the need for locally grounded, intersectional evaluation of AI systems in high-stakes economic domains. ## PromptSense: Sentiment-Aware Mental Health Chatbot with Dynamic Prompting DevFeed: [PromptSense: Sentiment-Aware Mental Health Chatbot with Dynamic Prompting]() Original publisher: [Read original article]() Author: Sugi Lee, Jianlong Zhou, Jing sUN Published: 2026-10-30T04:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [AI Chat](), [Prompt Engineering]() Tags: [analysis](), [api](), [chatbot](), [cs-ai](), [depression](), [empathy](), [experimental](), [framework](), [inference](), [insights](), [large-language-models-llms](), [mental-health](), [model-inference](), [perspective](), [prompt-engineering](), [prompting](), [relevance](), [responses](), [risk](), [safety](), [strategies]() ### AI overview PromptSense combines sentiment and depression analysis with rule-based filtering before model inference, then dynamically builds prompts from emotional and risk-related signals to guide chatbot responses. The paper evaluates four model strategies and two prompt-engineered variants on empathy, helpfulness, clarity, relevance, safety, and toxicity using the Perspective API. The abstract reports that prompt engineering outperformed DSFT and RLAIF by 1.8% and 1.1%, respectively; it does not provide further methodological or result details. ### Source excerpt arXiv:2610.38044v1 Announce Type: new Abstract: Current Large Language Models (LLMs)-based mental health chatbots often lack sufficient empathy, contextual awareness, and safety mechanisms, potentially leading to ineffective or harmful interactions. This paper proposes a novel framework that integrates sentiment and depression analysis with a rule-based filtering mechanism prior to model inference. By dynamically constructing prompts based on emotional and risk-related insights, the framework aims to significantly improve the contextual relevance, empathy, and risk-aware of model-generated responses. We evaluated four model strategies and two prompt engineered variants across qualitative metrics such as empathy, helpfulness, clarity, relevance, safety, as well as toxicity assessments using the Perspective API. Experimental results indicate that the prompt engineering approaches outperforms DSFT and RLAIF by 1.8\% and 1.1\% respectively. ## scTREAT: Latent Diffusion for Single-Cell Drug-Response Prediction DevFeed: [scTREAT: Latent Diffusion for Single-Cell Drug-Response Prediction]() Original publisher: [Read original article]() Author: Zhaokang Liang, Shuyang Zhuang, Xiaoran Jiao, Weian Mao, Hao Chen Published: 2026-10-28T04:00:00Z Content type: article Language: en Sources: [arXiv: Machine Learning]() Topics: [diffusion-transformers]() Tags: [attention](), [background](), [channel](), [condition](), [conditional](), [cs-lg](), [diffusion](), [drugs](), [embedding](), [expression](), [framework](), [guidance](), [identity](), [inference](), [metrics](), [model](), [modeling](), [prediction](), [predictions](), [q-bio-qm](), [responses](), [results]() ### AI overview scTREAT predicts drug-induced single-cell transcriptional responses by separating baseline cell state from drug identity and dose guidance in a shared latent expression space. Its Guided Decomposable Attention injects a baseline representation and dose-conditioned molecular embedding; at inference, three conditional predictions are combined using separate state and drug coefficients, with dose modulating drug-guidance strength. Evaluated on condition-level pseudobulk profiles from the Tahoe-100M atlas, it had the best value on 10 of 12 metrics for unseen cell line-drug combinations and 11 of 12 for unseen drugs. For unseen drugs, reported relative gains over the second-best model reached 36.11% on DEG logFC-Spearman and 34.21% on logFC-Pearson. The evidence provided is an abstract and does not detail dataset splits, comparator identities, or broader validation. ### Source excerpt arXiv:2510.11726v2 Announce Type: replace-cross Abstract: Generating drug-induced transcriptional responses requires a model to account for cellular background together with molecular identity and dose. We present scTREAT, the first diffusion framework to factorize state and drug guidance for single-cell drug-response prediction. A shared latent expression space connects baseline and response, while Guided Decomposable Attention (GD-Attn) injects a baseline representation and a dose-conditioned molecular embedding. Independent channel dropout exposes four conditioning states during training; inference combines three conditional predictions with separately specified state and drug coefficients, and dose modulates drug-guidance strength. On condition-level pseudobulk profiles derived from the Tahoe-100M single-cell atlas, scTREAT achieves the best value on 10 of 12 metrics for unseen cell line-drug combinations (UC) and 11 of 12 for unseen drugs (UD). Ablations examine condition injection, molecular representation, and guidance design. On UD, the relative gains over the second-best model reach 36.11% and 34.21% for DEG logFC-Spearman and logFC-Pearson. These results demonstrate the effectiveness of combining shared latent expression modeling with explicit state and drug conditioning for response prediction. ## API-TAIL: Detecting Model Substitution in LLM APIs through Temperature-Aligned Likelihood Tails DevFeed: [API-TAIL: Detecting Model Substitution in LLM APIs through Temperature-Aligned Likelihood Tails]() Original publisher: [Read original article]() Author: Haocheng Mei, Yuchuan Tian, Ye Yuan, Chao Xu, Allen Chen, Kai Han, Yunhe Wang Published: 2026-10-28T04:00:00Z Content type: article Language: en Sources: [arXiv: Machine Learning]() Topics: [audit](), [Large Language Model](), [Temperature settings in AI](), [API Monetization]() Tags: [audit](), [batch](), [budget](), [cs-cl](), [cs-lg](), [decoding](), [detection](), [distribution](), [llm-apis](), [rank](), [responses](), [routing](), [sampling](), [system-prompts](), [temperature]() ### AI overview API-TAIL is designed to detect when an LLM API routes some requests to a cheaper substitute while billing for the advertised model. It accounts for honest providers' differing decoding and serving choices by treating them as a composite null. The audit aligns one effective temperature to center standardized surprisal scores across the batch, then flags the endpoint when too many responses remain in the likelihood tails under a threshold calibrated across legitimate configurations. The abstract reports that tests on commercial APIs controlled false alarms across diverse honest providers and detected low-rate substitution with a small query budget. Under the same composite calibration, it substantially outperformed rank- and MMD-based audits. These findings are reported in the supplied abstract. ### Source excerpt arXiv:2610.36801v1 Announce Type: new Abstract: Commercial language model APIs return only text, allowing a provider to cut costs by quietly routing a fraction of user requests to a cheaper substitute model while billing for the advertised one. Detecting such partial substitution is challenging because honest providers also alter the output distribution through undocumented decoding and serving choices, such as temperature scaling, sampling truncation, repetition and presence penalties, and default system prompts, forcing an audit to test against a composite null of legitimate configurations rather than a single reference. Our key insight is that these legitimate variations primarily shift response typicality across the entire batch, whereas partial substitution injects individual responses that remain anomalous under the claimed model. Building on this, we introduce API-TAIL, a typicality audit that first absorbs batch-level shifts by aligning a single effective temperature to center standardized surprisal scores across the audited batch, and then flags the endpoint if the number of remaining tail responses exceeds a threshold calibrated across legitimate configurations. Evaluated on real-world commercial APIs, API-TAIL reliably controls false alarms across diverse honest providers while detecting even low-rate partial substitution with a small query budget, substantially outperforming existing rank- and MMD-based audits that lose detection power under the same composite calibration. ## PrivacyEvalClaw: A Tool-Augmented Deliberative Agent for Auditing Privacy Leakage in LLM Jailbreak Responses DevFeed: [PrivacyEvalClaw: A Tool-Augmented Deliberative Agent for Auditing Privacy Leakage in LLM Jailbreak Responses]() Original publisher: [Read original article]() Author: Krishna Kanth Nakka, Yujun Wang, Shijie Zhou, Tu Nguyen, Gabriele Gelardi, Xuebing Zhou Published: 2026-10-28T04:00:00Z Content type: article Language: en Sources: [arXiv: Cryptography and Security]() Topics: [Security & Privacy](), [Large Language Model](), [llm-reasoning]() Tags: [agent](), [auditing](), [cs-cr](), [evaluation](), [experiments](), [information](), [jailbreak](), [language-models](), [large-language-models](), [large-language-models-llms](), [llm](), [llm-as-judge](), [llm-jailbreak](), [llms](), [models](), [modules](), [personal](), [personal-information](), [privacy](), [responses](), [tool]() ### AI overview PrivacyEvalClaw addresses disagreement among privacy evaluators, especially when jailbreak responses reveal personal information indirectly or ambiguously. This training-free agent treats evaluators and analysis modules as tools, gathers evidence adaptively, investigates uncertain disclosures, and reconciles conflicting signals before judging leakage. In pairwise LLM-as-Judge evaluations of complete trajectories, it achieved favorable win rates over competing baselines on outcome, process, and overall quality, and produced reasoning traces intended to improve transparency and auditability. ### Source excerpt arXiv:2610.36711v1 Announce Type: new Abstract: Reliable evaluation is a critical bottleneck in privacy red-teaming of large language models (LLMs): determining whether a jailbreak response actually reveals personal information can be difficult when disclosures are indirect, contextual, partial, or obfuscated. Existing evaluation approaches increasingly rely on guard models and LLM-as-Judges, yet these evaluators may interpret the same response differently. To better characterize this evaluation setting, we conduct a systematic study of privacy evaluators and find substantial disagreement, particularly in ambiguous cases that require additional evidence or targeted analysis to resolve. Motivated by these findings, we introduce \textsc{PrivacyEvalClaw}, a training-free deliberative agent for evaluating privacy leakage in LLM jailbreak responses. Rather than relying on isolated single-pass judgments, PrivacyEvalClaw treats evaluators and analysis modules as tools, adaptively gathers evidence, investigates uncertain disclosures, and reconciles conflicting signals before producing a final judgment. Our primary experiments use pairwise LLM-as-Judge evaluation of complete trajectories, showing that PrivacyEvalClaw achieves favorable win rates against competing baselines across outcome, process, and overall quality, while producing explicit reasoning traces that make the evaluation process more transparent and auditable. ## Spring AI 2.1 Message Parts: Reasoning That Survives the Round Trip DevFeed: [Spring AI 2.1 Message Parts: Reasoning That Survives the Round Trip]() Original publisher: [Read original article]() Author: dimitarproynov Published: 2026-10-28T00:00:00Z Content type: article Language: en Sources: [Spring]() Topics: [Spring AI](), [AI Context Management](), [llm-reasoning](), [function calling](), [Claude](), [Streaming](), [AI Chat](), [Chat completions endpoints](), [Ollama]() Tags: [ai](), [anthropic](), [api](), [batch](), [bedrock](), [boot](), [cloud](), [deepseek](), [event-driven](), [gemini](), [genai](), [google](), [java](), [message](), [microservices](), [ollama](), [openai](), [order](), [parts](), [reactive](), [reasoning](), [reasoning-models](), [responses](), [serverless](), [spring](), [spring-ai](), [text](), [thinking](), [thought](), [web-applications]() ### AI overview Spring AI 2.1 replaces messages' text-plus-side-lists model with ordered message parts, preserving reasoning, tool calls, media and provider payloads across requests and streaming. The key rules are to keep parts in order, replay each provider's payload unchanged, and persist parts rather than text. This matters because providers impose different requirements: Anthropic and Gemini can reject missing or altered thinking signatures; OpenAI Responses requires prior reasoning items and tool exchanges for replay; DeepSeek and Mistral require reasoning in certain tool loops. Cross-provider histories retain text and tool calls but do not forward another provider's signed reasoning. Spring AI Session stores complete parts and tool-loop rounds as JSON, while the older JDBC chat-memory repository stores only text and drops tool calls and results. Earlier histories lack reasoning parts, and custom storage must retain part attributes for Mistral and unsigned Bedrock reasoning replay. ### Source excerpt Spring AI: Message Parts Doc | Upgrade Notes | OpenAI Responses Doc | Session Doc Reasoning models do more than think before they answer. They hand part of that thinking back to you and expect to get it back on the next request. Claude signs its thinking blocks. Gemini attaches thought signatures to its function calls. OpenAI's Responses API returns encrypted reasoning items. If you drop, reorder or edit them, the result ranges from an HTTP 400 to a model that quietly loses its earlier reasoning and starts over. Until now, a Spring AI message was a piece of text with side lists of tool calls and media. There was no place for that data, let alone its order. Spring AI 2.1 changes the data model itself: every message now holds its content as an ordered list of message parts. A part is one piece of a response, such as a run of text, a block of reasoning or a tool call. Providers call these pieces content blocks (Anthropic, Bedrock), parts (Gemini), items (OpenAI Responses) or chunks (Mistral), and Spring AI maps them all to message parts. With the 2.1.0-RC1 release candidate, the major chat models produce and consume parts: OpenAI Responses, Anthropic, Google GenAI, OpenAI Chat Completions, Ollama, Bedrock Converse, DeepSeek and Mistral AI. The release also adds Session memory, which persists them. This post covers the new abstraction, why the Responses API needed it, and what each chat model gains. It comes down to three rules: keep the order, replay what you were given verbatim, and persist the parts, not the text. Why a String Was Not Enough Here is what Claude, with thinking enabled, returns in a single tool-calling round: thinking "The user wants the weather in Paris. I should call get_weather..." signature: "EqQBCgIYAhIM..." text "Let me check that for you." tool_use get_weather {"location": "Paris"} When you send the tool result, Anthropic wants this turn back exactly as it was: same blocks, same order, signature included. Before 2.1, AssistantMessage could not d ## Python Typing Survey 2026: AI Tools and the Shifting Checker Landscape DevFeed: [Python Typing Survey 2026: AI Tools and the Shifting Checker Landscape]() Original publisher: [Read original article]() Author: Chris Wiltz Published: 2026-10-27T15:59:24Z Content type: news Language: en Sources: [Engineering at Meta]() Topics: [Python](), [AI Chat](), [Tool](), [Developer experience](), [Pydantic]() Tags: [2](), [2026](), [ai](), [ai-tools](), [but](), [culture](), [developers](), [devinfra](), [first](), [half](), [hint](), [information](), [library](), [little](), [meta](), [one](), [python](), [responses](), [selection](), [survey](), [tools](), [type]() ### AI overview Across 1,794 survey responses, frequent Python type hint use stayed near 86% over three years, while checker choice shifted and developers increasingly mixed tools. Missing or inadequate third party library types remained the leading difficulty (65.1%); 49.9% selected at least two checkers, and 42.7% of multi-checker users reported inconsistencies. Speed was the leading reason for checker choice. AI use in typing grew: 42.5% said AI tools wrote type hints, and some respondents described tools running checkers and fixing errors. But 29.8% were unsure whether types improve AI-assisted coding, and the article says evidence about helpful interactions is still needed. The authors point to better library annotations, clearer editor feedback and documentation, low-configuration first checker runs, and simpler callable syntax as practical areas for improvement. Survey findings reflect respondents, most of whom had at least five years of Python experience. ### Source excerpt The 2026 Python Typing Survey received 1,794 responses. Type hint use changed little across three years, but checker selection changed sharply. Among respondents who reported which checkers they use, half selected more than one, and missing library type information was the most common difficulty. For the third consecutive year, we asked Python developers how they [...] Read More... The post Python Typing Survey 2026: AI Tools and the Shifting Checker Landscape appeared first on Engineering at Meta. ## pnpm 11.28.9 DevFeed: [pnpm 11.28.9]() Original publisher: [Read original article]() Author: Zoltan Kochan Published: 2026-10-27T00:00:00Z Content type: release Language: en Sources: [pnpm]() Topics: [pnpm](), [Release notes]() Tags: [about](), [added](), [all](), [also](), [changes](), [container](), [content](), [cpu](), [crash](), [install](), [pnpm](), [pool](), [quota](), [registry](), [release](), [repeat](), [responses](), [this](), [with](), [worker]() ### AI overview pnpm 11.28.9 fixes a crash from registry responses with excessive content encodings via undici 7.30.1, and sizes install workers and default workspace concurrency to a container's CPU quota to avoid excessive workers and memory use. Repeat installs now respond to more configuration changes, including rebuilding node_modules or verifying the lockfile where needed; filtered installs also fail when required workspace state is inconsistent. Other changes cover registry routing and lockfile resolution, proxy and TLS settings for pnpm downloads, corrected audit exit and flag behavior, safer store pruning, and install, publish, deploy, and output fixes. ### Source excerpt This release fixes a crash on registry responses with too many content encodings (GHSA-xjj6-f33q-929c) and sizes the install worker pool by the container's CPU quota. A repeat pnpm install now acts on more setting changes. ## Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation DevFeed: [Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation]() Original publisher: [Read original article]() Author: Ancil Crayton Published: 2026-10-26T04:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [LLM evaluation / benchmarking](), [Large Language Model](), [AI Chat]() Tags: [audit](), [automatic](), [compare](), [comparison](), [comparisons](), [cs-ai](), [cs-cl](), [cs-hc](), [errors](), [evaluation](), [interactive](), [llm](), [matching](), [mistakes](), [model](), [models](), [performance](), [prompt](), [questions](), [responses](), [uncertainty](), [validating]() ### AI overview Overall scorer accuracy can conceal prompt-specific errors that distort measured differences between LLM follow-up prompts. In this study, a balanced audit of 288 responses had an overall F₁ of .916, yet the scorer flagged 2.6%-33.3% of responses the primary LLM rater judged not to contain unsupported claims, with uncertain prompt-specific estimates. For one comparison, an apparent 9.3-point increase fell to 2.3 points under two separate raters; the primary-rater interval included zero. The authors recommend auditing both prompts on matching cases and accounting for scorer and rater uncertainty. Their attempted prompt-specific correction remained too uncertain for a firm conclusion. ### Source excerpt arXiv:2610.31678v1 Announce Type: cross Abstract: Many LLM studies compare how models respond to different follow-up prompts. For example, one prompt may neutrally ask a model to reconsider, while another insists that an invented film really exists. The difference is used to measure the effect of conversational pressure. An automatic scorer then labels whether each response makes an unsupported factual claim. Researchers commonly validate the scorer with one overall accuracy or $F_1$ score. However, high overall performance can hide prompt-specific mistakes: if the scorer makes more errors after one prompt, the measured difference can be too large, too small, or point in the wrong direction. An audit containing mostly difficult responses can also misrepresent performance in the full study. We derive how such errors alter a prompt comparison and examine the problem using six models, 180 invented-entity questions, and six prompt conditions. In a balanced audit of 288 responses, overall $F_1$ was $.916$. Yet, depending on the prompt, the scorer flagged 2.6%-33.3% of responses that the primary LLM rater judged not to contain an unsupported claim; these prompt-specific estimates were uncertain. In a second audit, the scorer and two separate LLM raters evaluated the same 300 model-question cases under each prompt. An apparent 9.3-point increase from asserting certainty rather than neutrally asking the model to reconsider fell to 2.3 points under both raters. The primary-rater interval included zero, so it did not establish an increase. Two other prompt comparisons kept their direction under both raters, while an attempted prompt-specific correction remained too uncertain for a firm conclusion. Researchers should therefore validate the exact difference they plan to report, not just the scorer as a whole: audit both prompts on matching cases and include scorer and rater uncertainty in the final result. ## VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents DevFeed: [VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents]() Original publisher: [Read original article]() Author: Jiaju Chen, Min Yang, Jinghua Piao, Xiaochong Lan, Xu Xia, Xiangnan He, Yong Li Published: 2026-10-26T04:00:00Z Content type: article Language: en Sources: [arXiv: Machine Learning]() Topics: [Reinforcement learning](), [llm-reasoning](), [Large Language Model]() Tags: [agents](), [benchmarks](), [cost](), [cs-lg](), [experiments](), [feedback](), [generation](), [learning](), [lightweight](), [llm](), [llm-agents](), [performance](), [policy](), [ppo](), [reduce](), [responses](), [results](), [separate](), [sequences](), [temporal](), [token]() ### AI overview VETTA assigns credit at both response and token levels for multi-turn LLM agents. It uses separate turn- and token-value heads on a shared lightweight critic, combines turn advantages with within-response-centered token residuals for PPO updates, and reduces value-estimation cost by retaining only early Transformer blocks from the actor's pretrained checkpoint. The abstract reports success rates of 95.5% on ALFWorld, 76.0% on WebShop, and 52.2% on ScienceWorld with Qwen2.5-7B-Instruct; on Sokoban, Qwen2.5-VL-3B-Instruct reaches 84.9% with sparse rewards and 80.7% with dense rewards. Critic-depth comparisons reportedly retain strong task performance with substantially lower critic-side computation. ### Source excerpt arXiv:2610.08402v2 Announce Type: replace Abstract: Multi-turn LLM agents often receive sparse task feedback across several interactions, while generating each response token by token. This creates two related credit-assignment questions: which responses helped achieve the outcome, and which generation decisions mattered within each response? Existing methods typically focus on only one level: turn-level methods evaluate complete responses but do not distinguish the decisions within them; token-level methods can propagate feedback across turns but do not explicitly model credit for each response. These complementary limitations motivate learning credit at both levels and coordinating it in a single policy update. We introduce VETTA, a credit assignment method that jointly learns turn- and token-level values through separate heads on a shared lightweight critic. VETTA computes advantages along both temporal sequences and combines each turn advantage with a within-response-centered token residual for PPO updates. Furthermore, to reduce value-learning cost, the critic retains only early Transformer blocks from the pretrained checkpoint used to initialize the actor. Experiments on four agent benchmarks demonstrate VETTA's strong performance. With Qwen2.5-7B-Instruct, VETTA achieves success rates of 95.5% on ALFWorld, 76.0% on WebShop, and 52.2% on ScienceWorld. With Qwen2.5-VL-3B-Instruct, it reaches 84.9% and 80.7% success on Sokoban under sparse and dense rewards, respectively. Critic-depth comparisons further show strong task performance with substantially lower critic-side computation. These results demonstrate that coordinating turn- and token-level credit with a lightweight shared critic improves agent performance while keeping value estimation efficient. Code is available at https://github.com/Jiaju-Chen/VETTA-official. ## MusicTwins: Training Audio-Language Models to Listen like Their Persona DevFeed: [MusicTwins: Training Audio-Language Models to Listen like Their Persona]() Original publisher: [Read original article]() Author: Milan Liessens Dujardin, Kevin Miao Published: 2026-10-26T04:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [post-training](), [Fine-tuning](), [Reinforcement learning]() Tags: [audio](), [cs-ai](), [cs-sd](), [evaluation](), [fine-tuning](), [language-models](), [llms](), [memory](), [music](), [post-training](), [profile](), [reinforcement-learning](), [responses](), [training]() ### AI overview MusicTwins trains an audio-language model to produce responses shaped by a specific person's musical tastes and broader persona. It builds persona representations from human writings, covering background, views, personality, writing style, and musical profile, then combines persona summaries and personal memory with supervised fine-tuning and reinforcement learning of Audio Flamingo Next. Evaluation on ten held-out personas found notable gains in persona fidelity, including personality, writing style, and responses to music. The evidence describes a framework and evaluation, but gives no quantitative results or details about the training data and procedures. ### Source excerpt arXiv:2610.32495v1 Announce Type: cross Abstract: People differ in how they perceive music, what associations they make with it, what they like about it, and how they respond to it. Computational models intended to simulate people in environments that include music should account for these differences. Although LLMs have been shown to reproduce aspects of human behavior, whether audio-language models (ALMs) can reproduce a specific person's responses to textual and musical stimuli remains underexplored. We introduce MusicTwins, a framework that combines persona summaries, personal memory, and post-training to generate person-specific responses. We construct persona representations from human writings, capturing background, views, personality, writing style, and a musical profile, and post-train Audio Flamingo Next through supervised fine-tuning and reinforcement learning. Evaluation on ten held-out personas shows notable improvements for Audio Flamingo Next in persona fidelity, including personality, writing style, and responses to music, supporting MusicTwins as a foundation for training audio-language models to listen like their persona. ## Can Broad Biomedical Knowledge be Contextualized into Scenario-Grounded Propositions? DevFeed: [Can Broad Biomedical Knowledge be Contextualized into Scenario-Grounded Propositions?]() Original publisher: [Read original article]() Author: Qingyuan Zeng, Ziyang Chen, Pengxiang Cai, Zixin Guan, Anglin Liu, Lang Qin, Xinyao Lai, Jintai Chen Published: 2026-10-26T04:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [knowledge-engineering](), [Data analysis](), [hierarchical agent systems](), [AI Integration Strategies](), [clustering]() Tags: [agent](), [agent-framework](), [biomedical](), [clinical-trials](), [cs-ai](), [data](), [discovery](), [domain](), [feedback](), [framework](), [inspect](), [knowledge](), [multi-agent](), [optimization](), [patterns](), [responses](), [results](), [schema](), [search](), [support](), [validation]() ### AI overview SCENE aims to turn broad biomedical knowledge into propositions grounded in specific datasets. Its upper level derives search directions and maps them to the dataset schema; its lower level uses multi-objective optimization to find propositions balancing evidential strength and data support. Feedback iteratively refines the search. In clinical-trial scenarios, SCENE finds specific, well-supported patient subgroups with heterogeneous treatment benefits and outperforms existing baselines. In LINCS L1000 studies, it identifies perturbational contexts with strong target-response matching and high positive rates. The abstract presents these as traceable hypotheses for expert inspection and follow-up validation; it does not provide details of the baselines or metrics. ### Source excerpt arXiv:2605.27082v2 Announce Type: replace Abstract: Biomedical discovery often requires connecting broad biomedical knowledge with specific experimental or clinical data. Background knowledge suggests relevant mechanisms but is usually too general to map directly onto dataset variables, while data-driven patterns can be dataset-specific and hard to interpret mechanistically. We study this missing link as knowledge contextualization: transforming broad biomedical knowledge into evidence-supported, scenario-grounded propositions that domain experts can inspect, replay, and validate. We propose SCENE, a bi-level multi-agent framework that treats knowledge contextualization as iterative search. The upper level converts broad knowledge into search directions and grounds them in the dataset schema. The lower level executes these directions through multi-objective optimization to identify concrete propositions that balance evidential strength and data support. Feedback between the two levels progressively refines the search. We evaluate SCENE in two settings: discovering patient subgroups with heterogeneous treatment benefits in clinical trial scenarios, and identifying context-specific biological responses in LINCS L1000 studies. In clinical trials, SCENE discovers specific, well-supported subgroups and outperforms existing baselines. In L1000 studies, SCENE identifies perturbational contexts with strong target-response matching and high positive rates. These results show that SCENE bridges broad knowledge and scenario-specific evidence, producing traceable, inspectable hypotheses for follow-up validation. ## Reverse engineering a commercial spatial audio effect DevFeed: [Reverse engineering a commercial spatial audio effect]() Original publisher: [Read original article]() Author: Anne Barela Published: 2026-10-24T13:30:50Z Content type: article Language: en Sources: [Adafruit Industries]() Topics: [Reverse Engineering](), [Pipewire]() Tags: [algorithm](), [alignment](), [audio](), [convolution](), [effect](), [engineering](), [filter](), [invariant](), [linear](), [linux](), [llm](), [music](), [rebuild](), [responses](), [reverse-engineering](), [sound](), [spatial](), [version](), [windows]() ### AI overview To recreate the effect, Austin Shijo measured its filter response instead of reconstructing its algorithm. He played a purpose-built test file through the Windows program once and recorded the output; the file combined impulses, a sine sweep, tones, multitone, repeated sections, noise, and alignment clicks. Tests indicated the main surround mode was linear and time invariant, allowing it to be captured with four impulse responses. Two other modes varied gain over time and could not be reproduced with a fixed convolution. Earlier attempts using HRTF databases had stalled: MIT KEMAR sounded wrong to him, while SADIE II's room responses came closer. ### Source excerpt Austin Shijo wanted a Linux version of a Windows audio effect that pushed music out of the head and made it feel less pressed against the ears. Writing on epestr.com, he describes first trying to rebuild the effect by hand with PipeWire, HRTF databases and an LLM as a sounding board. The MIT KEMAR set [...] ## Metric Conformal Prediction via the Conditional Expected Local Radius DevFeed: [Metric Conformal Prediction via the Conditional Expected Local Radius]() Original publisher: [Read original article]() Author: Rui Qiu, Fang Yao, Zhou Yu Published: 2026-10-23T04:00:00Z Content type: article Language: en Sources: [arXiv: Machine Learning]() Topics: [Machine learning]() Tags: [conditional](), [conditional-probability](), [conformal-prediction](), [consistency](), [coverage](), [cs-lg](), [distribution](), [local](), [metric](), [multimodality](), [radius](), [responses](), [spaces](), [stat-ml](), [transport](), [uncertainty-quantification](), [vector]() ### AI overview The paper introduces conditional expected local radius (CELR), a conformal prediction method for responses in general metric spaces. CELR measures local conditional probability concentration, so prediction sets can reflect heterogeneity and multimodality without requiring densities or vector-space structure. Its conditional distribution provides nonconformity scores comparable across covariate values and avoids transport comparisons between distance profiles. A metric distributional random forest handles multivariate covariates and nonlinear interactions; one forest fit supplies the score components, avoiding repeated local fitting. The authors report uniform estimator consistency, asymptotic conditional validity under suitable regularity conditions, and finite-sample marginal coverage under exchangeability. Numerical studies report effective uncertainty quantification across metric spaces and computational savings over the conditional profile score method. ### Source excerpt arXiv:2610.31336v1 Announce Type: cross Abstract: We introduce the conditional expected local radius (CELR) and develop a new conformal prediction method for non-Euclidean responses in general metric spaces. CELR measures local conditional probability concentration, allowing prediction sets to accommodate heterogeneity and multimodality without requiring densities or a vector space structure. Its conditional distribution yields a nonconformity score that is comparable across covariate values and avoids costly transport comparisons between distance profiles. We further adapt a metric distributional random forest that handles multivariate covariates and nonlinear interactions, while a single forest fit supplies all score components, reducing computation by eliminating repeated local fitting. We establish uniform consistency of the estimators and asymptotic conditional validity under suitable regularity conditions, together with finite-sample marginal coverage under exchangeability. Numerical studies demonstrate effective uncertainty quantification across different metric spaces and substantial computational savings over the conditional profile score method. ## Beyond End-to-End Recovery: Towards a Better Understanding of LLM Steganography via Rolewise Decomposition DevFeed: [Beyond End-to-End Recovery: Towards a Better Understanding of LLM Steganography via Rolewise Decomposition]() Original publisher: [Read original article]() Author: Om Tailor, Yucheng Pan, Solaleh Mohammadi, Xinyan Liu, Kaiqing Zhang Published: 2026-10-23T04:00:00Z Content type: article Language: en Sources: [arXiv: Cryptography and Security]() Topics: [LLM security](), [Large Language Model]() Tags: [cs-cl](), [cs-cr](), [cs-ma](), [decoder](), [decoding](), [e2e](), [encoding](), [evaluation](), [framework](), [instructions](), [language-model](), [large-language-model](), [llm](), [messages](), [patterns](), [private](), [recovery](), [responses](), [safety](), [steganography](), [text](), [token](), [transmission]() ### AI overview End-to-end recovery alone can hide whether LLM steganography fails at encoding or decoding, and cannot show how a private message is represented in transmitted text. The paper proposes evaluating these roles separately, including deterministic verification. Among 7,200 encoder responses, it reports 168 verifiable encodings that decoders missed and 706 responses from which private messages were reconstructed using word-length or token-initial patterns not specified in encoder instructions. The authors argue this separation can clarify steganographic behavior and safety concerns. ### Source excerpt arXiv:2610.30516v1 Announce Type: new Abstract: In large language model (LLM) steganography, an encoder hides a private message in an otherwise ordinary response, as text transmitted to a decoder, who attempts to recover it. The success of this private-message transmission is commonly measured through end-to-end (E2E) recovery, which checks whether the decoder correctly reconstructs the private message. However, E2E recovery cannot distinguish encoding failures from decoding failures or reveal how the private message is represented in the encoder's transmitted text. To address these limitations, we introduce a rolewise decomposition framework that separately examines how private messages are represented in encoder transmitted texts, what decoders recover, and what deterministic verification can establish. Across 7,200 encoder responses, we identify 168 verifiable encodings missed by decoders and 706 responses containing private messages reconstructed through word-length or token-initial patterns not explicitly specified in the encoder's instructions. These findings demonstrate that E2E recovery alone may obscure important distinctions between encoding and decoding, motivating their separate evaluation to better understand and mitigate the safety issues caused by LLM steganography. ## Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals DevFeed: [Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals]() Original publisher: [Read original article]() Author: Louise Yorke Published: 2026-10-23T04:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [Large Language Model](), [Temperature settings in AI](), [AI Prompt Templates](), [A/B Testing](), [llm-reasoning]() Tags: [arena](), [cs-ai](), [cs-cl](), [cs-cy](), [evaluations](), [format](), [gpt](), [instructions](), [llm](), [models](), [prompt](), [reasoning](), [repeat](), [responses](), [score](), [setup](), [stable](), [temperature](), [version]() ### AI overview Across 500 Arena-Hard v2.0 prompts, GPT-4.1's win/tie/loss judgments changed substantially even when aggregate scores stayed similar. Repeating the default setup shifted its score by about one point while changing outcomes on 21.2% of prompts. Changing instructions produced further sensitivity: a setup omitting reasoning and several other instructions changed outcomes on 28.2 percentage points more prompts than unchanged reruns did on average and lowered the score by 4.09 points; an accuracy-prioritizing setup changed outcomes by 6.7 points more while barely moving the aggregate score. The authors conclude that aggregate scores alone do not establish prompt-level repeatability. For evaluations using individual verdicts, they recommend repeating each setup and reporting both aggregate scores and prompt-level agreement. ### Source excerpt arXiv:2610.30512v1 Announce Type: cross Abstract: LLM judges are used to determine which of two answers is better. Trusting these verdicts requires knowing how repeatable they are and how much they depend on the judge's instructions. We show that similar aggregate scores can conceal substantial changes in individual verdicts, both when the same setup is repeated and when the instructions change. Using 500 Arena-Hard v2.0 prompts with two models' answers held fixed, we run GPT-4.1 as the judge under all eight combinations of criteria, output format and task framing, and repeat every setup unchanged at temperature zero on a pinned model version. Rerunning the default setup moves the aggregate score by about one point but changes the win/tie/loss outcome on 21.2% of prompts. Instruction changes introduce further sensitivity: in one setup, which asks for the verdict without reasoning and also drops several other instructions, outcomes change on 28.2 percentage points more prompts than the two setups' unchanged reruns do on average, and the aggregate score falls by 4.09 points. In another, where the criteria prioritise accuracy, outcomes change on 6.7 percentage points more prompts by the same comparison, while the aggregate score barely moves. A stable aggregate score therefore does not establish that individual verdicts are repeatable. This distinction matters when those verdicts are used as preference labels or to assess individual responses. Evaluations that rely on individual verdicts should repeat each setup they use and report both aggregate scores and prompt-level agreement. ## RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use DevFeed: [RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use]() Original publisher: [Read original article]() Author: You Liu, Yue Liu, Quanchao Lu Published: 2026-10-23T04:00:00Z Content type: news Language: en Sources: [arXiv: Artificial Intelligence](), [arXiv: Cryptography and Security]() Topics: [retry](), [AI Chat](), [Multi-step AI processes](), [Error Handling]() Tags: [agent](), [control](), [cs-ai](), [cs-cr](), [error](), [gpt](), [interface](), [key](), [luna](), [models](), [none](), [order](), [permission](), [progress](), [recovery](), [reporting](), [required](), [responses](), [retries](), [retry](), [tool](), [tool-use]() ### AI overview RetryBench finds that successful recovery from a timed-out tool call does not establish that an agent reliably rejects retries unsupported by the visible contract. In a key control, offered changed keys led to unsupported retries by Sol (1/80), Terra (55/80), and Luna (60/80); when candidates were removed and models had to supply a key, Luna still made 8/80 unsupported dispatches, while Sol and Terra made none. Because the conditions and output requirements differed, this is not a pure display-effect comparison. Removing a weak-contract label increased retries in some Sol and Terra coded-action conditions. A separate canonical-action block found a positive Terra contrast, concentrated in two task/order conditions; the other five contrasts were unresolved after multiplicity correction. An explicit-contract cue produced deferral in all 762 returned responses, leaving recovery unresolved. The evidence is limited to four purposive tasks, fixed GPT endpoints, and one recovery decision; it does not establish internal mechanisms or episode-level safety. ### Source excerpt arXiv:2610.15884v3 Announce Type: replace-cross Abstract: False retry permission (FRP) occurs when an agent retries a timed-out tool call without support from the visible contract. RetryBench separates this error from realized recovery. Successful protected recovery does not imply reliable rejection of unsupported retries. In a 720-response key control, valid keys usually enabled safe progress, but offered changed keys elicited unsupported retries in Sol (1/80), Terra (55/80), and Luna (60/80). A later 240-response control removed the candidate and required models to supply a key. Luna still made 8/80 unsupported dispatches; Sol and Terra made none in 80 responses each. Accepting a displayed bad key therefore cannot explain all observed errors, although changed output requirements and conditions preclude a pure display-effect comparison. Removing the weak-contract label increased retries in Sol and Terra's coded-action conditions in a six-endpoint study. A separate 1,536-response canonical-action block also found a positive Terra contrast (115/128 versus 124/128 retries), concentrated in two task/order conditions; the other five contrasts remained unresolved after multiplicity correction. An explicit-contract cue control elicited deferral in all 762 returned responses, leaving recovery unresolved. These results support reporting permission and recovery together across declared interface variants. The evidence covers four purposive tasks, fixed GPT endpoints, and one recovery decision; it identifies neither internal mechanisms nor episode-level safety. ## D2R2: Discrete Diffusion with Regulation Reinforcement for Single-Cell Perturbation Prediction DevFeed: [D2R2: Discrete Diffusion with Regulation Reinforcement for Single-Cell Perturbation Prediction]() Original publisher: [Read original article]() Author: Ninghan Fan, Qi Liu, Xunuo Zhu, Yukai Sun, Luyuan Chen, Xuheng Zhou, Yuetian Du, Ming Kong, Xiaojun Zhu, Jie Liu, Zhan Zhou, Qiang Zhu Published: 2026-10-23T04:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [Reinforcement learning](), [grpo]() Tags: [cs-ai](), [diffusion](), [expression](), [generation](), [network](), [optimization](), [ordering](), [performance](), [policy](), [random](), [regulation](), [responses](), [reward](), [transcription](), [uncertainty](), [virtual-cell]() ### AI overview D²R² predicts single-cell perturbation responses by generating gene expression tokens step by step, with each generated response conditioning those still masked. Its Regulatory Policy Module starts from a gene regulatory network inferred from control cells and adapts gene ordering to the perturbation and partial profile; group-relative policy optimization then refines that ordering using final perturbation-effect agreement as reward. On Norman19, it reports the best results across five metrics, and it remains competitive on VCC-H1. Controlled ablations found biological-prior ordering outperformed random ordering and was more reliable than uncertainty-based heuristics; reversing the prior worsened every metric. Analyses indicate the refined policy generates regulatory genes early and promotes perturbation-specific transcription factors and responsive genes. ### Source excerpt arXiv:2608.15288v2 Announce Type: replace Abstract: Predicting single-cell transcriptomic responses to genetic perturbations is central to functional genomics and virtual-cell modeling. Existing approaches, however, typically predict an entire expression profile as a whole, leaving the order in which individual gene responses are generated unmodeled. To address this problem, we introduce \textbf{$D^{2}R^{2}$} (\textbf{D}iscrete \textbf{D}iffusion with \textbf{R}egulation \textbf{R}einforcement), which reformulates perturbation prediction as regulation-guided gene-wise progressive generation. A Masked Discrete Diffusion Model represents expression as ordinal tokens and reconstructs a fully masked profile step by step, allowing generated gene responses to condition those that remain masked. A Regulatory Policy Module initializes the generation policy from a gene regulatory network inferred from control cells and adapts it to the perturbation and current partially generated state. Then, group-relative policy optimization refines only the ordering policy using final perturbation-effect agreement as reward. Across Norman19 and VCC-H1, $D^{2}R^{2}$ achieves the best performance on all five metrics on Norman19 and remains competitive on H1. Controlled ablations holding the generator and generation budget fixed show that biological-prior ordering improves over random ordering and is more reliable than uncertainty-based heuristics, whereas reversing the biological-prior ordering degrades every metric. Biological analyses further show that the refined policy prioritizes regulatory genes early while promoting perturbation-specific transcription factors and responsive genes. These results establish gene generation order as an effective, controllable, and biologically interpretable dimension of single-cell perturbation prediction. ## Model-intrinsic, Weight-level, and Training-free Spectral Fingerprinting for Large Language Models DevFeed: [Model-intrinsic, Weight-level, and Training-free Spectral Fingerprinting for Large Language Models]() Original publisher: [Read original article]() Author: Yanan He, Yanli Chen, Ling Xiong, Gejian Zhao, Hanzhou Wu Published: 2026-10-23T04:00:00Z Content type: article Language: en Sources: [arXiv: Cryptography and Security]() Topics: [LLM security](), [Large Language Model](), [LLM Techniques](), [Post-training optimization](), [quantization](), [LLM evaluation / benchmarking]() Tags: [arxiv](), [auditing](), [cs-cr](), [distillation](), [fingerprint](), [inference](), [intrinsic](), [large-language-models](), [layer](), [ownership](), [post-training](), [projection](), [prompt](), [quantization](), [query](), [responses](), [robust](), [similarity](), [training](), [under](), [weight]() ### AI overview The paper proposes a training-free, weight-level fingerprint for auditing ownership of open-weight language models. It extracts scalar features from singular-value distributions of GATE projection weights in gated FFNs, pools layer features by depth into a compact model fingerprint, and compares fingerprints using linear centered kernel alignment. The depth-wise pooling preserves coarse layer organization while allowing comparisons across models with different depths. Unlike response- or activation-based approaches, the pipeline requires no input samples, generated outputs, forward activations, or auxiliary verifier. Experiments across representative model families and post-training variants report separation of source-related from unrelated models and stable similarity after instruction tuning, quantization, pruning, and distillation-related modifications. Evidence here is limited to the abstract. ### Source excerpt arXiv:2610.30626v1 Announce Type: new Abstract: The increasing release and reuse of Large Language Models (LLMs) as open-weight checkpoints make model fingerprinting important for ownership protection and offline auditing. Existing fingerprinting methods typically construct evidence from query responses, generated outputs, or input-induced activations. Although effective in representation-based settings, they can be affected by prompt selection, decoding configuration, inference environment, and forward-pass implementation, which may limit their applicability. Here we introduce a model-intrinsic weight-level framework that derives compact fingerprints from spectral structures encoded in LLM parameters. Specifically, the framework constructs GATE spectral scalar features from the singular-value distributions of GATE projection weights in gated feed-forward networks (FFNs), and aggregates the resulting layer-wise features through depth-wise segmented pooling to form a model-level fingerprint. Such representation preserves coarse layer-depth organization while reducing dependence on strict layer-to-layer alignment, enabling comparison across LLMs with different depths. The similarity between fingerprints is computed by linear centered kernel alignment. The entire pipeline is training-free and does not require input samples, generated outputs, forward activations, or an auxiliary verifier. Experiments on representative LLM families and post-training variants have demonstrated a clear separation between source-related models and unrelated ones, and stable similarity under instruction tuning, quantization, pruning, and distillation-related modifications, which verify that spectral features are robust evidence for LLM fingerprinting. [Next page]()