# Audit-then-score

Audit-then-Score is a protocol for revising AI evaluation benchmark labels and rationales through evidence-based auditing before model scoring.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Hearsay: Who Can Audit the Record a Deployed Agent Harness Writes?

DevFeed: [Hearsay: Who Can Audit the Record a Deployed Agent Harness Writes?](<https://devfeed.tech/articles/hearsay-who-can-audit-the-record-a-deployed-agent-harness-writes-142117.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.32495>)

Author: Jiahong Dai, Zhuochen Yang, Pengyang Shao, Kelvin Ng, Zhongyi Liu, Chengquan Ju, Yuting He, Bo Hu

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [audit](<https://devfeed.tech/topics/audit.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [audit](<https://devfeed.tech/tags/audit.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ce](<https://devfeed.tech/tags/cs-ce.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [frameworks](<https://devfeed.tech/tags/frameworks.md>), [harness](<https://devfeed.tech/tags/harness.md>), [ledger](<https://devfeed.tech/tags/ledger.md>), [model](<https://devfeed.tech/tags/model.md>), [records](<https://devfeed.tech/tags/records.md>), [sandbox](<https://devfeed.tech/tags/sandbox.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [writer](<https://devfeed.tech/tags/writer.md>)

### AI overview

The paper argues that an agent run record cannot independently verify itself when the harness is its sole writer. In a benchmark spanning five harnesses, fourteen tasks, and examiners from three vendors plus a human panel, readers often identified faults but could not check the record; model examiners overstated corroboration, and even the strongest detected only half of inserted alterations. The authors report that all sixteen surveyed frameworks had the harness as the only writer. Their proposed sandbox adds a ledger of exchanges between harness and model and compares it with the record in both directions to detect omissions and additions. On unseen records, it caught 64 of 66 alterations without flagging 84 clean records, and caught 109 of 115 held-out records across five formats.

### Source excerpt

arXiv:2609.32495v2 Announce Type: replace-cross Abstract: The record of an agent's run is written by the harness that drives the agent. When the run is later disputed, the auditor was not there and has only this record. We ask whether someone who was not there can check the record, and what decides the answer. It depends on who wrote what a check can rely on: if the harness wrote all of it, the record is only the harness's own word, and we call it hearsay. We build Hearsay, a benchmark that examines the record and not the task: five harnesses run fourteen tasks, model examiners from three vendors and a human panel read the records and answer questions about the runs, and every passage they cite is labelled mechanically by who wrote it. We also audit what is deployed: in all sixteen frameworks, the harness is the only writer. With one writer, readers can name the fault from the record, in up to nine runs in ten, but they cannot check the record. The model examiners claim corroboration for 18 to 55 points more of their evidence than the labels support, a gap we then predicted for runs driven by another vendor's model and found again. And the best of them notices only half of the alterations we make to a record, while the two that name the fault most often cannot tell what a harness left out. With a second writer, the record can be checked. Our sandbox keeps a ledger of what passes between harness and model, and reading the ledger against the record in both directions shows what the record leaves out and what it adds. We test this as an auditor would use it, on one unseen record at a time. On runs driven by a third vendor's model, with every rule committed before the records existed, it reports 64 of 66 alterations and flags none of 84 clean records. On held-out records in five formats it reports 109 of 115. What makes a record evidence is a second writer, not a better reader.

## Movie2Game-Bench: Holistic Evaluation of Film-to-Game Adaptation Beyond Playability

DevFeed: [Movie2Game-Bench: Holistic Evaluation of Film-to-Game Adaptation Beyond Playability](<https://devfeed.tech/articles/movie2game-bench-holistic-evaluation-of-film-to-game-adaptation-beyond-playability-142821.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24669>)

Author: Jialing Xie, Yifan Zhou, Yiwen Zhang, Junjun He

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [interactive storytelling](<https://devfeed.tech/topics/interactive-storytelling.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [bench](<https://devfeed.tech/tags/bench.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-mm](<https://devfeed.tech/tags/cs-mm.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [film](<https://devfeed.tech/tags/film.md>), [game](<https://devfeed.tech/tags/game.md>)

### AI overview

Movie2Game-Bench evaluates film-to-game adaptation through general and movie-specific assessments plus playtesting by three GUI model players. Its protocol links source content to gameplay steps and implementation evidence; it covers 22 films and 658 reference items. Across 14 models, playable sequences had the lowest content-presence mean for every model, averaging 37.9/100 versus 54.8 for iconic elements. In judgments of retained emotional-development content, narrative choices accounted for 60.0% and scenario operation 7.2%. The findings point to preserving film situations that can support gameplay and linking retained content to player actions and consequences.

### Source excerpt

arXiv:2611.24669v1 Announce Type: new Abstract: Adapting a film into a game requires selecting source elements and translating them into player actions, rules, and feedback. We introduce Movie2Game-Bench (M2G-bench), a benchmark for evaluating film-to-game adaptation beyond playability. M2G-bench combines general evaluation, movie-specific evaluation, and playtesting by three GUI model players. The protocol connects source content to gameplay steps and implementation evidence, capturing both content preservation and interactive realization. The benchmark comprises 22 films and 658 source-specific reference items. Evaluating 14 models on the same films with their selected primary harnesses reveals complementary strengths in game quality, adaptation, and player experience. A shared weakness concerns playable sequences: this category has the lowest content-presence mean in all 14 models, averaging 37.9 out of 100 compared with 54.8 for iconic elements. Among judgments of retained emotionaldevelopment content, narrative choices account for 60.0% and scenario operation for 7.2%. These findings identify two linked priorities for film-to-game adaptation: preserving source situations that can support gameplay and connecting retained content to player actions and consequences.

## Gemini 4 Neon Launches Same Day at Three Times Carbon's Speed, Ranks Third Overall With AA Score of 61

DevFeed: [Gemini 4 Neon Launches Same Day at Three Times Carbon's Speed, Ranks Third Overall With AA Score of 61](<https://devfeed.tech/articles/gemini-4-neon-launches-same-day-at-three-times-carbon-s-speed-ranks-third-overall-with-aa-score-of-61-141943.md>)

Original publisher: [Read original article](<https://joho-todai.com/en/gemini-4-neon-carbon-three-times-faster/>)

Author: Toru Misaki

Published: 2026-11-18T23:08:47Z

Content type: news

Language: en

Sources: [Joho no Todai](<https://devfeed.tech/sources/joho-no-todai.md>)

Topics: [open-llm-leaderboard](<https://devfeed.tech/topics/open-llm-leaderboard.md>), [Parallel reasoning](<https://devfeed.tech/topics/parallel-reasoning.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Cryptography](<https://devfeed.tech/topics/cryptography.md>), [Claude](<https://devfeed.tech/topics/claude.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [app](<https://devfeed.tech/tags/app.md>), [carbon](<https://devfeed.tech/tags/carbon.md>), [cost](<https://devfeed.tech/tags/cost.md>), [family](<https://devfeed.tech/tags/family.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [gemini-4](<https://devfeed.tech/tags/gemini-4.md>), [gemini-app](<https://devfeed.tech/tags/gemini-app.md>), [independent](<https://devfeed.tech/tags/independent.md>), [index](<https://devfeed.tech/tags/index.md>), [intelligence](<https://devfeed.tech/tags/intelligence.md>), [intelligence-index](<https://devfeed.tech/tags/intelligence-index.md>), [model](<https://devfeed.tech/tags/model.md>), [neon](<https://devfeed.tech/tags/neon.md>), [score](<https://devfeed.tech/tags/score.md>), [speed](<https://devfeed.tech/tags/speed.md>), [times](<https://devfeed.tech/tags/times.md>), [united-states](<https://devfeed.tech/tags/united-states.md>), [was](<https://devfeed.tech/tags/was.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Gemini 4 Neon began rolling out on its November 18 announcement day through Google AI Studio's paid API, Gemini for enterprises, Antigravity for Pro and Ultra subscribers, and the Gemini app for Pro and Ultra subscribers. Adapted from Carbon, it scored 61 and ranked third on Artificial Analysis' Intelligence Index v5.01 at Xhigh; the text cautions that the updated benchmark and newly evaluated models make direct comparison with Carbon's earlier score unreliable. Google reports three times Carbon's speed and 32% lower cost per task at the default high setting; API rates are $3 per million input tokens and $15 per million output tokens, 25% below Carbon's standard rates. The excerpt also reports strong coding, business, science, and agent benchmark results, while noting that harnesses affect outcomes and many scores are Google-reported. Its 0.0% prompt-injection attack success result applies specifically with Xhigh reasoning and real-time safeguards. Serious biological and cybersecurity risks cannot be ruled out, according to the cited safety framework.

### Source excerpt

Neon, the fourth model in the Gemini 4 family, was announced November 18 in the United States and began rolling out the same day through the API and Gemini app. Adapted from Carbon, it promises three times the speed and 40% lower cost while scoring 61 and ranking third overall on the independent AA Intelligence Index.

## Offline Preference-Based Trajectory Evaluation

DevFeed: [Offline Preference-Based Trajectory Evaluation](<https://devfeed.tech/articles/offline-preference-based-trajectory-evaluation-139782.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.17541>)

Author: Fernando Diaz

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Trustworthy AI](<https://devfeed.tech/topics/trustworthy-ai.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-systems](<https://devfeed.tech/tags/agentic-systems.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-efficiency](<https://devfeed.tech/tags/data-efficiency.md>), [design](<https://devfeed.tech/tags/design.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [offline](<https://devfeed.tech/tags/offline.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [progress](<https://devfeed.tech/tags/progress.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [size](<https://devfeed.tech/tags/size.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [terminal](<https://devfeed.tech/tags/terminal.md>), [time](<https://devfeed.tech/tags/time.md>), [using](<https://devfeed.tech/tags/using.md>)

### AI overview

Binary terminal-success evaluation discards partial progress and produces tied comparisons on roughly 75% of instances in the reported benchmarks. The proposed preference-based trajectory evaluation compares pairs of trajectories using temporal preferences between their time-to-return profiles. The authors report that this reduces ties to roughly 35%, improving discriminative power, ranking stability, and data efficiency. They suggest evaluation design may contribute to apparent benchmark saturation, alongside data collection or task difficulty.

### Source excerpt

arXiv:2606.17541v2 Announce Type: replace-cross Abstract: Offline evaluation of agentic systems often collapses trajectories into a binary terminal success value, discarding information about partial progress. This, in turn, results in widespread ties that create statistical inefficiency by reducing effective sample size and weakening the ability to distinguish systems. We propose preference-based trajectory evaluation, an evaluation measure that compares pairs of trajectories directly by using temporal preferences between time-to-return profiles. We find that, across diverse agentic and interactive benchmarks, success-based metrics result in tied comparisons on roughly 75% of instances, while trajectory-aware preferences reduce ties to roughly 35%, improving discriminative power, ranking stability, and data efficiency. Our results suggest that benchmark saturation, often portrayed as the result of poor data collection or problem difficulty, may also be explained by the design of the evaluation measure.

## Eliciting Research Taste in LLMs through Future Research Direction Choice

DevFeed: [Eliciting Research Taste in LLMs through Future Research Direction Choice](<https://devfeed.tech/articles/eliciting-research-taste-in-llms-through-future-research-direction-choice-137494.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19794>)

Author: Dhruv Trehan, Paras Chopra

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI for science](<https://devfeed.tech/topics/ai-for-science.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>)

Tags: [choice](<https://devfeed.tech/tags/choice.md>), [comparisons](<https://devfeed.tech/tags/comparisons.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [llms](<https://devfeed.tech/tags/llms.md>), [ml](<https://devfeed.tech/tags/ml.md>), [research](<https://devfeed.tech/tags/research.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [steering](<https://devfeed.tech/tags/steering.md>)

### AI overview

The study operationalizes LLM research taste as choosing between future research directions, using human-authored recommendations as a behavioral reference rather than treating them as objectively correct. Its order-controlled evaluation covers 80 comparisons drawn from 19 recent ML papers across four subdomains. Across eight LLMs, author alignment ranges from 45.0% to 67.5%, and stable divergence from 15.0% to 36.2%; divergences largely persist under similarity controls, prompt ablations, and an author-preference framing check. The authors report shared non-author preferences across models, with recurring movement toward scaling or generalization and trade-offs between methodological and infrastructure or data extensions. They present this as an early descriptive evaluation, not a measure of objectively correct research choices.

### Source excerpt

arXiv:2611.19794v1 Announce Type: cross Abstract: Research taste is increasingly discussed in the context of AI scientists, yet remains little studied or measured in LLMs. This judgment matters especially in open-ended research, where a direction must be chosen before a verifiable objective is available. We operationalize research taste through one task of the scientific workflow: choosing which future research direction to pursue given a research goal and current results. Rather than predicting downstream scientometric metrics or treating author choices as objectively correct, we evaluate models against human-authored future-work directions and report author alignment, stable divergence, and unstable choice as behavioral measures of research-direction preference. We construct an order-controlled evaluation of 80 pairwise comparisons from 19 ML papers published between December 2025 and February 2026 across four subdomains. Each query presents a research goal, hypotheses tested with key results, and two candidate future directions: one author-recommended direction and one plausible distractor sampled from a similar paper. Across eight LLMs, author alignment ranges from 45.0% to 67.5%, while stable divergence ranges from 15.0% to 36.2%. These divergences largely persist under similarity controls, prompt ablations, and an author-preference framing check, and inter-model agreement suggests that some non-author preferences are shared across models. We characterize what models diverge toward using transition-category labels and selected reasoning examples, finding recurring movement toward scaling/generalization and trade-offs between methodological and infrastructure/data extensions. We view this as an early descriptive evaluation for eliciting research taste profiles in LLMs, and as a step toward understanding and steering their research preferences as they take on more autonomous roles in novel open-ended scientific discovery.

## CrossPloit: A Benchmark for Cross-Version PoC Migration in Python Web Applications

DevFeed: [CrossPloit: A Benchmark for Cross-Version PoC Migration in Python Web Applications](<https://devfeed.tech/articles/crossploit-a-benchmark-for-cross-version-poc-migration-in-python-web-applications-131736.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16588>)

Author: Hung Nguyen-Huu, Nguyen Anh Tuan, Ting Zhang, Yue Liu, Chengran Yang, Lwin Khin Shar, Eng Lieh Ouh, David Lo

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [aws inspector](<https://devfeed.tech/topics/aws-inspector.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agent](<https://devfeed.tech/tags/agent.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [api](<https://devfeed.tech/tags/api.md>), [applications](<https://devfeed.tech/tags/applications.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attack](<https://devfeed.tech/tags/attack.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [automated](<https://devfeed.tech/tags/automated.md>), [based](<https://devfeed.tech/tags/based.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [best](<https://devfeed.tech/tags/best.md>), [can](<https://devfeed.tech/tags/can.md>), [cross](<https://devfeed.tech/tags/cross.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [migration](<https://devfeed.tech/tags/migration.md>), [poc](<https://devfeed.tech/tags/poc.md>), [python](<https://devfeed.tech/tags/python.md>), [version](<https://devfeed.tech/tags/version.md>), [web](<https://devfeed.tech/tags/web.md>), [web-applications](<https://devfeed.tech/tags/web-applications.md>)

### AI overview

CrossPloit tests how well agents can migrate Python web application proof-of-concept exploits to earlier, unlisted vulnerable versions and verify the result by execution. Its 127 instances span 45 CVEs; each includes a version-pinned container, reference PoC, and vulnerability-specific oracle, with two validators checking that the observed effect comes from the disclosed vulnerability. Construction identified 149 vulnerable prior versions missing from four databases, 127 requiring migration. Across 18 agent configurations, the best resolved 58.3% of instances versus 8.7% for an adapted trace-guided baseline. Attack-vector shifts were especially difficult, with no configuration exceeding 37.9%. In one configuration, removing the reference PoC and environment specification cut resolution from 50.4% to 15.0%, showing the package's value. The benchmark measures both under-reported exposure and the limits of automating its enumeration.

### Source excerpt

arXiv:2611.16588v1 Announce Type: cross Abstract: Affected-version ranges in vulnerability databases decide which deployments scanners, patching pipelines, and incident responders treat as vulnerable, yet these ranges are often incomplete, so vulnerable releases are silently treated as safe. Auditing them requires executable evidence: given a proof-of-concept (PoC) exploit for a listed version, show that the same vulnerability can be triggered on an earlier, unlisted version. This cross-version PoC migration combines source-code analysis, environment repair, and execution-based verification. We introduce CrossPloit, a benchmark of 127 migration instances across 45 CVEs in the Python web ecosystem. Each instance includes a version-pinned container, a reference PoC, and a vulnerability-specific oracle; two validators check that the observed effect arises from the disclosed vulnerability. Benchmark construction confirms 149 vulnerable prior versions absent from the affected-version lists of four databases, 127 of which require migration. Across 18 agent configurations (nine models and two harnesses), the best configuration resolves 58.3% of instances, compared with 8.7% for an adapted trace-guided baseline. Attack-vector shifts are particularly difficult: no configuration resolves more than 37.9% of these instances. Higher resolve rates are associated with a smaller share of missed-API failures and a larger share of incomplete executions among the remaining failures. In one configuration, withholding the reference PoC and its environment specification reduces resolve rate from 50.4% to 15.0%, highlighting the value of the supplied reference package. CrossPloit provides an execution-based measurement of under-reported exposure and of how far its enumeration can be automated.

## Protocol-Aware Auditing of Domain-Blind Feature Attribution in Network Intrusion Detection

DevFeed: [Protocol-Aware Auditing of Domain-Blind Feature Attribution in Network Intrusion Detection](<https://devfeed.tech/articles/protocol-aware-auditing-of-domain-blind-feature-attribution-in-network-intrusion-detection-133063.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.16018>)

Author: Emmanuelli El-Mahmoud, Abanisenioluwa Kolawole Orojo, Pablo Rivas, Erika Leal

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [Feature Engineering](<https://devfeed.tech/topics/feature-engineering.md>), [Endpoint security](<https://devfeed.tech/topics/endpoint-security.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [attacks](<https://devfeed.tech/tags/attacks.md>), [attribution](<https://devfeed.tech/tags/attribution.md>), [audit](<https://devfeed.tech/tags/audit.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [blindness](<https://devfeed.tech/tags/blindness.md>), [can](<https://devfeed.tech/tags/can.md>), [control](<https://devfeed.tech/tags/control.md>), [controls](<https://devfeed.tech/tags/controls.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [detection](<https://devfeed.tech/tags/detection.md>), [domain](<https://devfeed.tech/tags/domain.md>), [feature](<https://devfeed.tech/tags/feature.md>), [intrusion-detection](<https://devfeed.tech/tags/intrusion-detection.md>), [network](<https://devfeed.tech/tags/network.md>), [network-intrusion](<https://devfeed.tech/tags/network-intrusion.md>), [protocol](<https://devfeed.tech/tags/protocol.md>)

### AI overview

Feature attribution in network intrusion detection can be distorted when methods query network states that violate protocol rules. The paper formalizes this as domain blindness and proves valid-query invariance: if models agree on the needed oracle information within a declared rule-valid set, explanations restricted to that set return the same attribution. It applies protocol-aware query controls to LIME, SHAP, Integrated Gradients, and DeepLIFT across four datasets and three classifier families. In six settings, standard LIME showed perfect top-feature repeatability while PA-LIME exposed variability (mean 0.575). After FGSM attacks and protocol repair, PA-DeepLIFT and PA-IG had the largest attribution-similarity curve areas (0.452 and 0.413); mean within-PA rank agreement rose from 0.217 to 0.366. The abstract identifies support modeling, raw-flow reconstruction, and adaptive explanation attacks as complementary validation layers.

### Source excerpt

arXiv:2611.16018v1 Announce Type: new Abstract: Feature-attribution methods can query rule-violating network states through perturbations, coalitions, references, paths, and ablations, making explanations and their evaluation sensitive to model behavior outside the operational domain. We formalize this failure mode as domain blindness and prove valid-query invariance: when models agree on the required oracle information within a declared rule-valid set, a procedure confined to that set returns the same attribution. We instantiate one protocol-aware query-control principle in LIME, SHAP, Integrated Gradients, and DeepLIFT and audit paired configurations across four NIDS datasets and three classifier families. In six settings where standard LIME reports perfect top-feature repeatability, PA-LIME reveals substantial variability, with a mean of 0.575. Under FGSM followed by protocol repair, PA-DeepLIFT and PA-IG achieve the largest attribution-similarity curve areas, 0.452 and 0.413, while mean within-PA rank agreement rises to 0.366 from 0.217. PA-XAI therefore turns structured domain assumptions into explicit, testable controls for explanation auditing; support modeling, raw-flow reconstruction, and adaptive explanation attacks provide complementary validation layers.

## PAL-Bench: Evidence-Grounded Profile Reconstruction from Longitudinal Personal Albums

DevFeed: [PAL-Bench: Evidence-Grounded Profile Reconstruction from Longitudinal Personal Albums](<https://devfeed.tech/articles/pal-bench-evidence-grounded-profile-reconstruction-from-longitudinal-personal-albums-131934.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.16175>)

Author: Qiwei Yan, Zhiqiang Yuan, Zexi Jia, Nanxing Hu, Kailin Lyu, Jie Zhou, Jinchao Zhang

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Entity resolution](<https://devfeed.tech/topics/entity-resolution.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Image processing](<https://devfeed.tech/topics/image-processing.md>), [sensitive data](<https://devfeed.tech/topics/sensitive-data.md>)

Tags: [album](<https://devfeed.tech/tags/album.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [best](<https://devfeed.tech/tags/best.md>), [builds](<https://devfeed.tech/tags/builds.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [entity-resolution](<https://devfeed.tech/tags/entity-resolution.md>), [framework](<https://devfeed.tech/tags/framework.md>), [identity](<https://devfeed.tech/tags/identity.md>), [metric](<https://devfeed.tech/tags/metric.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [profile](<https://devfeed.tech/tags/profile.md>), [provenance](<https://devfeed.tech/tags/provenance.md>), [public-records](<https://devfeed.tech/tags/public-records.md>), [records](<https://devfeed.tech/tags/records.md>), [social](<https://devfeed.tech/tags/social.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [temporal](<https://devfeed.tech/tags/temporal.md>), [timestamps](<https://devfeed.tech/tags/timestamps.md>)

### AI overview

PAL-Bench evaluates whether systems can reconstruct personal profiles and social relationships from noisy, longitudinal photo-album records while grounding claims in evidence. Its Evidence Compiler creates synthetic private worlds, renders album images, re-measures them through perception pipelines, and gives agents only perception-derived public records; targets, identity maps, and evidence paths stay hidden. The benchmark includes 50 synthetic users, 36,659 photo records, and 2,799 targets. An audit with 10 participants found its evidence structures match real albums, while equivalent public releases remain privacy-prohibitive. Across seven systems, results show that plausible owner summaries do not ensure accurate recurring-identity resolution or evidence citation. PAL-TRACE performs best by freezing identity bindings before mining owner facts, but difficult identity resolution remains unsolved.

### Source excerpt

arXiv:2606.16175v2 Announce Type: replace Abstract: Longitudinal personal albums are weak-schema multimodal databases: noisy perceptual records whose key facts require joins across faces, text, timestamps, locations, and repeated events. Existing visual, video, document, and lifelog benchmarks test sub-problems, but not album-scale profile reconstruction with social identity binding and evidence citation. Benchmarking this task is difficult because the ground truth needed for evaluation--owner profiles, social graphs, face-name maps, and evidence provenance--is private state that real albums cannot safely release. We introduce PAL-Bench, a controlled benchmark for evidence-grounded reconstruction under a public-record contract. Its Evidence Compiler builds latent private worlds, programs target-level evidence paths, renders album pixels, re-measures them through perception pipelines, and exports audited public/private views. Agents receive only perception-derived public records; targets, identifier maps, and evidence paths remain hidden. PAL-Bench contains 50 synthetic users, 36,659 public photo records, and 2,799 targets over owner facts, identities, and relations. A privacy-preserving audit with 10 participants confirms that PAL-Bench evidence structures match real private albums, though equivalent releases remain privacy-prohibitive. Across seven systems and two compute-matched diagnostics, a seven-metric protocol reveals a gap between plausible profile summarization and faithful social reconstruction: systems recover some owner facts but struggle with recurring identities and evidence citation. PAL-TRACE, a reference framework that freezes identity bindings before owner-fact mining, performs best but leaves hard identity resolution far from solved. PAL-Bench provides a testbed for perceptual entity resolution, multimodal data integration, temporal evidence aggregation, and provenance-aware structured prediction.

## Agreeing on the Answer but Not the Reason: Measurement Limits of LLM Reasoning Failure Taxonomies

DevFeed: [Agreeing on the Answer but Not the Reason: Measurement Limits of LLM Reasoning Failure Taxonomies](<https://devfeed.tech/articles/agreeing-on-the-answer-but-not-the-reason-measurement-limits-of-llm-reasoning-failure-taxonomies-128257.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13140>)

Author: Madhu Sudhan Rao Beesetty

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [ai evaluation](<https://devfeed.tech/topics/ai-evaluation.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [failure](<https://devfeed.tech/tags/failure.md>), [limits](<https://devfeed.tech/tags/limits.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-reasoning](<https://devfeed.tech/tags/llm-reasoning.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [taxonomies](<https://devfeed.tech/tags/taxonomies.md>)

### AI overview

Across 12 models and 11 languages, measured rates of "Compositional Collapse"--noticing a required part of a problem, then dropping or misapplying it--varied substantially with how failures were classified. Half of failures supported multiple categories; changing the classifier's test order shifted Collapse's share from 26.4% to 76.4% across orderings. Human agreement with the inspector was high on answer correctness (κ=0.932) but low on Collapse (κ=0.093). Using human instructions with the same inspector raised agreement with a second human from κ=0.147 to 0.636, suggesting prompt wording explains much of the gap. A deleted clause in Bengali and Chinese benchmark translations also inflated the inspector's labeling rate. The authors recommend reporting label co-occurrence, testing alternative tie-break orders, validating on a shared independent sample, and auditing systematic disagreements before comparing failure-mode rates.

### Source excerpt

arXiv:2611.13140v1 Announce Type: new Abstract: Failure taxonomies for LLM reasoning are increasingly reported as measurements: papers state what fraction of a model's errors fall into named categories, then compare those fractions across models. We show that for one such category the reported fraction is largely a property of the measurement procedure. We defined Compositional Collapse (a model notices a required sub-part of a problem, then drops or misapplies it) and measured it across twelve models and eleven languages on a public multilingual math benchmark (2,316 graded records). First, 50.1% of failures carry evidence for more than one category, so the reported plurality depends on the order in which the classifier tests them: reversing two tests moves it in eleven of twelve models, and across all six orderings Collapse's share spans 26.4% to 76.4%. Second, agreement collapses for the category but not for the task. Native speakers of all eleven languages judged 179 transcripts, agreeing with our inspector at kappa = 0.932 on answer correctness but only kappa = 0.093 on Collapse. A second native speaker per language re-rated 142 items; on 121 determinate pairs the two humans agree with each other at kappa = 0.659, while each agrees with the inspector at kappa <= 0.094. Re-running the same inspector model with the human instructions verbatim raises kappa against Human 2 from 0.147 to 0.636, so the gap is largely attributable to the prompt, not the model. Third, the inspector labels Collapse 1.81 times as often as native speakers, but only 1.17 times once a defective benchmark item (a deleted clause in the Bengali and Chinese translations) is excluded. We recommend four cheap checks before comparing failure-mode rates across models: report label co-occurrence, permute the tie-break, validate on a shared independent sample, and audit systematic disagreements.

## Governed Self-Improvement for a Local-First Security Analyzer: Baseline Measurements and a Preregistered Evaluation Protocol for Tantular Cyber

DevFeed: [Governed Self-Improvement for a Local-First Security Analyzer: Baseline Measurements and a Preregistered Evaluation Protocol for Tantular Cyber](<https://devfeed.tech/articles/governed-self-improvement-for-a-local-first-security-analyzer-baseline-measurements-and-a-preregistered-evaluation-protocol-for-tantular-cyber-126928.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10425>)

Author: Raditio Ghifiardi, Rifny Rachman

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Local AI](<https://devfeed.tech/topics/local-ai.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Self Hosted AI Tools](<https://devfeed.tech/topics/self-hosted-ai-tools.md>), [Code quality](<https://devfeed.tech/topics/code-quality.md>)

Tags: [analyzer](<https://devfeed.tech/tags/analyzer.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [local-first](<https://devfeed.tech/tags/local-first.md>), [measurements](<https://devfeed.tech/tags/measurements.md>), [security](<https://devfeed.tech/tags/security.md>), [self-improvement](<https://devfeed.tech/tags/self-improvement.md>)

### AI overview

Tantular Cyber's v0.3.0 benchmark suggests its deterministic scanner combination performs better than Semgrep alone: precision/recall were 0.843/0.915 versus 0.818/0.766. Its locally served model provides advisory explanations in Bahasa Indonesia, but only 57% passed the evidence-grounding check, compared with 95% for a different 8B model under identical prompts. Re-scoring found that grounding failures cited identifiers absent from the displayed evidence; a prompt change also appeared to improve the score partly by citing nothing. The authors report silent model-file migration and unreproducible figures as evaluation hazards. They propose a privacy-conscious improvement process using code-free feedback, curated data, staged interventions from prompts to LoRA and weight-space edits, hard gates, and human release approval. The first cycle is preregistered, but no results for it are reported.

### Source excerpt

arXiv:2611.10425v1 Announce Type: new Abstract: Local-first security tools let organizations analyze code without sending it to third-party services, but the small language models they can run locally are weaker than hosted ones and are hard to improve without breaching the privacy guarantee that motivates them. We present Tantular Cyber, an open-source, read-only repository security analyzer that keeps deterministic scanner findings authoritative and confines a locally served language model to advisory explanations in Bahasa Indonesia. We report its measured v0.3.0 baseline on a synthetic, ground-truth benchmark: combining Semgrep with an interprocedural call-graph scanner yields precision 0.843 and recall 0.915 (Semgrep alone: 0.818 and 0.766). The advisory layer is the weak point: 57% of explanations met an evidence-grounding check, against 95% for a different 8B model under identical prompts. A forensic re-scoring of retained outputs shows that every grounding failure cites a code identifier absent from the evidence shown, and that a prompt change which appeared to raise grounding from 0.56 to 0.72 did so partly by citing nothing. We document two evaluation-validity hazards we encountered: silent model-file migration during a run and non-reproducible published figures. Building on these lessons and on constraint-aware evaluation from security operations research, we propose a governed self-improvement loop (code-free feedback, curated data, a cheapest-first intervention ladder from prompts to LoRA and weight-space edits, hard gates, and human release approval) and preregister the protocol by which its first cycle will be judged. We report no results for the loop itself.

## ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound

DevFeed: [ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound](<https://devfeed.tech/articles/chaincheck-near-perfect-evidence-sufficiency-scores-can-conceal-sensitivity-to-construction-artifacts-paired-conditional-contrasts-that-expose-an-evaluation-confound-126208.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11965>)

Author: Hyojung Han, Ryangkyung Kang, Youngeun Kim, Seung-Hun Jeon

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [also](<https://devfeed.tech/tags/also.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [audit](<https://devfeed.tech/tags/audit.md>), [based](<https://devfeed.tech/tags/based.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bridge](<https://devfeed.tech/tags/bridge.md>), [can](<https://devfeed.tech/tags/can.md>), [cells](<https://devfeed.tech/tags/cells.md>), [complete](<https://devfeed.tech/tags/complete.md>), [conditional](<https://devfeed.tech/tags/conditional.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [construction](<https://devfeed.tech/tags/construction.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [edits](<https://devfeed.tech/tags/edits.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [external](<https://devfeed.tech/tags/external.md>), [extra](<https://devfeed.tech/tags/extra.md>)

### AI overview

ChainCheck finds that evidence-sufficiency scores can reflect artifacts from how broken passage sets are constructed rather than semantic multi-hop reasoning. In an exploratory crossed control on 2,214 HotpotQA pairs, a scorer that never reads the reasoning chain achieved nominal AUC 0.997, while a one-string match reached the maximum CS4 score of 0.500; the 11 evaluated systems scored 0.308-0.484. These are different metrics, and the authors interpret positive CS4 as operational bridge-consistency sensitivity. All systems were sensitive to an extra edit, which model-based naturalness ratings found detectable. On an independent 2,276-pair HotpotQA set, patterns specified on MuSiQue and 2WikiMultiHopQA did not replicate: five of six trained readouts were edit-dominated, and the judge's selectivity interval included zero. Judge scores also varied with batch layout. MuSiQue and 2Wiki records are public, but HotpotQA records and fine-tuned checkpoints are not, limiting external reproduction.

### Source excerpt

arXiv:2611.11965v1 Announce Type: new Abstract: Evidence-sufficiency benchmarks ask whether a model can tell a complete passage set from one whose reasoning chain is broken. When broken sets are made by editing text, near-perfect sufficiency scores--and even positive crossed-control scores--need not demonstrate multi-hop reasoning. We present ChainCheck, an evaluation-validity audit that pairs validator-certified counterfactual cells with artifact-only baselines. In an exploratory crossed control on 2,214 HotpotQA pairs, a language-model scorer that never reads the chain reaches a nominal AUC of 0.997, and a one-string match reaches the maximum of the crossed chain-sensitivity statistic CS4 (0.500), while the 11 evaluated systems score from 0.308 to 0.484; the two statistics are different metrics. We therefore read positive CS4 as operational bridge-consistency sensitivity, not as semantic multi-hop reasoning. Every system is also sensitive to one extra edit, and the edits are detectable according to model-based naturalness ratings. On an independent HotpotQA set of 2,276 pairs, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate: 5 of 6 trained readouts were edit-dominated, and the reference judge's selectivity interval [-0.007, +0.042] includes zero. Judge scores also depended on batch layout. The MuSiQue and 2Wiki records are public; the HotpotQA records and the fine-tuned checkpoints are not, so external reproduction is partial.

## CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds

DevFeed: [CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds](<https://devfeed.tech/articles/cleanscore-black-box-benchmark-audits-with-negative-controls-and-sensitivity-bounds-123210.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.22183>)

Author: Jeffery Opoku, David Banahene

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>)

Tags: [about](<https://devfeed.tech/tags/about.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [arc](<https://devfeed.tech/tags/arc.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [audit](<https://devfeed.tech/tags/audit.md>), [audits](<https://devfeed.tech/tags/audits.md>), [bank](<https://devfeed.tech/tags/bank.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [black](<https://devfeed.tech/tags/black.md>), [box](<https://devfeed.tech/tags/box.md>), [budget](<https://devfeed.tech/tags/budget.md>), [can](<https://devfeed.tech/tags/can.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [conclusion](<https://devfeed.tech/tags/conclusion.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [effect](<https://devfeed.tech/tags/effect.md>), [estimator](<https://devfeed.tech/tags/estimator.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [stat-me](<https://devfeed.tech/tags/stat-me.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

CleanScore estimates whether benchmark performance has a public-wording advantage using only correct-or-incorrect scores. It compares each public question with two independently written fresh forms and reports an interval, using a private control bank and transport radius to account for ordinary differences between forms. In a registered audit of five open models on GSM8K and ARC-Challenge, it found no exposure-consistent advantage; declared reruns preserved that result. The intervals remained uncertain: four of ten model-benchmark pairs had upper bounds around five points or less, and none exceeded 21 points. The registered claim rule was less sensitive, flagging a planted ten-point advantage in only 5%-11% of audits. Positive controls also show a key limitation: leaked items improved scores on unseen paraphrases, with an estimated 76% of the effect invisible to a paraphrase audit. Thus, a null wording advantage does not rule out contamination.

### Source excerpt

arXiv:2609.22183v2 Announce Type: replace Abstract: Public benchmark scores reflect skill, prior exposure to the questions, or both, and for most models the training data are unknown. CleanScore is a black-box audit: the estimator needs only a correct-or-incorrect score for each form, not training data or weights. Each benchmark question becomes a parent item with one public form and two independently written fresh forms that keep its numbers, facts and answer. The audit reports an interval for the public-form advantage, not a verdict, and a private control bank with an explicit transport radius provides a proxy for the ordinary form gap that a rewrite alone can produce. A registered controlled-exposure experiment, a preliminary validation, detects planted exposure in one of two model families. A registered audit of five open models on 200 GSM8K and 200 ARC-Challenge items (ARC with stem-only rewrites, scored by option log probabilities) finds no exposure-consistent advantage, and two reruns declared before they were run (GSM8K at a longer token budget, ARC scored from generated letters) leave that conclusion unchanged. For four of the ten model-benchmark pairs the upper end of the interval is about five points or less, and for none is it above 21 points. The registered claim rule is much less sensitive than the primary interval: it flags a planted ten-point advantage in 5% to 11% of audits. Registered positive controls show what a null can mean. In four models, each trained under four seeds, items leaked into fine-tuning gained 25 points on average over unleaked items on the leaked wording and 19 points on paraphrases the model never saw, so a substantial share of the effect, estimated at 76% (95% interval 50% to 113%), is invisible to a paraphrase audit. A null on surface form says much less about contamination than the name suggests.

## SuiChat-CN: Benchmarking Contextual Suicide Risk Assessment in Chinese Group Chats

DevFeed: [SuiChat-CN: Benchmarking Contextual Suicide Risk Assessment in Chinese Group Chats](<https://devfeed.tech/articles/suichat-cn-benchmarking-contextual-suicide-risk-assessment-in-chinese-group-chats-124429.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.27911>)

Author: Xiangyu Wang, Zhiwei Yu, Chengze Du, Dingchang Wang, Yuhan Ye, Fangyu Zheng

Published: 2026-11-09T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Context-aware AI](<https://devfeed.tech/topics/context-aware-ai.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>)

Tags: [assessment](<https://devfeed.tech/tags/assessment.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [detection](<https://devfeed.tech/tags/detection.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [group-chats](<https://devfeed.tech/tags/group-chats.md>), [risk](<https://devfeed.tech/tags/risk.md>), [risk-assessment](<https://devfeed.tech/tags/risk-assessment.md>), [suicide](<https://devfeed.tech/tags/suicide.md>)

### AI overview

SuiChat-CN benchmarks contextual suicide risk assessment in Chinese Telegram group chats, where fragmented, multi-party messages can make isolated posts insufficient. It comprises 13,312 conversational segments from 1,406 users, covering 258,228 messages. The authors form segments using signal-word extraction and bidirectional context expansion, then annotate user risk levels through an expert-validated, LLM-assisted process. Experiments with pretrained language models and more than 40 LLMs find context essential to reliable assessment; fine-tuning and partial-context evaluation also expose challenges for early detection. Because of ethical and sensitivity concerns, the dataset is not public and is available upon reasonable request to accredited mental health and suicide-prevention research institutions.

### Source excerpt

arXiv:2605.27911v2 Announce Type: replace Abstract: Suicide is a critical global public health challenge, causing approximately 720,000 deaths each year and calling for timely, effective prevention strategies. Existing computational studies primarily focus on post-based social media platforms such as Twitter and Weibo, leaving instant messaging environments such as Telegram underexplored. Yet group chats pose distinct challenges: messages are short, fragmented, multi-party, and often rely on implicit or culturally specific expressions, making isolated post-level analysis insufficient. We introduce SuiChat-CN, a Chinese group-chat benchmark for contextual suicide risk assessment. We collect public Telegram group-chat data, construct coherent conversational segments through signal-word extraction and bidirectional context expansion, and annotate user risk levels with an expert-validated, LLM-assisted paradigm. SuiChat-CN contains 13,312 contextual segments from 1,406 users, covering 258,228 raw chat messages. Extensive experiments with PLMs and more than 40 LLMs demonstrate that contextual information is essential for reliable risk assessment, while fine-tuning and partial-context evaluation further reveal the challenges of early detection in multi-party conversations. Due to ethical and sensitivity concerns, the dataset is not publicly released but will be shared with accredited mental health and suicide-prevention research institutions upon reasonable request.

## Fifty-Two Shades of Pig: What the UCR Archive Really Asks Classifiers to Do

DevFeed: [Fifty-Two Shades of Pig: What the UCR Archive Really Asks Classifiers to Do](<https://devfeed.tech/articles/fifty-two-shades-of-pig-what-the-ucr-archive-really-asks-classifiers-to-do-123463.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.09843>)

Author: Omar Swelam, Jaris K\\"uken, Shi Bin Hoo, Frank Hutter, Lennart Purucker

Published: 2026-11-09T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Computer science](<https://devfeed.tech/topics/computer-science.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [archive](<https://devfeed.tech/tags/archive.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [audit](<https://devfeed.tech/tags/audit.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [classification](<https://devfeed.tech/tags/classification.md>), [classifiers](<https://devfeed.tech/tags/classifiers.md>), [critical](<https://devfeed.tech/tags/critical.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [https](<https://devfeed.tech/tags/https.md>), [original](<https://devfeed.tech/tags/original.md>), [outlier](<https://devfeed.tech/tags/outlier.md>), [problems](<https://devfeed.tech/tags/problems.md>), [question](<https://devfeed.tech/tags/question.md>), [recordings](<https://devfeed.tech/tags/recordings.md>), [sequences](<https://devfeed.tech/tags/sequences.md>)

### AI overview

The paper argues that scores across the UCR Time Series Classification Archive may not reflect performance on deployment-relevant classification tasks. It cites PigAirwayPressure, originally recorded to study whether airway pressure could reveal hemorrhage, but used in the archive to identify which of 52 pigs produced each trace. The authors report that some datasets are repurposed, toy setups, or have flawed test splits; they also question whether ordered non-temporal sequences represent real TSC problems and note potential downsides of decomposing multivariate problems. They call for more careful interpretation of archive-wide scores and greater attention to deployment-relevant evaluation.

### Source excerpt

arXiv:2611.09843v1 Announce Type: new Abstract: The pigs were bleeding. That was the point of the experiment: could airway pressure reveal hemorrhage before it became critical? In the UCR Time Series Classification Archive, the de facto benchmark for Time Series Classification (TSC), the same recordings ask a different question: "which of 52 pigs produced this trace?". The dataset PigAirwayPressure is not an outlier. Although the archive is the standard benchmark, whether its tasks resemble the problems TSC is meant to solve has rarely been examined. We find that several datasets are repurposed from their original task, built from toy setups, or have flawed test splits. We further question how representative the archive's ordered non-temporal sequences (e.g. spectra, shape outlines) are of real TSC, and the potential downsides from the decomposition of multivariate problems. Our findings call for more careful interpretation of archive-wide scores and greater attention to deployment-relevant evaluation. The audit is at https://github.com/automl/ucr-provenance-audit.

## Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks

DevFeed: [Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks](<https://devfeed.tech/articles/follow-the-policy-fail-the-benchmark-auditing-and-revising-policy-based-decision-benchmarks-120805.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.07042>)

Author: Yanru Xiao, Yifei Cai

Published: 2026-11-06T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [auditing](<https://devfeed.tech/tags/auditing.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cases](<https://devfeed.tech/tags/cases.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [errors](<https://devfeed.tech/tags/errors.md>), [execution](<https://devfeed.tech/tags/execution.md>), [exemption](<https://devfeed.tech/tags/exemption.md>), [financial](<https://devfeed.tech/tags/financial.md>), [gap](<https://devfeed.tech/tags/gap.md>), [impact](<https://devfeed.tech/tags/impact.md>), [intermediate](<https://devfeed.tech/tags/intermediate.md>), [model](<https://devfeed.tech/tags/model.md>), [models](<https://devfeed.tech/tags/models.md>), [operational](<https://devfeed.tech/tags/operational.md>), [ownership](<https://devfeed.tech/tags/ownership.md>), [points](<https://devfeed.tech/tags/points.md>), [policy](<https://devfeed.tech/tags/policy.md>), [program](<https://devfeed.tech/tags/program.md>), [public](<https://devfeed.tech/tags/public.md>), [required](<https://devfeed.tech/tags/required.md>)

### AI overview

An audit of nine public policy-based decision benchmarks found errors that can mis-score policy-following systems: unsupported targets, faulty references, and evaluators that reject acceptable answers or credit incomplete decisions. Rescoring unchanged model responses on 96 LegalBench SARA numeric cases raised acceptance by more than 80 percentage points for each of three models. In RuleArena Tax, restoring a required zero floor changed 27 of 300 references and narrowed an observed model gap. A controlled federal-exemption task showed that errors in supplied intermediate answers can affect final decisions. BAR (Benchmark Audit and Revision) provides versioned audit evidence and revisions for reevaluation.

### Source excerpt

arXiv:2611.07042v1 Announce Type: cross Abstract: Policy-based decision benchmarks evaluate whether systems apply written rules to financial and operational cases. Yet a rule-following response can be marked wrong when the benchmark misstates the required decision. We audit nine public policy-based decision benchmarks through source tracing, program execution, and controlled checks. We find unsupported targets, reference errors that survive replay or propagate as solved steps, and evaluators that reject acceptable answers or credit incomplete decisions. To measure the scoring impact, we rescore unchanged model responses. On the 96 LegalBench SARA numeric cases, correcting the scorer raises acceptance by more than 80 percentage points for each of three models, without changing references or responses. In RuleArena Tax, restoring a required zero floor changes 27 of 300 references and narrows an observed model gap. A controlled federal-exemption task with explicit ownership and eligibility assumptions also shows that errors in supplied intermediate answers can affect final decisions. We present BAR (Benchmark Audit and Revision), a versioned collection of audit evidence and revisions for reevaluation.

## A Trust Layer for Agent Evaluation

DevFeed: [A Trust Layer for Agent Evaluation](<https://devfeed.tech/articles/a-trust-layer-for-agent-evaluation-120100.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.07274>)

Author: Mohammadreza Sediqin, Shivali Dalmia, Srinivasa Karthikeya Reddy Kovvuri, Abhishek Mukherji

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [trust](<https://devfeed.tech/topics/trust.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-evaluation](<https://devfeed.tech/tags/agent-evaluation.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [trust](<https://devfeed.tech/tags/trust.md>)

### AI overview

The Trust Layer adds four post-hoc checks beside an agent's benchmark score: support from the benchmark's grading logic, traceable computation for passing answers, truthful completion claims, and stability across repeated runs. The first three inspect saved artifacts; the stability check reruns the agent. Model judgments label evidence by majority vote, while deterministic rules produce verdicts without changing scores. In five configurations tested on 108 Agents' Last Exam tasks, every model had passes without traceable computation, confirmed false completion claims, and instability: 18-46% of tasks changed score band across five runs. Only 22.6% of 84 recorded passes cleared all four checks (95% CI 15.0-32.6).

### Source excerpt

arXiv:2610.07274v2 Announce Type: replace Abstract: Deterministic benchmark scores show that an agent received credit, but not whether that credit was earned, reported honestly, or would hold on a second run. We introduce a Trust Layer for Agent Evaluation, an additive post-hoc framework that reports, beside each recorded score, whether it should be believed. It verifies four properties: whether the result is supported by the benchmark's own grading logic, whether a passing answer was earned through traceable computation, whether the agent's completion claim matches what occurred, and whether the result is stable under repeated execution. The first three use only saved artifacts; the fourth re-runs the agent. Model judgments only label evidence under majority voting; all verdicts follow deterministic rules and never modify the recorded score. Applied to five agent configurations on 108 tasks from Agents' Last Exam, every model shows passing runs with no traceable computation (at rates varying tenfold), confirmed false completion claims, and unstable results: 18-46% of tasks do not stay in one score band over five runs. Only 22.6% of recorded passes clear all four checks (95% CI 15.0-32.6, n=84). Measuring what an agent can do and verifying that it did it are different problems, and current benchmarks address only the first.

## Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test

DevFeed: [Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test](<https://devfeed.tech/articles/are-llm-safety-judges-policy-invariant-a-three-principle-stress-test-120017.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2605.06161>)

Author: Shihao Weng, Yang Feng, Xiaofei Xie

Published: 2026-11-06T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [responsible-ai](<https://devfeed.tech/topics/responsible-ai.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-safety](<https://devfeed.tech/tags/agent-safety.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [failure](<https://devfeed.tech/tags/failure.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [invariance](<https://devfeed.tech/tags/invariance.md>), [invariant](<https://devfeed.tech/tags/invariant.md>), [jitter](<https://devfeed.tech/tags/jitter.md>), [leaderboards](<https://devfeed.tech/tags/leaderboards.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [llm-safety](<https://devfeed.tech/tags/llm-safety.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [policy](<https://devfeed.tech/tags/policy.md>), [principles](<https://devfeed.tech/tags/principles.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [safety](<https://devfeed.tech/tags/safety.md>), [stress](<https://devfeed.tech/tags/stress.md>)

### AI overview

The study finds that LLM safety judges can react similarly to meaningful changes in policy strictness and to equivalent rewrites that preserve policy meaning. Testing four judges on trajectories from ASSEBench and R-Judge, the authors report that content-preserving rewrites flipped up to 9.1% of verdicts beyond baseline jitter; 18-43% of observed flips under these rewrites affected unambiguous cases. This suggests safety scores can reflect prompt wording as well as agent behavior. The authors propose three checks: invariance to equivalent rubric rewrites, appropriate response to threshold shifts, and instability concentrated on ambiguous cases. They introduce a Policy Invariance Score and Judge Card reporting protocol, reporting an order-of-magnitude spread in reliability not visible from accuracy alone. The abstract says findings persisted with a frontier-scale judge, a third benchmark, and a second base policy.

### Source excerpt

arXiv:2605.06161v2 Announce Type: replace Abstract: LLM-as-a-Judge pipelines have become the de facto evaluator for agent safety, yet existing benchmarks treat their verdicts as ground-truth proxies without checking whether the verdicts depend on the agent's behavior or merely on how the evaluation policy happens to be worded. We argue that any trustworthy safety judge must satisfy a basic property we call policy invariance, and we operationalize it as three testable principles: rubric-semantics invariance under certified-equivalent rewrites, rubric-threshold invariance under intentional strict-to-lenient shifts, and ambiguity-aware calibration so that verdict instability concentrates on genuinely ambiguous cases. Instantiating these principles as a stress-test protocol with four agent-class judges on trajectories drawn from ASSEBench and R-Judge, we surface a previously unmeasured failure mode: today's judges respond to meaningful normative shifts and to meaningless structural rewrites with comparable strength, and cannot tell the two apart. Content-preserving policy rewrites flip up to 9.1% of verdicts above baseline jitter, and 18-43% of all observed flips occur on unambiguous cases under such rewrites, so safety scores produced by these judges partly conflate what the agent did with how the evaluator was prompted. These findings persist with a frontier-scale judge, a third benchmark, and a second base policy. Beyond the diagnosis, we contribute the Policy Invariance Score and the Judge Card reporting protocol, which expose an order-of-magnitude spread in judge reliability that is invisible to accuracy-only leaderboards. We release the protocol and code so that future agent-safety benchmarks can audit their own evaluators rather than trust them by default: https://github.com/wengshihao/policy-invariance-judge

## Silent Instrument Errors Reorder Agent Leaderboards

DevFeed: [Silent Instrument Errors Reorder Agent Leaderboards](<https://devfeed.tech/articles/silent-instrument-errors-reorder-agent-leaderboards-116191.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.04399>)

Author: Yangze Liu (Shandong University)

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Error Handling](<https://devfeed.tech/topics/error-handling.md>), [ML model versioning](<https://devfeed.tech/topics/ml-model-versioning.md>)

Tags: [agent-evaluation](<https://devfeed.tech/tags/agent-evaluation.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [errors](<https://devfeed.tech/tags/errors.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [leaderboards](<https://devfeed.tech/tags/leaderboards.md>)

### AI overview

An audit of the HAL and tau2-bench leaderboards finds that silent evaluation errors can distort agent scores and rankings. In one case, a general-purpose agent appeared 13 points behind on database tasks because only about half its episodes were graded; applying the benchmark's own rule to the ungraded episodes closed the pooled gap without rerunning the agent. Other reported problems include double-counted nested model calls in cost totals, grader-written actions logged as agent actions, and evaluator-version changes that alter results. A mode check that misread four files silently reduced their scores by 59-82 points. The authors recommend reporting the graded fraction, calculating cost from root spans, tagging action provenance, and pinning evaluator versions while publishing re-grade differences.

### Source excerpt

arXiv:2611.04399v1 Announce Type: cross Abstract: On a public agent leaderboard, a general-purpose agent trails a scaffold built for an airline customer-service benchmark by 13 points on the tasks that touch its database, and a reader could conclude that generality costs accuracy. The gap comes from the grader. The general agent often ends its conversations in a way that never triggers grading, so only about half of its episodes were scored and the rest were stored as failures. Grading the ungraded half under the benchmark's own rule closes the pooled gap, without rerunning any agent. We audit two carefully maintained public leaderboards, HAL and tau2-bench, from their published artifacts alone and find three more instrument errors. A cost column counts nested model calls twice, unevenly across rows. Archived trajectories record grader-written actions as agent actions. Re-grading saved trajectories under a newer evaluator is itself a new measurement: a correct and announced fix reorders a leaderboard, and a mode check that misreads four files collapses their scores by 59 to 82 points while printing no error. Each produced a plausible number, and the HAL maintainers confirmed their three within five weeks. Together the cases show that the infrastructure of agent evaluation, meaning the code that grades, logs, replays and versions an evaluation, is part of the measurement and fails silently, so scores and orderings can move with no visible sign. These failure modes follow from how agent evaluation pipelines are built, and those pipelines change as fast as the agents they measure, so we expect more of them elsewhere. We propose cheap checks that expose the instrument: report the graded fraction, aggregate cost from root spans, tag the provenance of every recorded action, and pin evaluator versions with a published re-grade diff.

## Widening Harm Evaluation Coverage with a Judge that Reads the World

DevFeed: [Widening Harm Evaluation Coverage with a Judge that Reads the World](<https://devfeed.tech/articles/widening-harm-evaluation-coverage-with-a-judge-that-reads-the-world-115337.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.02624>)

Author: Prabhpreet Singh Setia, Bhavesh Neekhra, Debayan Gupta

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [least privilege](<https://devfeed.tech/topics/least-privilege.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [llm](<https://devfeed.tech/tags/llm.md>), [osworld](<https://devfeed.tech/tags/osworld.md>), [robust](<https://devfeed.tech/tags/robust.md>), [safety](<https://devfeed.tech/tags/safety.md>), [state](<https://devfeed.tech/tags/state.md>), [taxonomy](<https://devfeed.tech/tags/taxonomy.md>), [union](<https://devfeed.tech/tags/union.md>)

### AI overview

The study evaluates an LLM harm judge with read-only access to environment state alongside a judge that inspects an agent's trajectory. Across two DecodingTrust-Arena environments, combining the two channel-specific verdicts recalled 94% of confirmed harms, versus 53% for trajectory-only judging; on OS-Harm, the union recalled 77%, versus 67%. A single judge observing both channels did not improve performance and produced more false negatives than the state-only judge. The channels also missed different episodes. The abstract's results suggest that combining separate trajectory and state-only judges can broaden harm detection in these tested settings.

### Source excerpt

arXiv:2611.02624v1 Announce Type: new Abstract: An agent acting in a stateful environment can leave evidence of harm in two places: the record of what it did and said, or the world it changed. Safety evaluations for discovering harm examine one or the other in limited ways, using either an LLM judge over the trajectory or deterministic checks over the final state. Although LLM-based verifiers that inspect the environment exist, they have been used to verify task completion, not to detect and confirm harm in a safety setting. We address this gap with an LLM that has read-only access to the environment state and is given a harm taxonomy. We evaluate this approach in three environments: two from DecodingTrust-Arena and one from OSWorld. We hold the judge model and harm taxonomy fixed, varying only what the judge observes: the trajectory, read-only tools for inspecting the environment, or both channels together. In the two DecodingTrust-Arena environments, combining the trajectory and state-only judges recalls 94 percent of confirmed harms, compared with 53 percent for the trajectory judge alone. On OS-Harm, the union recalls 77 percent, compared with 67 percent. A single judge given both channels does not perform better and produces more false negatives than the state-only judge. We also record disagreements and find that the two channels miss different episodes. These results suggest that combining verdicts from two channel-specific judges is a robust and flexible approach to detecting and verifying harm in agentic settings.

## Geometry of Confabulation and Refusal in Small Language Models

DevFeed: [Geometry of Confabulation and Refusal in Small Language Models](<https://devfeed.tech/articles/geometry-of-confabulation-and-refusal-in-small-language-models-114516.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.01272>)

Author: Vladimir Kazei

Published: 2026-11-03T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hallucination detection](<https://devfeed.tech/topics/hallucination-detection.md>), [qwen](<https://devfeed.tech/topics/qwen.md>), [gemma](<https://devfeed.tech/topics/gemma.md>), [llama](<https://devfeed.tech/topics/llama.md>), [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>)

Tags: [autoencoder](<https://devfeed.tech/tags/autoencoder.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [linear](<https://devfeed.tech/tags/linear.md>), [llama](<https://devfeed.tech/tags/llama.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [models](<https://devfeed.tech/tags/models.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [optimized](<https://devfeed.tech/tags/optimized.md>), [portable](<https://devfeed.tech/tags/portable.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [signal](<https://devfeed.tech/tags/signal.md>), [small-language-models](<https://devfeed.tech/tags/small-language-models.md>), [speech](<https://devfeed.tech/tags/speech.md>), [state](<https://devfeed.tech/tags/state.md>), [vector](<https://devfeed.tech/tags/vector.md>)

### AI overview

ConfabQA, a 784-item benchmark designed to separate lack of knowledge from knowledge-cutoff effects, shows that hidden states before generation can help distinguish refusal from confabulation, with results varying by model. Probes improve over prompt-feature baselines on Qwen3-1.7B and Gemma 2 2B; the ConfabQA margin on Llama 3.2 3B is within fold-to-fold spread, though refusal probes on PopQA and TriviaQA improve by 18-25 percentage points. On Qwen, pushes along three substantially different candidate directions often change confabulations into refusal-style openings, but forcing the opening phrase alone achieves the same one-in-three judged-refusal rate. Opposite pushes mostly turn refusals into confabulations while leaving originally correct answers intact, suggesting these interventions affect the opening speech act rather than underlying knowledge. The abstract reports no concentrated confabulation feature or token signature on Qwen, and says correctness signal transfers across datasets only for Llama 3.2 3B.

### Source excerpt

arXiv:2611.01272v1 Announce Type: new Abstract: Faced with an unanswerable question, small language models can either refuse or confabulate: confidently produce a wrong answer. Can the hidden state before the first generated token provide a clue? Can we flip confabulations into refusals? I introduce ConfabQA, a 784-item benchmark whose obscure-pre-cutoff category breaks the typical correctness/knowledge-cutoff confound and map the geometry of the hidden state with linear probes, a sparse autoencoder (SAE), and direct optimization of intervention directions. On Qwen3-1.7B and Gemma 2 2B the probe separates upcoming refusals from confabulations well beyond prompt features (+7.4/+16.3 pp); on Llama 3.2 3B the ConfabQA margin is within fold-to-fold spread, but a direct refusal probe on PopQA and TriviaQA recovers +18-25 pp. On Qwen I closely inspect three candidate directions for the refusal vector: probe-recovered, Qwen-Scope SAE feature, and directly optimized. Although the three directions are pairwise 50-81 degrees apart, a push at or below the state's own norm along any of them flips most confabulations into refusal openings dominated by the "As..." family. Forcing the opener without touching the hidden state yields the same one-in-three judged-refusal rate as any push. The opposite push collapses refusals mostly into confabulations, yet leaves originally correct answers intact, so the direction gates the opening speech act rather than the knowledge behind it. Confabulations do not resolve into a concentrated feature in a dictionary or a token signature on Qwen3-1.7B. Scoring every probe on the margin over the strongest of four prompt-feature baselines extracts the hidden state's own contribution. Correctness signal is model-dependent and portable only for Llama 3.2 3B: 81% of the probe's margin over test majority survives transfer between ConfabQA, PopQA, and TriviaQA. Benchmark, judge, and code are released.

## RubricCraft: Where Solutions Diverge, Criteria Emerge

DevFeed: [RubricCraft: Where Solutions Diverge, Criteria Emerge](<https://devfeed.tech/articles/rubriccraft-where-solutions-diverge-criteria-emerge-99887.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.31707>)

Author: Hosein Hasanbeig, Priyanshu Gupta, Ashish Tiwari, Vu Le, Sumit Gulwani

Published: 2026-10-26T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [audit](<https://devfeed.tech/topics/audit.md>), [Parallel reasoning](<https://devfeed.tech/topics/parallel-reasoning.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [audit](<https://devfeed.tech/tags/audit.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [candidate](<https://devfeed.tech/tags/candidate.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [language](<https://devfeed.tech/tags/language.md>), [model](<https://devfeed.tech/tags/model.md>), [reference](<https://devfeed.tech/tags/reference.md>), [repair](<https://devfeed.tech/tags/repair.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [solution](<https://devfeed.tech/tags/solution.md>), [specifications](<https://devfeed.tech/tags/specifications.md>)

### AI overview

RubricCraft builds natural-language evaluation rubrics by comparing independently sampled solutions, then revises criteria that miss errors or reject valid alternatives. This is intended to capture partial correctness without treating one solution as a complete reference. On SpreadsheetBench v2, compared with a prompt-only rubric written and applied by the same model, RubricCraft passed fewer incorrect outputs (29.5% versus 42.6%), rejected none accepted by the benchmark checker, and ranked outputs by correctness more often, while separating models as often. Repair based on one attempt did not change grading; attempts from three model families prompted repairs on different tasks, and an evidence audit judged most repairs improvements, including many on tasks where the reference solution was wrong.

### Source excerpt

arXiv:2610.31707v1 Announce Type: cross Abstract: Evaluating language agents on open-ended, artifact-producing tasks is difficult: model-based graders can be inconsistent, while exhaustive programmatic checks are often impractical. Textual rubrics make evaluation criteria explicit, but writing them requires domain expertise. Rubrics generated from task descriptions alone can miss requirements and failure modes that become apparent in attempted solutions. We introduce RubricCraft, which treats rubric construction as the generation of natural-language specifications guided by candidate solutions. Given a task description and independently sampled solver trajectories, RubricCraft compares how each attempt satisfies or violates the task requirements and uses these differences to construct rubric criteria. It applies the criteria to the collected solutions and revises checks that overlook errors or penalize valid alternatives. The resulting rubrics capture partial correctness without requiring any single solution to serve as a complete reference. On SpreadsheetBench v2, against a prompt-only rubric written and applied by the same model, RubricCraft rubrics pass fewer wrong outputs (29.5% against 42.6%), fail none of the outputs the benchmark's own checker accepts, and rank outputs by correctness more often, while separating models as often. Repair on a single attempt leaves grading unchanged, but repair driven by attempts from three model families acts on different tasks, and an evidence audit judges most of those repairs improvements, many on tasks whose reference solution is itself wrong.

## What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation

DevFeed: [What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation](<https://devfeed.tech/articles/what-the-trace-shows-and-what-the-label-says-auditing-behavioral-detectors-in-agent-evaluation-96642.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.30491>)

Author: Idunnuoluwa A. Adeniji, Shilpika, Michael E. Papka

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [human review](<https://devfeed.tech/topics/human-review.md>)

Tags: [agent-evaluation](<https://devfeed.tech/tags/agent-evaluation.md>), [auditing](<https://devfeed.tech/tags/auditing.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [execution](<https://devfeed.tech/tags/execution.md>), [measurement](<https://devfeed.tech/tags/measurement.md>), [tool-use](<https://devfeed.tech/tags/tool-use.md>), [trace](<https://devfeed.tech/tags/trace.md>), [transparency](<https://devfeed.tech/tags/transparency.md>)

### AI overview

In a GSM8K calculator setting with occasionally corrupted outputs, the study audits three trace-level detectors across six production models and finds that final correctness, visible re-verification, and fault propagation can diverge. In silent tool-output override, a model uses a different, often correct value without calling the calculator again: the bad value stops propagating, yet the verification detector records no re-check. Human review matched automated labels for correctness, fault impact, and fault relevance, but verification labels matched in 67 of 81 cases; the 14 disagreements fit four recurring patterns. Execution budgets also affected correctness in an ablation: one model ranged from 21/30 to 27/30, while another ranged from 27/30 to 29/30. The authors advocate showing trace evidence, detector applicability, execution-limit cutoffs, and human corrections alongside labels; their trace viewer supports inspection. These findings are based on the supplied abstract and excerpt.

### Source excerpt

arXiv:2610.30491v1 Announce Type: cross Abstract: LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are measurements produced by parsers, behavioral detectors, and harness rules, and they may not fully capture what the trace shows. We study this in a tool-use setting built on GSM8K, where agents call a calculator whose outputs are occasionally corrupted, giving clear ground truth for what the tool returned, what the model later used, and what the detectors reported. We audit three trace-level detectors across six production models and 30 problems per model. Final correctness, visible re-verification, and fault propagation can disagree in systematic ways. A key failure mode is silent tool-output override: the calculator returns a corrupted value, but the model later uses a different, often correct, value without calling the tool again, so the bad value stops propagating while a verification detector records no re-check. Human review of 90 traces agrees perfectly with the automated labels for final correctness (67/67), fault impact (19/19), and fault relevance (18/18), but only 67/81 for verification; all 14 disagreements fall into four recurring patterns. Execution limits also change the reported result: in an ablation across two models, one ranges from 21/30 to 27/30 correct as step and token budgets change, while the other stays between 27/30 and 29/30. These findings motivate measurement transparency: an automated label should come with the trace evidence behind it, whether the question the detector asks applied to that trace, whether the run was cut off by an execution limit, and any human corrections. Our trace viewer makes this information inspectable alongside the reported results.

## What Pairwise Evaluation Misses: Reward Scores across Prompts

DevFeed: [What Pairwise Evaluation Misses: Reward Scores across Prompts](<https://devfeed.tech/articles/what-pairwise-evaluation-misses-reward-scores-across-prompts-98163.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.31090>)

Author: Mateusz Modrzejewski, Tomasz Radzikowski

Published: 2026-10-23T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [audit](<https://devfeed.tech/topics/audit.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [reward](<https://devfeed.tech/tags/reward.md>)

### AI overview

Pairwise evaluation can miss prompt-level components in reward scores because comparing responses to the same prompt cancels anything shared across them. The study audits that variation against human ratings using between-prompt variance, sibling-rating regression, and prompt substitutions. On HelpSteer2, six reward models' between-prompt variance shares range from 0.34 to 0.64, while the human share is near zero. Replacing the original prompt with a fixed instruction retains substantial score clustering but weakens agreement with prompt-level human ratings, so clustering alone does not establish human alignment. Prompt variation is not simply noise: subtracting prompt means reduces the helpfulness of globally selected responses, and scores remain sensitive to a wrong prompt even after training shuffles prompt means. The authors recommend evaluating between-prompt variation, wrong-prompt sensitivity, and usefulness for selection separately.

### Source excerpt

arXiv:2610.31090v1 Announce Type: cross Abstract: Reward models score responses for preference optimization, filtering and ranking, and pooling scores across prompts needs them to be comparable. Yet pairwise evaluation compares two responses to the same prompt, so any score component shared by a prompt's responses cancels and goes untested. We study this component by comparing between-prompt variation in model scores with the same variation in human ratings of the same responses. We propose a human-referenced audit, built from a between-prompt variance share, a sibling-rating regression and prompt substitutions, and apply it to eighteen released scorers and six matched training conditions. We find that scores cluster by prompt more than the ratings do: on HelpSteer2, six reward models have between-prompt shares of 0.34 to 0.64 against a human share near zero. Replacing the original prompt with a fixed instruction keeps substantial clustering but weakens agreement with prompt-level human ratings, so clustering alone does not show agreement with humans. Part of that variation is useful: subtracting prompt means lowers the helpfulness of globally selected responses, and scores stay sensitive to a wrong prompt even when training shuffles prompt means. We propose that reward evaluation check between-prompt variation, sensitivity to a wrong prompt and usefulness for selection separately.

## ExeBench-FHD: Addressing Fidelity Issues in Decompiler Evaluation

DevFeed: [ExeBench-FHD: Addressing Fidelity Issues in Decompiler Evaluation](<https://devfeed.tech/articles/exebench-fhd-addressing-fidelity-issues-in-decompiler-evaluation-93721.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.28212>)

Author: Akarsh Ashok, Dinil Mon Divakaran, Vivek Balachandran

Published: 2026-10-21T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Audit-then-score](<https://devfeed.tech/topics/audit-then-score.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Dataset development](<https://devfeed.tech/topics/dataset-development.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-pl](<https://devfeed.tech/tags/cs-pl.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [decompiler](<https://devfeed.tech/tags/decompiler.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [llm](<https://devfeed.tech/tags/llm.md>)

### AI overview

ExeBench-FHD addresses nine fidelity problems in ExeBench that can hinder C-code recovery or weaken test validity. It adds 16 backward-compatible fields, including call-site assembly, data sections, test wrappers, linked-binary disassembly and warning flags. In the original benchmark, 38.9% of functions have at least one of five flags; excluding records with any of four flags, while retaining unused_param records made evaluable by the new wrappers, leaves 63.9% of the corpus. At -O2 across four architectures, the added fields increased GPT 6 Sol's re-execution rate by up to 26.5% for functions reading data sections, 19.1% for mixed integer and floating-point parameters, and 21.0% for unused parameters.

### Source excerpt

arXiv:2610.28212v1 Announce Type: new Abstract: ExeBench is one of the most widely used benchmarks for evaluating LLM-based decompilers. We analyze ExeBench and identify nine issues that affect how faithfully a test measures a decompiler. Three of them prevent LLM decompilers from recovering C code from ExeBench's function. These are ambiguous parameter order when integer and floating-point types are mixed, unused parameters that optimized assembly may drop, and constants stored outside the function's assembly code. Six more affect test validity: non-void functions without a return, placeholder types, static or inline functions removed by the optimizer, functions named main, trivial bodies and truncated definitions. We present ExeBench-FHD, a backward-compatible revision of ExeBench that adds 16 new fields: call-site assembly, data sections, test wrappers, linked-binary disassembly and five warning flags. In ExeBench, 38.9% of functions have at least one of the five flags; dropping records with any of the four warning flags (all except unused_param, which marks functions that the new wrappers make evaluable) keeps 63.9% of the corpus. In an evaluation at -O2 on four architectures, the new fields raise GPT 6 Sol's re-execution rate by up to 26.5%, 19.1% and 21.0% on functions that read data sections, mix parameter types and have unused parameters, respectively.

[Next page](<https://devfeed.tech/topics/audit-then-score.md?cursor=WyIyMDI2LTEwLTIxVDA0OjAwOjAwKzAwOjAwIiwgImY5ZTE3YzUyLTU1YTAtNGQ3Zi1iZTliLTgyYTMyMDVjMGNhZSJd>)