# test

An assessment method that exercises assessment objects under specified conditions to compare actual with expected behavior and evaluate control effectiveness.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## How Standard Bank Group automates AI risk assessment for generative AI

DevFeed: [How Standard Bank Group automates AI risk assessment for generative AI](<https://devfeed.tech/articles/how-standard-bank-group-automates-ai-risk-assessment-for-generative-ai-146575.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/how-standard-bank-group-automates-ai-risk-assessment-for-generative-ai/>)

Author: Tala Qraitem

Published: 2026-11-20T19:28:56Z

Content type: article

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [ai-governance](<https://devfeed.tech/topics/ai-governance.md>), [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [Amazon Bedrock](<https://devfeed.tech/topics/amazon-bedrock.md>), [Amazon Bedrock Guardrails](<https://devfeed.tech/topics/amazon-bedrock-guardrails.md>), [audit](<https://devfeed.tech/topics/audit.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-governance](<https://devfeed.tech/tags/ai-governance.md>), [ai-risk](<https://devfeed.tech/tags/ai-risk.md>), [amazon-bedrock](<https://devfeed.tech/tags/amazon-bedrock.md>), [amazon-bedrock-guardrails](<https://devfeed.tech/tags/amazon-bedrock-guardrails.md>), [amazon-cloudwatch](<https://devfeed.tech/tags/amazon-cloudwatch.md>), [approval-gates](<https://devfeed.tech/tags/approval-gates.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [automated](<https://devfeed.tech/tags/automated.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-professional-services](<https://devfeed.tech/tags/aws-professional-services.md>), [bank](<https://devfeed.tech/tags/bank.md>), [bedrock](<https://devfeed.tech/tags/bedrock.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [llm-as-a-judge](<https://devfeed.tech/tags/llm-as-a-judge.md>), [policies](<https://devfeed.tech/tags/policies.md>), [risk](<https://devfeed.tech/tags/risk.md>), [risk-assessment](<https://devfeed.tech/tags/risk-assessment.md>), [standards](<https://devfeed.tech/tags/standards.md>), [thought-leadership](<https://devfeed.tech/tags/thought-leadership.md>), [use-case](<https://devfeed.tech/tags/use-case.md>)

### AI overview

Standard Bank Group automated evidence gathering for generative AI approval using an assessment deployed in its own AWS account. A pre-assessment agent maps repository files and documentation to about 40 intake questions; a pipeline checks documents and code, tests live applications, and uses Amazon Bedrock LLM-as-a-judge to evaluate more than 300 checks against international standards and bank policies. Narrow yes-or-no judgments feed inspectable scores, with repeated evaluations flagging inconsistencies; findings link to evidence, policy clauses, and mitigations. Reports and a dashboard support reviewers, while risk committees retain approval and regulatory decisions. The assessment is optional during proof of concept and mandatory at build-stage approval. Standard Bank Group reports that six assessments reduced a manual two-week review to about two hours and that remediation improved use-case risk scores by 14%. The assessment provides evidence; it does not certify regulatory compliance.

### Source excerpt

Standard Bank Group automated the evidence behind its generative AI approval gates. Working with AWS Professional Services, the bank built an assessment in its own AWS account that scores each use case against international AI standards and its own policies using LLM-as-a-judge on Amazon Bedrock, giving risk committees consistent, traceable evidence.

## HireWatch: Evaluating LLM Compliance with U.S. Employment Law Under Contextual Pressure

DevFeed: [HireWatch: Evaluating LLM Compliance with U.S. Employment Law Under Contextual Pressure](<https://devfeed.tech/articles/hirewatch-evaluating-llm-compliance-with-u-s-employment-law-under-contextual-pressure-145449.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24970>)

Author: Huy Nghiem, Maria Kostylew, Noam Kolt

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [employment-law](<https://devfeed.tech/tags/employment-law.md>), [llm-compliance](<https://devfeed.tech/tags/llm-compliance.md>)

### AI overview

HireWatch tests whether LLMs select interview questions that comply with U.S. employment law. It uses 200 questions classified under federal statutes, regulations, and agency guidance, and measures unlawful selections across occupations and contexts. In about 100,000 trials with 18 closed and open-weight models, unlawful selections rose sharply under greater contextual pressure. Explicit instructions to comply improved results but did not prevent all violations. The evidence provided is an abstract, so it does not give the specific pressure conditions or model-by-model results.

### Source excerpt

arXiv:2611.24970v1 Announce Type: cross Abstract: The deployment of large language models (LLMs) in hiring pipelines is growing, yet existing evaluations generally overlook their compliance with legal rules. To fill this gap, we introduce HireWatch, an evaluation that assesses whether LLMs comply with U.S. laws that govern the questions that job candidates may be asked. We construct a 200-item bank of lawful and unlawful interview questions, whose legal classification is based on U.S. federal statutes, regulations, and agency guidance. We instruct LLMs to select questions from this bank in preparing for job interviews, across a set of various occupations and contexts, and report the fraction of unlawful questions that LLMs select. Evaluation of 18 closed and open-weight LLMs across ~100,000 trials under different types of contextual pressure reveals that LLMs' legal compliance is brittle. While LLMs generally avoid selecting unlawful questions in low-pressure contexts, legal violations increase sharply under higher pressure. Explicit prompts to comply with law can improve compliance rates, but do not altogether eliminate legal violations. We publicly release our code and data at https://github.com/hnghiem-nlp/hirewatch_repo.

## Who Is Behind the Harness? Verifying Model Identity in Coding Agents through Behavioral Fingerprinting

DevFeed: [Who Is Behind the Harness? Verifying Model Identity in Coding Agents through Behavioral Fingerprinting](<https://devfeed.tech/articles/who-is-behind-the-harness-verifying-model-identity-in-coding-agents-through-behavioral-fingerprinting-142116.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2609.28559>)

Author: Chuyi Wang, Xiaohui Xie, Tongze Wang, Fangchen Luo, Yong Cui

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [actions](<https://devfeed.tech/tags/actions.md>), [agents](<https://devfeed.tech/tags/agents.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [execution](<https://devfeed.tech/tags/execution.md>), [failure-recovery](<https://devfeed.tech/tags/failure-recovery.md>), [harness](<https://devfeed.tech/tags/harness.md>), [identity](<https://devfeed.tech/tags/identity.md>), [lidar](<https://devfeed.tech/tags/lidar.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [model](<https://devfeed.tech/tags/model.md>), [model-weights](<https://devfeed.tech/tags/model-weights.md>), [models](<https://devfeed.tech/tags/models.md>), [opencode](<https://devfeed.tech/tags/opencode.md>), [openhands](<https://devfeed.tech/tags/openhands.md>), [runtime](<https://devfeed.tech/tags/runtime.md>), [security](<https://devfeed.tech/tags/security.md>), [top](<https://devfeed.tech/tags/top.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

LIDAR identifies coding-agent models by actively probing their runtime decisions rather than relying on final text or token distributions. Three probe pairs test post-edit verification, recovery from transient failures, and resolution of specification-test conflicts. It represents execution trajectories with instance-level and distribution-level features, then compares them with clean references using a lightweight probabilistic identifier. The method needs no access to weights, logits, or provider internals, and uses separate references for OpenCode, OpenHands, and mini-swe-agent. Across 24 models from eight families, closed-set Top-1 accuracy ranged from 91.39% to 98.89%, exceeding three baselines in each harness. Including unregistered models from the same families reduced balanced accuracy to 79.58%-80.42%.

### Source excerpt

arXiv:2609.28559v2 Announce Type: replace-cross Abstract: LLMs increasingly operate through coding-agent harnesses that inspect repositories, invoke tools, and modify files. Substituting the model behind such an agent can therefore change security-relevant decisions, including whether it verifies changes or recovers safely from failures. Existing LLM fingerprints largely infer identity from direct text or token distributions. In coding agents, these signals are mediated by system instructions, controller logic, tools, and execution feedback, limiting their transfer. We present LIDAR (LLM Identification from Decisions and Actions at Runtime), an active black-box fingerprinting method for coding-agent execution. Three coding probe pairs expose post-edit verification, transient-failure recovery, and specification--test conflict resolution under controlled changes. LIDAR represents the resulting trajectories with complementary instance-level and distribution-level features and compares them with clean references using a lightweight probabilistic identifier. It requires no access to model weights, logits, or provider internals. We evaluate 24 models from eight families on OpenCode, OpenHands, and mini-swe-agent, fitting separate references for each harness. Closed-set Top-1 accuracy ranges from 91.39\% to 98.89\% across the three harnesses, exceeding all three baselines in each setting. When the audit also includes unregistered models from the same families, balanced accuracy ranges from 79.58\% to 80.42\%. These results show that agent execution behavior provides model-identity evidence beyond final outputs.

## When Figures Disagree with Text: Evaluating Autonomous Visual Evidence Verification in Scientific Papers

DevFeed: [When Figures Disagree with Text: Evaluating Autonomous Visual Evidence Verification in Scientific Papers](<https://devfeed.tech/articles/when-figures-disagree-with-text-evaluating-autonomous-visual-evidence-verification-in-scientific-papers-142783.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24353>)

Author: Guijia Zhang, Hongbo Qiu, Jiajun Zha, Guohao Cai, Hao Zheng, Huihao Jing, Kexu Cheng, Shipeng Luo, JinYang Huang, Shing-Chi Cheung, Changsheng Lu, Haoxuan Che, Harry Yang

Published: 2026-11-19T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [test](<https://devfeed.tech/topics/test.md>), [Image](<https://devfeed.tech/topics/image.md>), [audit](<https://devfeed.tech/topics/audit.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [paper](<https://devfeed.tech/tags/paper.md>), [papers](<https://devfeed.tech/tags/papers.md>), [pdf](<https://devfeed.tech/tags/pdf.md>), [reviews](<https://devfeed.tech/tags/reviews.md>), [study](<https://devfeed.tech/tags/study.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

In a study of 31 manuscripts with hidden, reviewer-documented figure-text inconsistencies, six multimodal models often identified the flawed figure but rarely diagnosed its contradiction: 60.6% of 1,445 open-ended reviews mentioned it, while 3.3% met the rubric. The authors caution that a blind audit found the rubric lenient, so they treat accepted rates as comparative rather than absolute. In a controlled run, naming the figure raised accepted diagnosis from 4.3% to 20.4%; a consistency instruction and coarse visual cues did not. For contradictions visible only in rendered figures, text-only review had no accepted diagnoses, versus 15.8% when rendered pages were reviewed and the figure named. A centrality-based figure-selection baseline reached 4.8%. The findings suggest that deciding what evidence to check remains a major challenge; the tested selection rule did not solve it.

### Source excerpt

arXiv:2611.24353v1 Announce Type: new Abstract: Scientific figures carry evidence for the claims that cite them, so reviewing a paper means checking that a figure agrees with the text, and not only describing what the figure shows. Benchmarks of figure-text verification typically hand the model the figure, the claim, or the knowledge that an inconsistency exists, which leaves untested whether a model can find the figure worth checking in a complete paper, relate a visual element to the claim it conflicts with, and recognize the conflict. We study this autonomous setting on 31 full manuscripts, each carrying a reviewer-documented figure-text inconsistency from PRISMM-Bench that is hidden from the model. Six multimodal models review each PDF under a neutral prompt, and we then add one input at a time to the same manuscripts. Of 1,445 open-ended reviews, 60.6% mention the flawed figure, yet a grading rubric accepts only 3.3% as diagnosing its contradiction; a blind audit finds the rubric lenient, so we read these as rubric-accepted rates and compare conditions rather than absolute values. In a separate controlled run, naming the figure, without stating that it is flawed, raises rubric-accepted diagnosis from 4.3% to 20.4%, while a consistency instruction and coarse visual cues do not. When the contradiction is visible only in the rendered figure, text-only review yields no accepted diagnosis, while review of the rendered pages with the figure named reaches 15.8%. A baseline that lets the model choose which figures to check, ranked by centrality, stays at 4.8%. Deciding what to verify is thus a large part of the difficulty, and the selection rule we tested does not resolve it. Multimodal document understanding should be evaluated on whether a model can propose what to verify, not only answer questions about a figure it is handed.

## Ai2 and Taylor Geospatial partner to make geospatial AI easier to compare, run, and build on

DevFeed: [Ai2 and Taylor Geospatial partner to make geospatial AI easier to compare, run, and build on](<https://devfeed.tech/articles/ai2-and-taylor-geospatial-partner-to-make-geospatial-ai-easier-to-compare-run-and-build-on-141078.md>)

Original publisher: [Read original article](<https://allenai.org/blog/taylor-geospatial-partnership>)

Published: 2026-11-18T08:00:00Z

Content type: news

Language: en

Sources: [Ai2 Blog](<https://devfeed.tech/sources/ai2-blog.md>)

Topics: [geospatial](<https://devfeed.tech/topics/geospatial.md>), [AI Foundation Models](<https://devfeed.tech/topics/ai-foundation-models.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [Digital Public Good](<https://devfeed.tech/topics/digital-public-goods.md>), [test](<https://devfeed.tech/topics/test.md>), [human feedback](<https://devfeed.tech/topics/human-feedback.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [accelerate](<https://devfeed.tech/tags/accelerate.md>), [accessible](<https://devfeed.tech/tags/accessible.md>), [agriculture](<https://devfeed.tech/tags/agriculture.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai2](<https://devfeed.tech/tags/ai2.md>), [applications](<https://devfeed.tech/tags/applications.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [build](<https://devfeed.tech/tags/build.md>), [compare](<https://devfeed.tech/tags/compare.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [geospatial](<https://devfeed.tech/tags/geospatial.md>), [good](<https://devfeed.tech/tags/good.md>), [imagery](<https://devfeed.tech/tags/imagery.md>), [models](<https://devfeed.tech/tags/models.md>), [open](<https://devfeed.tech/tags/open.md>), [partner](<https://devfeed.tech/tags/partner.md>), [public](<https://devfeed.tech/tags/public.md>), [run](<https://devfeed.tech/tags/run.md>), [satellite](<https://devfeed.tech/tags/satellite.md>)

### AI overview

Ai2 and Taylor Geospatial are partnering to provide open resources that help practitioners use and evaluate geospatial AI for conservation, agriculture, and humanitarian work. The effort combines Benchmarks of The World for comparing models, Ai2's OlmoEarth open Earth-observation model family with a no-code platform to fine-tune and run models globally, and an expansion of Fields of The World to improve farm-field boundary precision and add crop type and yield. The partners say datasets, benchmarks, and model weights will be free under open licenses on Source Cooperative and GitHub. Fields of The World already uses a 13-member practitioner advisory group; the partners want similar input to shape shared benchmarks around field needs.

### Source excerpt

Ai2 and Taylor Geospatial are partnering to make geospatial AI more accessible and useful for the public good through open models, shared benchmarks, and improved satellite imagery datasets for applications in conservation, agriculture, and humanitarian work.

## Explore, Map, Remember, Decide: A Benchmark for VLM Readiness in Safety-Critical Scenarios

DevFeed: [Explore, Map, Remember, Decide: A Benchmark for VLM Readiness in Safety-Critical Scenarios](<https://devfeed.tech/articles/explore-map-remember-decide-a-benchmark-for-vlm-readiness-in-safety-critical-scenarios-139800.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2608.08077>)

Author: Gabriele La Malfa, Nitay Alon, Emanuele La Malfa, Reuth Mirsky, Stefan Sarkadi

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Robot Navigation](<https://devfeed.tech/topics/robot-navigation.md>), [test](<https://devfeed.tech/topics/test.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [choice](<https://devfeed.tech/tags/choice.md>), [controllers](<https://devfeed.tech/tags/controllers.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [environment](<https://devfeed.tech/tags/environment.md>), [error](<https://devfeed.tech/tags/error.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [explore](<https://devfeed.tech/tags/explore.md>), [external](<https://devfeed.tech/tags/external.md>), [framework](<https://devfeed.tech/tags/framework.md>), [gathering](<https://devfeed.tech/tags/gathering.md>), [horizon](<https://devfeed.tech/tags/horizon.md>), [limits](<https://devfeed.tech/tags/limits.md>), [map](<https://devfeed.tech/tags/map.md>), [maps](<https://devfeed.tech/tags/maps.md>), [memory](<https://devfeed.tech/tags/memory.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [models](<https://devfeed.tech/tags/models.md>), [safety](<https://devfeed.tech/tags/safety.md>), [vlm](<https://devfeed.tech/tags/vlm.md>)

### AI overview

EMRD evaluates VLM spatial reasoning in four stages--Explore, Map, Remember, and Decide--using 18 metrics across 16 buildings and three environmental stresses. Open-weight models left rooms unvisited in 95% of episodes and detected at most 51% of objects; closed-weight models explored more, though two of three using a notepad stopped prematurely. Without external memory, most models retained fewer observed objects, especially in larger buildings. Starting location constrained exploration, and open-weight models starting in different rooms agreed on the final choice no more often than chance. Room-label swaps often changed that choice, indicating sensitivity to semantic priors. Motion noise raised object-position error by 12-60% for closed-weight models; strong darkness cut map recall by 28-31 percentage points, and dense smoke reduced objects seen by 25-39 points. Among rooms entered, environmental stress did not significantly change the final choice.

### Source excerpt

arXiv:2608.08077v2 Announce Type: replace Abstract: Vision-Language Models (VLMs) are often deployed as embodied controllers for robotics, yet it remains unclear how well they can explore an environment, maintain an internal map, and ground safety-critical decisions in long-horizon sequential trajectories. In this work, we introduce EMRD, an evaluation framework that decouples spatial reasoning into four stages: Explore (what the agent observes), Map (what it records and where), Remember (what it retains), and Decide (whether its final choice is justified), scored across 18 stage-specific metrics. We use EMRD to evaluate eleven open- and closed-weight VLMs across 16 buildings under nominal conditions and three environmental stresses (darkness, smoke, and motion noise), where an embodied VLM explores unknown rooms, maps observed objects, and selects the safest gathering place. Open-weight models leave rooms unvisited in 95% of episodes and detect at most 51% of objects; closed-weight models explore more, though with an external memory (notepad), two of the three stop prematurely. Without external memory, most VLMs keep fewer of the objects they saw, especially in larger buildings. The start limits which rooms are explored, and open-weight models started in different rooms agree on the final room not significantly more often than chance. Moreover, these choices are biased by semantic priors: swapping a room's label is often enough to change the final choice. Finally, motion noise increases object position error (by 12-60% for closed-weight models), strong darkness lowers map recall by 28-31 percentage points, and dense smoke reduces seen objects by 25-39 points, while the choice among the rooms entered does not change significantly.

## Seeing What Still Holds: How Waiting Control Changes Cooperative Recovery

DevFeed: [Seeing What Still Holds: How Waiting Control Changes Cooperative Recovery](<https://devfeed.tech/articles/seeing-what-still-holds-how-waiting-control-changes-cooperative-recovery-140019.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22368>)

Author: Kangjie Huang, Yucheng Xin, Xinchun Wang, Dong Liu, Yan Fan, Shuming Liu

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Robotics](<https://devfeed.tech/topics/robotics.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [cad](<https://devfeed.tech/tags/cad.md>), [completion](<https://devfeed.tech/tags/completion.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-ro](<https://devfeed.tech/tags/cs-ro.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [execution](<https://devfeed.tech/tags/execution.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [reference](<https://devfeed.tech/tags/reference.md>), [reverse](<https://devfeed.tech/tags/reverse.md>), [time](<https://devfeed.tech/tags/time.md>), [tracking](<https://devfeed.tech/tags/tracking.md>), [waiting](<https://devfeed.tech/tags/waiting.md>)

### AI overview

Changing the holding reference during a wait can reverse which visual recovery method performs better. In AuditEmbodied's 960-execution comparison, CAD-feedback and landmark-based recovery were tested from matched post-wait states while executor behavior was held fixed. After a 1-second wait with registered wrist references, protected completions were 6/40 versus 19/40 under registered holding, but 34/40 versus 22/40 under position-only holding. Trajectory analysis points to grasp formation: small target residuals at observation time preceded large tracking errors, with diagnostics and interventions examining persistent tracking constraints. On fresh seeds, gradual position reanchoring raised CAD recovery's long-wait completion by 42.5 percentage points and cut mean complete recovery time by 6.06 seconds versus registered holding. Further evaluations covered pretrained policies, handover, and mass-matched geometry. The results support treating holding reference as an explicit condition when comparing recovery performance.

### Source excerpt

arXiv:2611.22368v1 Announce Type: new Abstract: Changing the holding reference during a fixed-duration wait can reverse the ranking of visual recovery methods. We introduce AuditEmbodied, a paired execution and trajectory evaluation framework that independently varies holding references and waiting duration while keeping recovery executors frozen. A 960-execution factorial on 20 initialization seeds compares CAD-feedback and landmark-based recovery from identical post-wait physical states within each holding and waiting condition. Protected completion requires successful recovery under the declared execution constraints. At a 1 s wait with registered wrist references, their protected completion counts change from 6/40 versus 19/40 under registered holding to 34/40 versus 22/40 under position-only holding. Trajectory analysis localizes the main separation to grasp formation: small observation-time target residuals precede large tracking errors. Joint diagnostics and controlled interventions characterize persistent tracking constraints. On fresh seeds, gradual position reanchoring improves CAD recovery's long-wait completion by 42.5 percentage points and reduces mean complete recovery time by 6.06 s relative to registered holding. Additional evaluations with pretrained policies, handover, and a mass-matched geometry characterize holding-reference sensitivity across executors and physical regimes. These findings establish holding reference as an explicit evaluation condition for comparing visual recovery performance.

## Reading Postgres EXPLAIN ANALYZE when the planner expects 300 rows and gets 15,000

DevFeed: [Reading Postgres EXPLAIN ANALYZE when the planner expects 300 rows and gets 15,000](<https://devfeed.tech/articles/reading-postgres-explain-analyze-when-the-planner-expects-300-rows-and-gets-15-000-136913.md>)

Original publisher: [Read original article](<https://hackernoon.com/reading-postgres-explain-analyze-when-the-planner-expects-300-rows-and-gets-15000?source=rss>)

Author: Anatolii Zhadan

Published: 2026-11-17T06:40:19Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [JOIN](<https://devfeed.tech/topics/join.md>), [groupby](<https://devfeed.tech/topics/groupby.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [amd](<https://devfeed.tech/tags/amd.md>), [amd-epyc](<https://devfeed.tech/tags/amd-epyc.md>), [analyze](<https://devfeed.tech/tags/analyze.md>), [arm](<https://devfeed.tech/tags/arm.md>), [b-tree](<https://devfeed.tech/tags/b-tree.md>), [backend](<https://devfeed.tech/tags/backend.md>), [backend-development](<https://devfeed.tech/tags/backend-development.md>), [batch](<https://devfeed.tech/tags/batch.md>), [batches](<https://devfeed.tech/tags/batches.md>), [cache](<https://devfeed.tech/tags/cache.md>), [clock](<https://devfeed.tech/tags/clock.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cost](<https://devfeed.tech/tags/cost.md>), [count](<https://devfeed.tech/tags/count.md>), [database](<https://devfeed.tech/tags/database.md>), [epyc](<https://devfeed.tech/tags/epyc.md>), [explain](<https://devfeed.tech/tags/explain.md>), [explain-analyze](<https://devfeed.tech/tags/explain-analyze.md>), [node](<https://devfeed.tech/tags/node.md>), [performance](<https://devfeed.tech/tags/performance.md>), [plan](<https://devfeed.tech/tags/plan.md>), [planner](<https://devfeed.tech/tags/planner.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [programming](<https://devfeed.tech/tags/programming.md>), [query-optimization](<https://devfeed.tech/tags/query-optimization.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

Start at the lowest plan node where estimated and actual rows diverge: in the example, PostgreSQL expected 300 Lisbon users but found 15,000, leading to 15,000 index lookups and a sort spill. That Nested Loop was faster in memory (159 ms versus 344 ms for a Hash Join) but slower from disk (2.15 s versus 455 ms), showing why plain queries should be timed warm and cold before comparing plans. Read plans bottom-up, account for per-loop row and time averages by multiplying by loops, and subtract child times to locate work. `Buffers` read counts mean pages requested from the OS, not necessarily disk; `track_io_timing` can show read waits. `Disk:` and multi-batch hashes indicate spills, whose cost depends on the serving machine and memory settings. `EXPLAIN ANALYZE` adds measurement overhead: in the reported tests it slowed `count(*)` 2.4-3.0 times, while `TIMING OFF` retains rows, loops and buffers but still has overhead. Wrap analyzed writes in a transaction and roll them back; their work and sequence increments can persist.

### Source excerpt

A bad plan starts at the lowest node where expected and actual rows split, and every node above it is built on that wrong number.

## ScenarioForge: Resource-Grounded Generation and Execution Validation of Cyber Exercises

DevFeed: [ScenarioForge: Resource-Grounded Generation and Execution Validation of Cyber Exercises](<https://devfeed.tech/articles/scenarioforge-resource-grounded-generation-and-execution-validation-of-cyber-exercises-137719.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21579>)

Author: Jaime C. Acosta, Anantaa Kotal, Aritran Piplai

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [test](<https://devfeed.tech/topics/test.md>), [open-llm-leaderboard](<https://devfeed.tech/topics/open-llm-leaderboard.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-assisted](<https://devfeed.tech/tags/ai-assisted.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [attack](<https://devfeed.tech/tags/attack.md>), [attack-paths](<https://devfeed.tech/tags/attack-paths.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [build](<https://devfeed.tech/tags/build.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [container](<https://devfeed.tech/tags/container.md>), [container-image](<https://devfeed.tech/tags/container-image.md>), [core](<https://devfeed.tech/tags/core.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cyber](<https://devfeed.tech/tags/cyber.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [dependency](<https://devfeed.tech/tags/dependency.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [execution](<https://devfeed.tech/tags/execution.md>), [generation](<https://devfeed.tech/tags/generation.md>), [grounded](<https://devfeed.tech/tags/grounded.md>), [resource](<https://devfeed.tech/tags/resource.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

ScenarioForge converts high-level requirements into resource-grounded cyber exercises, then builds and validates them in CORE. Its dataset contains 226 materialized scenarios spanning 294 Vulhub images and 85 custom challenge-resource generators; all passed construction through deployment, strict runtime validation, and live artifact checks. Across 867 individually verifiable steps, 457 are vulnerability-backed and 410 use generated resources. In a separate 50-prompt benchmark, ScenarioForge paired with either Qwen3.8 or GPT5.6 achieved 50/50 topology/build success and 160/160 intent fields. Direct frontier generation reached 41/50 builds and 147/160 fields only with cumulative structure, vocabulary, policy, and container-image guidance. That benchmark omitted downstream resource generation, dependency and flag sequencing, deployment, and live validation, so it does not compare those stages.

### Source excerpt

arXiv:2611.21579v1 Announce Type: cross Abstract: Cyber-range and capture-the-flag scenario construction remains labor-intensive because authors must coordinate infrastructure, vulnerable services, attack paths, artifacts, and success conditions. LLMs can propose detailed exercises, but plausible descriptions do not guarantee deployable software or verifiable objectives. We present ScenarioForge, which turns high-level requirements into resource-grounded topologies, services, traffic, segmentation, and challenge chains, then instantiates and validates them in the Common Open Research Emulator (CORE). The ScenarioForge Execution Dataset contains 226 materialized scenarios covering 294 Vulhub images and all 85 custom-built challenge-resource generators developed with our AI-assisted scaffolding. All 226 passed construction, preview, sequencing, deployment, strict runtime validation, and live artifact checks; their 867 steps, each an individually verifiable challenge objective, comprise 457 vulnerability-backed and 410 generated-resource-backed instances. In a separate 50-prompt matched benchmark, ScenarioForge+Qwen3.8 and ScenarioForge+GPT5.6 each achieved 50/50 (100%) topology/build success and 160/160 intent fields; direct frontier generation reached 41/50 builds and 147/160 fields only with the full cumulative scaffold: structure, vocabulary, policy, and container-image guidance. The test still omitted ScenarioForge's downstream challenge-resource generation, dependency/flag sequencing, deployment, and live validation. The artifacts support operationally rich, reproducible exercises for training and evaluation.

## A Testing Method for Detector Error Models in Quantum Error Correction Compilers

DevFeed: [A Testing Method for Detector Error Models in Quantum Error Correction Compilers](<https://devfeed.tech/articles/a-testing-method-for-detector-error-models-in-quantum-error-correction-compilers-138002.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20501>)

Author: Junjie Luo, Yasunari Suzuki

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [test](<https://devfeed.tech/topics/test.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [compiler](<https://devfeed.tech/tags/compiler.md>), [compilers](<https://devfeed.tech/tags/compilers.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [decoder](<https://devfeed.tech/tags/decoder.md>), [defect](<https://devfeed.tech/tags/defect.md>), [dem](<https://devfeed.tech/tags/dem.md>), [error-correction](<https://devfeed.tech/tags/error-correction.md>), [fixed](<https://devfeed.tech/tags/fixed.md>), [model-comparison](<https://devfeed.tech/tags/model-comparison.md>), [models](<https://devfeed.tech/tags/models.md>), [noise](<https://devfeed.tech/tags/noise.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [quantum-error-correction](<https://devfeed.tech/tags/quantum-error-correction.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [software](<https://devfeed.tech/tags/software.md>), [testing](<https://devfeed.tech/tags/testing.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

DEMVal checks each detector error model emitted by a quantum error correction compiler against a reconstructed reference whose error rates come from outside the compiler under test. Exact comparison detects rate errors regardless of physical error rate, unlike sampling, which can miss rate-only defects; the abstract says even optimal moment tests miss two such defects. References can be built in three ways, jointly covering the pipeline from source program to emitted model. Testing CUDA-Q QEC, Deltakit and QDK-EC's deq component found six defects, all confirmed and fixed. Sampling could expose only four, and required millions of samples per experiment to expose all four; DEMVal detected all six without sampling.

### Source excerpt

arXiv:2611.20501v1 Announce Type: new Abstract: A quantum error correction (QEC) compiler produces, from a code and a noise model, both the physical circuit to execute and a detector error model (DEM). The DEM lists the probability of each error mechanism and the measurements it flips. Decoders consume the DEM without verification, so a defect that describes the noise incorrectly calibrates the decoder against wrong rates and silently degrades the logical error rate. Existing techniques for testing quantum software judge correctness by crashes or by statistics of sampled outputs. DEM defects that only distort rates lie below what sampling can resolve at the error rates where QEC operates. We present DEMVal, a testing method that validates each model the compiler emits against a reconstructed reference, with rates obtained from outside the compiler under test. It compares the two models exactly, so its sensitivity is independent of the physical error rate. The reference can be built in three ways that jointly cover the pipeline from source program to emitted model. Applied to NVIDIA's CUDA-Q QEC, Riverlane's Deltakit, and the deq component of Microsoft's QDK-EC, DEMVal found six defects, all confirmed by the developers and fixed. Sampling could have exposed only the four defects that drop, merge, or misplace mechanisms, and exposing all four requires millions of samples per experiment. Even analytically optimal moment tests miss the two rate-level defects, while exact model comparison detects all six with no samples. To our knowledge, DEMVal is the first systematic testing method for the error-model semantics of QEC compilers.

## A Systematic Study of Security Enforcement at Android Inter-Component Boundaries

DevFeed: [A Systematic Study of Security Enforcement at Android Inter-Component Boundaries](<https://devfeed.tech/articles/a-systematic-study-of-security-enforcement-at-android-inter-component-boundaries-139244.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.13597>)

Author: Sandiyo Christan A

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [Android Security](<https://devfeed.tech/topics/android-security.md>), [Android](<https://devfeed.tech/topics/android.md>), [out-of-process](<https://devfeed.tech/topics/out-of-process.md>), [Access Control](<https://devfeed.tech/topics/access-control.md>), [test](<https://devfeed.tech/topics/test.md>), [software security audit](<https://devfeed.tech/topics/software-security-audit.md>)

Tags: [android](<https://devfeed.tech/tags/android.md>), [android-17](<https://devfeed.tech/tags/android-17.md>), [aosp](<https://devfeed.tech/tags/aosp.md>), [api](<https://devfeed.tech/tags/api.md>), [authentication-tokens](<https://devfeed.tech/tags/authentication-tokens.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [ipc](<https://devfeed.tech/tags/ipc.md>), [medical](<https://devfeed.tech/tags/medical.md>), [payment](<https://devfeed.tech/tags/payment.md>), [permission](<https://devfeed.tech/tags/permission.md>), [platform-security](<https://devfeed.tech/tags/platform-security.md>), [security](<https://devfeed.tech/tags/security.md>), [security-enforcement](<https://devfeed.tech/tags/security-enforcement.md>), [sensitive-data](<https://devfeed.tech/tags/sensitive-data.md>), [study](<https://devfeed.tech/tags/study.md>), [taxonomy](<https://devfeed.tech/tags/taxonomy.md>)

### AI overview

On Android 17 (API 37), the study reports 97 enforcement gaps among 120 evaluations of IPC boundaries and platform security controls across more than 30 Google and AOSP packages, using a zero-permission attacker model. Results vary by interface: evaluated Activities and BroadcastReceivers had 100% gap rates, while ContentProviders had a 27.6% hold rate; the authors note framework defenses such as setStrict(). All seven evaluated security-intended compatibility changes were disabled or gated at a target SDK level none of the sampled apps used. The study also documents 18 custom URI schemes handling sensitive data and derives a 14-category taxonomy. Findings describe a purposively selected corpus, not all Android applications.

### Source excerpt

arXiv:2610.13597v2 Announce Type: replace Abstract: Android's inter-process communication (IPC) mechanisms (Activities, Services, BroadcastReceivers, ContentProviders, and URI scheme handlers) form the primary interface between applications and between applications and the system. We present a systematic study of security enforcement comprising 120 evaluations (99 inter-component interface evaluations across five interface categories, 10 platform security-control evaluations, and 11 dedicated positive controls) in over 30 Google and AOSP packages on Android 17 (API 37). Using an expected-versus-observed framework under a zero-permission attacker model, we evaluate whether each boundary's enforcement mechanism functions as designed. We find 97 enforcement gaps (80.8%) and 23 enforcement holds (19.2%). From these observations, we derive a 14-category enforcement gap taxonomy organized across manifest, code, platform, and architectural layers. Within our purposively selected corpus, observed enforcement outcomes differ substantially across interface categories: evaluated Activities and BroadcastReceivers exhibit a 100% gap rate, while ContentProviders, which benefit from framework-provided defenses such as setStrict(), achieve a 27.6% holds rate. We find that all seven evaluated platform compatibility changes with security intent are either disabled or gated at a targetSdkVersion that none of the applications in our evaluated corpus targets. We document 18 custom URI scheme registrations handling sensitive data (authentication tokens, payment credentials, medical information) despite the known unverifiability of custom schemes (RFC 8252 Section 8.1). Our study includes 23 positive controls, calibration cases demonstrating that the evaluation procedure distinguishes functioning enforcement from observed enforcement failures.

## Surrogate-Assisted Compact-Model Extraction for SKY130 CMOS at 77 K

DevFeed: [Surrogate-Assisted Compact-Model Extraction for SKY130 CMOS at 77 K](<https://devfeed.tech/articles/surrogate-assisted-compact-model-extraction-for-sky130-cmos-at-77-k-139195.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21594>)

Author: Aaron Wang, Olivia Gzamouranis, Vaibhav Lohia, Hershel Fei Xun Weiner, Olivia Seidel, Richard Cavanaugh, Giuseppe Di Guglielmo, Nhan Tran, Davide Braga

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Neural Network](<https://devfeed.tech/topics/neural-network.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [cmos](<https://devfeed.tech/tags/cmos.md>), [cryogenic](<https://devfeed.tech/tags/cryogenic.md>), [cs-et](<https://devfeed.tech/tags/cs-et.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [error](<https://devfeed.tech/tags/error.md>), [model](<https://devfeed.tech/tags/model.md>), [neural](<https://devfeed.tech/tags/neural.md>), [physics-app-ph](<https://devfeed.tech/tags/physics-app-ph.md>), [physics-ins-det](<https://devfeed.tech/tags/physics-ins-det.md>), [search](<https://devfeed.tech/tags/search.md>), [temperature](<https://devfeed.tech/tags/temperature.md>), [transistor](<https://devfeed.tech/tags/transistor.md>), [transistor-models](<https://devfeed.tech/tags/transistor-models.md>)

### AI overview

A neural surrogate of NGSpice helps fit BSIM4 compact-model parameters to measured SKY130 transistor I-V curves at 77 K. The bounded least-squares workflow uses NGSpice simulations for initial candidates, surrogate finite differences for the local Jacobian, and NGSpice re-simulations to select model cards. Across 18 measured geometries, searches over seven parameters and 43 sensitivity-selected parameters reduced in-sample mean RRMS by 17% and 26%, respectively, versus re-simulations of previously released cards. The 43-parameter search had lower overall error, while the seven-parameter search had lower mean error on high-bias curves; some individual curves were worse than with released cards under either search.

### Source excerpt

arXiv:2611.21594v1 Announce Type: cross Abstract: Cryogenic circuit design requires compact transistor models that reproduce measured low-temperature behavior and run in standard circuit simulators. We fit BSIM4 parameters to measured I-V curves of SkyWater 130 nm (SKY130) transistors at 77 K using a neural surrogate of NGSpice. The surrogate maps compact-model parameters and device geometry to the complete set of output and transfer characteristics. Extraction is a bounded least-squares search: a bank of NGSpice simulations supplies initial candidates, finite differences of the surrogate supply the local Jacobian, and NGSpice re-simulates the resulting cards before selection. On 18 measured geometries, we compare a search over seven card parameters with a search over 43 parameters selected by NGSpice sensitivity. Relative to NGSpice re-simulations of the previously released cards, the two searches reduce the in-sample mean relative root-mean-square error (RRMS) by 17% and 26%, respectively. The 43-parameter search has the lower overall error, but the seven-parameter search has the lower mean on high-bias curves, and some individual curves have higher error than with the released cards under both searches.

## LoopSkill: Evidence-Bound Lifecycle Control and the Safety-Liveness Tradeoff in Coding Agents

DevFeed: [LoopSkill: Evidence-Bound Lifecycle Control and the Safety-Liveness Tradeoff in Coding Agents](<https://devfeed.tech/articles/loopskill-evidence-bound-lifecycle-control-and-the-safety-liveness-tradeoff-in-coding-agents-137959.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18573>)

Author: Zengjian Liu, Tommy He, ELE Zhao, HongLiang Sun

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [control-plane](<https://devfeed.tech/topics/control-plane.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [test](<https://devfeed.tech/topics/test.md>), [long-running](<https://devfeed.tech/topics/long-running.md>)

Tags: [artifacts](<https://devfeed.tech/tags/artifacts.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [independent](<https://devfeed.tech/tags/independent.md>), [lifecycle](<https://devfeed.tech/tags/lifecycle.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [routing](<https://devfeed.tech/tags/routing.md>), [safety](<https://devfeed.tech/tags/safety.md>), [swe-bench](<https://devfeed.tech/tags/swe-bench.md>)

### AI overview

LoopSkill adds evidence-bound controls to coding-agent workflows, including a schema-validated sole writer, durable outbox routing, artifact-bound reports, recovery, independent review, and heartbeat-gated finalization. In a locally frozen v3.3.12 integration tested for 1,800 seconds on a selected SWE-bench Lite subset, blind artifact success was 7/19 for LoopSkill, 11/19 for implementation plus independent review, and 14/20 for an ordinary coding agent; two cells were apparatus null. No scored LoopSkill workflow finalized, illustrating a safety-liveness and fixed-budget cost tradeoff. The 60-cell analysis was exploratory and nonconfirmatory because the original 140-cell plan was administratively truncated after interim outcomes were visible. The results do not establish comparative superiority, long-horizon efficacy, or a false-accept rate.

### Source excerpt

arXiv:2611.18573v1 Announce Type: new Abstract: Coding agents can edit repositories and announce completion, but conversational claims do not prove that artifacts, dispatches, state versions, and authority still agree. LoopSkill is an evidence-bound lifecycle control plane with a schema-validated sole writer, durable outbox routing, artifact-bound reports, recovery, independent review, and heartbeat-gated finalization. We evaluate a locally frozen v3.3.12 integration under a common 1,800-second budget on a selected subset of SWE-bench Lite. The original 140-cell plan was administratively truncated after interim outcomes were visible, so the 60-cell analysis is exploratory and nonconfirmatory. Of 58 scored cells, blind artifact success was 7/19 for LoopSkill, 11/19 for implementation plus independent review, and 14/20 for an ordinary coding agent; two cells were apparatus null. Across 18 complete LoopSkill-versus-review pairs, the descriptive risk difference was -0.222 with exact McNemar p=0.125. No scored LoopSkill workflow finalized. The results expose a safety-liveness and fixed-budget cost tradeoff. They do not establish comparative superiority, long-horizon efficacy, or a false-accept rate.

## Scenario-Based Risk Certificates for Multi-Property Vulnerability Discovery in Parametric MDPs

DevFeed: [Scenario-Based Risk Certificates for Multi-Property Vulnerability Discovery in Parametric MDPs](<https://devfeed.tech/articles/scenario-based-risk-certificates-for-multi-property-vulnerability-discovery-in-parametric-mdps-137624.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20861>)

Author: Zhiming Chi, Andrea Turrini, Yong Li, Lijun Zhang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [test](<https://devfeed.tech/topics/test.md>), [distributed algorithm](<https://devfeed.tech/topics/distributed-algorithm.md>)

Tags: [certificates](<https://devfeed.tech/tags/certificates.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [error](<https://devfeed.tech/tags/error.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [markov](<https://devfeed.tech/tags/markov.md>), [property](<https://devfeed.tech/tags/property.md>), [region](<https://devfeed.tech/tags/region.md>), [requirements](<https://devfeed.tech/tags/requirements.md>), [risk](<https://devfeed.tech/tags/risk.md>), [safety](<https://devfeed.tech/tags/safety.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [space](<https://devfeed.tech/tags/space.md>), [testing](<https://devfeed.tech/tags/testing.md>), [verification](<https://devfeed.tech/tags/verification.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>), [vulnerability-discovery](<https://devfeed.tech/tags/vulnerability-discovery.md>)

### AI overview

The paper presents a two-layer method for discovering parameter settings that may violate multiple quantitative requirements in parametric Markov decision processes (pMDPs). For each requirement, scenario approximation identifies conservative candidate vulnerability regions with PAC guarantees. Random testing then explores parameters under a specified distribution, treating the regions as overlapping; its stopping certificate uses their marginal masses alone. A composition theorem bounds expected non-discovery by the scenario-approximation error plus testing risk. Candidate hits are confirmed by model checking, and discovery budgets rely on independently certified probabilities. Experiments evaluate the certificates using empirical surrogates and independent confidence bounds.

### Source excerpt

arXiv:2611.20861v1 Announce Type: cross Abstract: Safety-critical systems such as network protocols and distributed algorithms depend on environmental parameters whose exact values are uncertain. Parametric Markov decision processes (pMDPs) model such systems by making transition probabilities symbolic functions of the parameters. For a single quantitative requirement, parametric verification asks which parameter valuations satisfy the requirement. In practice, however, engineers often monitor several safety and performance requirements at once, and one valuation may make several requirements vulnerable simultaneously. In this paper we study multi-property vulnerability discovery under requirement-wise semantics: given a pMDP and M quantitative requirements, we construct candidate regions of the parameter space that may contain violations, and study budgets for random testing under a specified parameter distribution. To solve this, we adopt a two-layered approach in which the scenario-approximation layer provides PAC guarantees on identifying conservative vulnerability regions for each requirement while the testing layer treats random parameter exploration as coverage of overlapping regions and yields a stopping certificate computed from the marginal region masses alone; a composition theorem then bounds the expected non-discovery score by the scenario approximation error plus the testing risk from the two layers. For confirmed discoveries, candidate hits are checked by model checking and budgets use independently certified discovery probabilities. Experiment evaluate these certificates using empirical surrogates and independent confidence bounds.

## Revisiting Re-executability: Coverage-Guided Symbolic Feedback to Improve Decompilation

DevFeed: [Revisiting Re-executability: Coverage-Guided Symbolic Feedback to Improve Decompilation](<https://devfeed.tech/articles/revisiting-re-executability-coverage-guided-symbolic-feedback-to-improve-decompilation-137973.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18892>)

Author: Yuxin Hu, Xiaohan Wang, Kevin Leach

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Reverse Engineering](<https://devfeed.tech/topics/reverse-engineering.md>), [Fuzzing/Fuzz testing](<https://devfeed.tech/topics/fuzzing.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [behavior](<https://devfeed.tech/tags/behavior.md>), [branch](<https://devfeed.tech/tags/branch.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [decompilation](<https://devfeed.tech/tags/decompilation.md>), [execution](<https://devfeed.tech/tags/execution.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [fuzzing](<https://devfeed.tech/tags/fuzzing.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-assisted](<https://devfeed.tech/tags/llm-assisted.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [reference](<https://devfeed.tech/tags/reference.md>), [repair](<https://devfeed.tech/tags/repair.md>), [symbolic-execution](<https://devfeed.tech/tags/symbolic-execution.md>)

### AI overview

SymDec uses coverage-guided symbolic execution to find inputs that distinguish an original binary from its recompiled decompilation. It replays those inputs on both programs to confirm behavioral differences, then sends structured feedback to an LLM for repair. Across 960 tasks covering x86-64 and ARM64 binaries at optimization levels O0-O3, symbolic comparison found replay-confirmed differences in 4.5% of candidates that passed all supplied tests, showing that test-suite success alone can miss discrepancies. With the same input-generation budget, symbolic execution achieved higher mean line and branch coverage than black-box fuzzing. Symbolic feedback also raised the mean single-step test-pass rate from 90.1% to 93.3%. More than half of tasks produced candidates that passed the supplied tests and had no differences detected by symbolic comparison; this indicates evidence of consistency, not proof of full behavioral equivalence.

### Source excerpt

arXiv:2611.18892v1 Announce Type: new Abstract: When source code is unavailable, decompilation provides a source-level representation of a binary. However, code recovered by conventional or LLM-assisted decompilation may not preserve the original behavior of the binary. Recent approaches evaluate re-executability using fixed test suites, but passing these suites does not ensure behavioral preservation. We present SymDec, an LLM-assisted decompiled-code repair system that uses the original binary as a behavioral reference. SymDec uses coverage-guided symbolic execution to explore the I/O-level behavior of the original and recompiled binaries, and symbolic comparison to identify inputs that expose behavioral differences. These inputs are replayed on both binaries to confirm the differences, which are returned as structured feedback to guide subsequent LLM revisions. We evaluate SymDec on 120 functions from ExeBench and DecompileBench, compiled for x86-64 and ARM64 at optimization levels O0-O3, yielding 960 binary decompilation tasks. More than half of the tasks yield candidates that pass the supplied test suite and complete symbolic comparison without detected behavioral differences. Among candidates that pass all supplied tests, symbolic comparison exposes replay-confirmed behavioral differences in 4.5% of cases, showing that test-suite re-executability can miss behavioral discrepancies. Under the same input-generation budget, symbolic execution achieves higher mean line and branch coverage than black-box fuzzing. Symbolic feedback also improves the conventional metric, increasing the mean single-step test-pass rate from 90.1% to 93.3%. These results show that symbolic comparison provides stronger evidence of behavioral consistency than test-suite re-executability alone and supplies effective feedback for repairing decompiled code.

## Macro Control and Capability Evaluation of Black-Box World Models: A Two-Regime Framework for LLM-Agent Markets

DevFeed: [Macro Control and Capability Evaluation of Black-Box World Models: A Two-Regime Framework for LLM-Agent Markets](<https://devfeed.tech/articles/macro-control-and-capability-evaluation-of-black-box-world-models-a-two-regime-framework-for-llm-agent-markets-137428.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19311>)

Author: Zhiyang Niu, Chuanjie Li, Huayuan Shen, Mengmeng Zhao, Jiajia Zhang

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [World models](<https://devfeed.tech/topics/world-models.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [black-box](<https://devfeed.tech/tags/black-box.md>), [capability](<https://devfeed.tech/tags/capability.md>), [control](<https://devfeed.tech/tags/control.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [llm](<https://devfeed.tech/tags/llm.md>), [markets](<https://devfeed.tech/tags/markets.md>), [world-models](<https://devfeed.tech/tags/world-models.md>)

### AI overview

The framework separates LLM-agent market capability into information simulation (macro information to micro behavior) and macro emergence (micro behavior to macro outcomes), using macro observations alone. It announces a public reference path without changing individual decisions, orders, or matching. In the reference-only regime, R measures realization and B structural bias; with external information, only the true-versus-flipped gap Δ remains interpretable. Across five single-index markets, reference paths shifted outcomes toward targets: tracking scores rose 0.306 and 0.415 without news, and 0.183 and 0.329 with news, relative to no reference. The reported scan found lowest bias at a 70/30 stock/cash mix (B=0.042), and peak realization at a 3:7 technical share (R=0.894, versus 0.358 for technical-only). Δ was 0.0341 with fundamental data and 0.0487 with news.

### Source excerpt

arXiv:2611.19311v1 Announce Type: new Abstract: Large language model (LLM) agents can be organized into multi-agent markets that act as world models, mapping macro information to emergent macro dynamics while the evaluator observes only macro inputs and outputs. This paper separates a model's capability into two stages that a single aggregate score cannot separate: information simulation, from macro information to micro behavior, and macro emergence, from micro behavior to the macro outcome. We introduce a soft macro-control method that announces a public reference path day by day without touching individual decisions, orders, or matching, and a two-regime evaluation framework that measures both stages from macro observations alone. In the reference-only regime the symmetric true and flipped references share a common level R, which measures realization under the announced reference, while their paired gap B measures structural bias; in the information-injection regime external information dilutes the reference, so only the true-flipped gap Delta remains interpretable. On five single-index LLM-agent markets, announcing a target path shifts the realized path toward the announced target on both announced references under both information conditions. Relative to the no-reference path, the tracking score is higher by 0.306 and 0.415 without news and by 0.183 and 0.329 with news; within the stock/cash scan structural bias is lowest at 70/30 (B=0.042); realization peaks at a 3:7 technical share (R=0.894 versus 0.358 for technical-only); and Delta is 0.0341 with fundamental data and 0.0487 with news. Both stages use the same two observable scores, each read in the regime where it is interpretable.

## Beyond the Patch: Reconstructing an Incomplete-Fix Patch Bypass in LLM Agent Frameworks with Control-Gated, Benign Probes

DevFeed: [Beyond the Patch: Reconstructing an Incomplete-Fix Patch Bypass in LLM Agent Frameworks with Control-Gated, Benign Probes](<https://devfeed.tech/articles/beyond-the-patch-reconstructing-an-incomplete-fix-patch-bypass-in-llm-agent-frameworks-with-control-gated-benign-probes-139304.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.20514>)

Author: Chaitanya Garware

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [test](<https://devfeed.tech/topics/test.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>)

Tags: [agent-frameworks](<https://devfeed.tech/tags/agent-frameworks.md>), [agno](<https://devfeed.tech/tags/agno.md>), [boundary](<https://devfeed.tech/tags/boundary.md>), [bypass](<https://devfeed.tech/tags/bypass.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cve](<https://devfeed.tech/tags/cve.md>), [exploit](<https://devfeed.tech/tags/exploit.md>), [harness](<https://devfeed.tech/tags/harness.md>), [llm](<https://devfeed.tech/tags/llm.md>), [loader](<https://devfeed.tech/tags/loader.md>), [loading](<https://devfeed.tech/tags/loading.md>), [override](<https://devfeed.tech/tags/override.md>), [praisonai](<https://devfeed.tech/tags/praisonai.md>), [security](<https://devfeed.tech/tags/security.md>), [study](<https://devfeed.tech/tags/study.md>), [testing](<https://devfeed.tech/tags/testing.md>), [tool](<https://devfeed.tech/tags/tool.md>), [version](<https://devfeed.tech/tags/version.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>)

### AI overview

Replaying a patched vulnerability's original proof of concept can miss an incomplete fix on a sibling code path. In a control-gated pilot, PraisonAI 4.6.31 blocked the original tool-loading trigger but still violated the tested invariant through tool_override, reproducing the documented CVE-2026-44334 bypass of CVE-2026-40287; neither path showed a violation in 4.6.32. CrewAI showed a violation in tested release 1.11.0 and confinement in 1.11.1, without establishing the earliest fixed release. Probes used inert markers or builtins-reachability checks, and 14 harness-validation scenarios returned expected verdicts. The PraisonAI probes called loader functions directly, so they establish boundary behavior under the supplied setup, not an end-to-end exploit. A non-violation applies only to the tested probe, version, and configuration; the pilot does not estimate vulnerability prevalence.

### Source excerpt

arXiv:2611.20514v1 Announce Type: new Abstract: Replaying a vulnerability's original proof-of-concept after patching tests whether its reported trigger remains reproducible, but does not establish that the underlying security invariant holds on other code paths. We present a benign, control-gated reconstruction method and a pilot study across four LLM agent frameworks and four invariant classes. For PraisonAI, we compare original-trigger replay (B0) with a distinct sibling-path probe (M). We also perform five before/after reconstructions across Langroid, Agno, and CrewAI. Each run reports VIOLATION, NO_VIOLATION_OBSERVED, or INCONCLUSIVE, gated by a positive authorized control. Probes use inert markers or builtins-reachability checks. At PraisonAI 4.6.31, the original tool-loading trigger is blocked while the tool_override path still violates the invariant. This reproduces the publicly documented CVE-2026-44334 bypass of CVE-2026-40287; neither tested path shows a violation at 4.6.32. For CrewAI, we observe a violation in tested release 1.11.0 and confinement in 1.11.1, without claiming the earliest fixed release. Fourteen harness-validation scenarios produced their expected verdicts. Only one same-version B0-versus-M divergence is demonstrated. The PraisonAI probes invoke loader functions directly and establish boundary behavior under the supplied setup, not a full end-to-end exploit. A non-violation verdict applies only to the tested probe, version, and configuration and is not a proof of safety. This pilot reconstructs a known incomplete fix; it neither introduces a new testing technique nor estimates vulnerability prevalence.

## Microsoft outage: A cascading failure that started with an undetected bug

DevFeed: [Microsoft outage: A cascading failure that started with an undetected bug](<https://devfeed.tech/articles/microsoft-outage-a-cascading-failure-that-started-with-an-undetected-bug-136600.md>)

Original publisher: [Read original article](<https://www.networkworld.com/article/4249392/microsoft-outage-a-cascading-failure-that-started-with-an-undetected-bug-2.html>)

Author: Evan Schuman

Published: 2026-11-17T00:02:57Z

Content type: news

Language: en

Sources: [Network World](<https://devfeed.tech/sources/network-world.md>)

Topics: [Post Mortem](<https://devfeed.tech/topics/post-mortem.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [microsoft 365](<https://devfeed.tech/topics/microsoft-365.md>), [test](<https://devfeed.tech/topics/test.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Microsoft](<https://devfeed.tech/topics/microsoft.md>), [Microsoft Teams](<https://devfeed.tech/topics/microsoft-teams.md>)

Tags: [availability](<https://devfeed.tech/tags/availability.md>), [bug](<https://devfeed.tech/tags/bug.md>), [defect](<https://devfeed.tech/tags/defect.md>), [edition-us](<https://devfeed.tech/tags/edition-us.md>), [exchange](<https://devfeed.tech/tags/exchange.md>), [exchange-online](<https://devfeed.tech/tags/exchange-online.md>), [failure](<https://devfeed.tech/tags/failure.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [global](<https://devfeed.tech/tags/global.md>), [global-outage](<https://devfeed.tech/tags/global-outage.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [high-availability](<https://devfeed.tech/tags/high-availability.md>), [incident](<https://devfeed.tech/tags/incident.md>), [incident-report](<https://devfeed.tech/tags/incident-report.md>), [independent](<https://devfeed.tech/tags/independent.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [issue](<https://devfeed.tech/tags/issue.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [microsoft-outage](<https://devfeed.tech/tags/microsoft-outage.md>), [networking](<https://devfeed.tech/tags/networking.md>), [outage](<https://devfeed.tech/tags/outage.md>), [storytype-news](<https://devfeed.tech/tags/storytype-news.md>)

### AI overview

A Microsoft 365 gateway software defect, deployed globally for months without impact, surfaced when Teams failed over to the primary gateway, increasing and changing traffic while nearly 14% of gateway capacity was already unavailable. Microsoft's load tests missed the defect because they used different traffic and SKU mixes and tended to use weaker, lower-core hardware; the report says more cores and newer hardware worsened contention. Requests accumulated, and automatic rerouting spread load across fewer healthy servers, causing a fleet-wide cascade that left more than 80% unresponsive at the low point. Recovery automation and normal rollback, flighting and feature flags could not quickly restore service. The preliminary report did not explain the unrelated capacity issue, and the article notes that Microsoft had yet to provide customers concrete reassurance against recurrence.

### Source excerpt

Following a roughly day-long global outage of many Microsoft services last week, Microsoft on Monday issued a preliminary post-incident report that identified multiple causes for the issues, including the admission that the environment used to test the underlying problematic software update did not match the actual infrastructure it would run on. The report, a summary of which was posted on social media by Microsoft EVP Charles Lamanna, identified some of the factors that cascaded to cause the outage. "A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions," the report said, adding, "this defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers." Its deployment began in July, and, the report said, "for more than three months, the change was deployed worldwide without impact." Microsoft admitted that, although it did test the change before deployment, its test environment didn't match reality. "Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs," it said. "The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores." A surge in traffic caused by unrelated issues also helped trigger the problem and delay it from being fixed, the company explained. "Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability," Microsoft said. "This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the primary gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns." To make things worse, the report said, "nearly 14% of server capacit

## Microsoft outage: A cascading failure that started with an undetected bug

DevFeed: [Microsoft outage: A cascading failure that started with an undetected bug](<https://devfeed.tech/articles/microsoft-outage-a-cascading-failure-that-started-with-an-undetected-bug-136867.md>)

Original publisher: [Read original article](<https://www.computerworld.com/article/4249364/microsoft-outage-a-cascading-failure-that-started-with-an-undetected-bug.html>)

Author: Evan Schuman

Published: 2026-11-16T23:54:42Z

Content type: news

Language: en

Sources: [Computerworld](<https://devfeed.tech/sources/computerworld.md>)

Topics: [cloud service outage management](<https://devfeed.tech/topics/cloud-service-outage-management.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [Post Mortem](<https://devfeed.tech/topics/post-mortem.md>), [microsoft 365](<https://devfeed.tech/topics/microsoft-365.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [test](<https://devfeed.tech/topics/test.md>), [Microsoft Teams](<https://devfeed.tech/topics/microsoft-teams.md>)

Tags: [bug](<https://devfeed.tech/tags/bug.md>), [defect](<https://devfeed.tech/tags/defect.md>), [edition-us](<https://devfeed.tech/tags/edition-us.md>), [exchange](<https://devfeed.tech/tags/exchange.md>), [exchange-online](<https://devfeed.tech/tags/exchange-online.md>), [failure](<https://devfeed.tech/tags/failure.md>), [gateway](<https://devfeed.tech/tags/gateway.md>), [global-outage](<https://devfeed.tech/tags/global-outage.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [high-availability](<https://devfeed.tech/tags/high-availability.md>), [incident](<https://devfeed.tech/tags/incident.md>), [incident-report](<https://devfeed.tech/tags/incident-report.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [load](<https://devfeed.tech/tags/load.md>), [load-testing](<https://devfeed.tech/tags/load-testing.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [microsoft-365](<https://devfeed.tech/tags/microsoft-365.md>), [microsoft-365-office-suites-productivity-software-vendors-and-providers](<https://devfeed.tech/tags/microsoft-365-office-suites-productivity-software-vendors-and-providers.md>), [microsoft-outage](<https://devfeed.tech/tags/microsoft-outage.md>), [network](<https://devfeed.tech/tags/network.md>), [office-suites](<https://devfeed.tech/tags/office-suites.md>), [outage](<https://devfeed.tech/tags/outage.md>), [productivity-software](<https://devfeed.tech/tags/productivity-software.md>), [storytype-news](<https://devfeed.tech/tags/storytype-news.md>), [vendors-and-providers](<https://devfeed.tech/tags/vendors-and-providers.md>)

### AI overview

Microsoft's preliminary report attributes the outage to a latent bug in a networking gateway update, exposed when Teams traffic shifted to the primary gateway and unrelated capacity loss left it under greater load. Testing missed the defect because lab traffic and hardware differed from production; the problem was worse on newer, higher-core servers. As overloaded servers stopped responding, traffic shifted onto fewer healthy servers, spreading the failure. More than 80% of servers became unresponsive at the low point, and restarting them could trigger the bug again. Automated recovery, rollback, flighting and feature flags could not quickly restore service. Consultants cited the production-testing mismatch and delayed alerting as concerns; Microsoft did not explain the unrelated capacity issue in the excerpt.

### Source excerpt

Following a roughly day-long global outage of many Microsoft services last week, Microsoft on Monday issued a preliminary post-incident report that identified multiple causes for the issues, including the admission that the environment used to test the underlying problematic software update did not match the actual infrastructure it would run on. The report, a summary of which was posted on social media by Microsoft EVP Charles Lamanna, identified some of the factors that cascaded to cause the outage. "A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions," the report said, adding, "this defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers." Its deployment began in July, and, the report said, "for more than three months, the change was deployed worldwide without impact." Microsoft admitted that, although it did test the change before deployment, its test environment didn't match reality. "Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs," it said. "The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores." A surge in traffic caused by unrelated issues also helped trigger the problem and delay it from being fixed, the company explained. "Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability," Microsoft said. "This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the primary gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns." To make things worse, the report said, "nearly 14% of server capacit

## OpenAI will embed safety researchers to investigate AI subversion

DevFeed: [OpenAI will embed safety researchers to investigate AI subversion](<https://devfeed.tech/articles/openai-will-embed-safety-researchers-to-investigate-ai-subversion-136062.md>)

Original publisher: [Read original article](<https://techcrunch.com/2026/11/16/openai-will-embed-apollo-researchers-to-investigate-ai-subversion/>)

Author: Rebecca Bellan

Published: 2026-11-16T17:02:43Z

Content type: news

Language: en

Sources: [TechCrunch | AI News & Artificial Intelligence](<https://devfeed.tech/sources/techcrunch-ai-news-artificial-intelligence.md>)

Topics: [ai safety](<https://devfeed.tech/topics/ai-safety.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>), [test](<https://devfeed.tech/topics/test.md>), [openai-hugging-face-incident](<https://devfeed.tech/topics/openai-hugging-face-incident.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [ai-safety-research](<https://devfeed.tech/tags/ai-safety-research.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [apollo](<https://devfeed.tech/tags/apollo.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [disclosure](<https://devfeed.tech/tags/disclosure.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [models](<https://devfeed.tech/tags/models.md>), [openai](<https://devfeed.tech/tags/openai.md>), [research](<https://devfeed.tech/tags/research.md>), [safety](<https://devfeed.tech/tags/safety.md>), [safety-researchers](<https://devfeed.tech/tags/safety-researchers.md>), [subversion](<https://devfeed.tech/tags/subversion.md>)

### AI overview

OpenAI's pilot places four outside evaluators inside its organization to examine risks that post-training checks near release may miss, including subversion during training or internal deployment. Apollo will investigate whether models try to undermine their own or successor models' safeguards, using agreed training code and datasets, reinforcement-learning traces, and intermediate checkpoints. Goodfire will use interpretability tools to monitor training and help investigate undesired behavior; METR will independently choose the scope and methods for one evaluation of dangerous model behavior, while the scope of its second evaluation remains undecided; Transluce will assess OpenAI's post-training safety evaluations. The pilot has limits: access is restricted to agreed claims, models, periods, and authorized staff, and OpenAI may withhold information for technical, security, or intellectual-property reasons. Apollo may publish conclusions, but OpenAI can limit supporting evidence made public. OpenAI's Safety and Security Committee will receive unredacted reports. Apollo and Goodfire are compensated; METR and Transluce are not, though they may accept credits.

### Source excerpt

OpenAI will embed six Apollo Research evaluators to investigate whether its AI models engage in subversive behavior, but restrictions on access and disclosure raise questions about their independence.

## Our first phase of embedded assessments

DevFeed: [Our first phase of embedded assessments](<https://devfeed.tech/articles/our-first-phase-of-embedded-assessments-136430.md>)

Original publisher: [Read original article](<https://openai.com/index/our-first-phase-of-embedded-assessments>)

Published: 2026-11-16T08:00:00Z

Content type: news

Language: en

Sources: [OpenAI News](<https://devfeed.tech/sources/openai-news.md>)

Topics: [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [test](<https://devfeed.tech/topics/test.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [post-training](<https://devfeed.tech/topics/post-training.md>)

Tags: [alignment](<https://devfeed.tech/tags/alignment.md>), [apollo](<https://devfeed.tech/tags/apollo.md>), [assessment](<https://devfeed.tech/tags/assessment.md>), [effectiveness](<https://devfeed.tech/tags/effectiveness.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [experts](<https://devfeed.tech/tags/experts.md>), [first](<https://devfeed.tech/tags/first.md>), [metr](<https://devfeed.tech/tags/metr.md>), [model-alignment](<https://devfeed.tech/tags/model-alignment.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [openai](<https://devfeed.tech/tags/openai.md>), [safety](<https://devfeed.tech/tags/safety.md>), [transluce](<https://devfeed.tech/tags/transluce.md>)

### AI overview

OpenAI has begun embedded safety assessments with Apollo Research, METR, Transluce, and Goodfire, giving independent experts access to selected internal models, checkpoints, data, monitoring systems, and staff. Apollo will examine possible covert subversion and whether safeguards detect it; Goodfire will use interpretability methods for training monitoring, debugging, and alignment; METR will assess misalignment in a large reinforcement learning run, with a broader monitor assessment still pending agreement; Transluce will review post-training alignment evaluations and may conduct a broader follow-on assessment in early 2027. Each engagement has an agreed scope and safety arrangements. Assessors can refine methods and request additional access, with alternatives considered for sensitive material. The Safety and Security Committee receives unredacted reports and can speak privately with assessors. Assessors retain publication independence, while OpenAI may redact confidential information or request delays for safety or security remediation. The article says these early protocols may evolve and identifies access, space, and security challenges.

### Source excerpt

OpenAI begins embedded assessments with Apollo Research, METR, and Transluce, and Goodfire to independently evaluate model alignment, safeguards, and monitoring systems.

## LEGAL-UQA: Testing Constitutional Question-Answering Systems for Urdu Speakers with a Parallel English-Urdu Benchmark

DevFeed: [LEGAL-UQA: Testing Constitutional Question-Answering Systems for Urdu Speakers with a Parallel English-Urdu Benchmark](<https://devfeed.tech/articles/legal-uqa-testing-constitutional-question-answering-systems-for-urdu-speakers-with-a-parallel-english-urdu-benchmark-134944.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2410.13013>)

Author: Faizan Faisal, Umair Yousaf, Muhammad Hassnain, Zeerak Babar

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Language Engineering](<https://devfeed.tech/topics/language-engineering.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [english](<https://devfeed.tech/tags/english.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [human-review](<https://devfeed.tech/tags/human-review.md>), [legal](<https://devfeed.tech/tags/legal.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [qa](<https://devfeed.tech/tags/qa.md>), [questions](<https://devfeed.tech/tags/questions.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

LEGAL-UQA provides 619 parallel English-Urdu question-answer pairs linked to passages from Pakistan's Constitution. Its evaluation separates answering Urdu questions with a passage supplied from retrieving the relevant passage. Model rankings varied with the answer-similarity measure, and simple word matching retrieved passages more effectively than the tested embedding methods. At the evaluated cutoff, the automated judge JEV agreed with human grades less often than a baseline that gave every answer full credit. An audit also found generated reference answers that omitted source-supported information. The authors recommend evaluating retrieval and answering separately and using human review to validate judges and references.

### Source excerpt

arXiv:2410.13013v2 Announce Type: replace-cross Abstract: Legal information assistants could help Urdu speakers understand their constitutional rights, but building reliable systems requires test data and evaluation methods suited to the language. We present LEGAL-UQA, a parallel English-Urdu benchmark of 619 question-answer pairs linked to passages from Pakistan's Constitution. Each example includes the question, reference answer, and source passage in both languages. We evaluate two components separately: answering Urdu questions when the source passage is supplied, and retrieving the passage from which each question was generated. We also compare JEV, a decision model used as an automated judge, with human grades. The findings show that model rankings depend on how answer similarity is measured, while a simple word-matching method retrieves source passages more effectively than the embedding methods tested. At the evaluated cutoff, JEV agrees with human grades less often than a baseline that gives every answer full credit. An audit also identifies cases where generated references omit information supported by the source. Together, these findings highlight the need to test retrieval and answering separately, include simple retrieval baselines, and validate automated judges and reference answers with human review when engineering legal QA software for Urdu speakers.

## Knowledge After Collapse: Context and Wording Shape Factual Accessibility in Language Models

DevFeed: [Knowledge After Collapse: Context and Wording Shape Factual Accessibility in Language Models](<https://devfeed.tech/articles/knowledge-after-collapse-context-and-wording-shape-factual-accessibility-in-language-models-135107.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.17763>)

Author: Sujato Dutta, Prithviraj Patil, Chandrajit Bajaj

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [test](<https://devfeed.tech/topics/test.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [qwen](<https://devfeed.tech/topics/qwen.md>), [llama](<https://devfeed.tech/topics/llama.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cue](<https://devfeed.tech/tags/cue.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [history](<https://devfeed.tech/tags/history.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llama](<https://devfeed.tech/tags/llama.md>), [persistence](<https://devfeed.tech/tags/persistence.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [qwen](<https://devfeed.tech/tags/qwen.md>), [recovery](<https://devfeed.tech/tags/recovery.md>), [reliability](<https://devfeed.tech/tags/reliability.md>)

### AI overview

Knowledge Answer Divergence (KAD) evaluates what happens after a model explicitly adopts a designated false answer: it freezes that conversation and branches independent probes to measure persistence, recovery, or abstention. This allows matched comparisons of follow-up behavior and history-management interventions beyond a single susceptibility score. In corrected Llama 3.2 Instruct 3B controls, 42 of 46 responses were correct with a neutral history versus 16 of 46 with the full history. A matched cue-placement control shifted false adoption by 27.3 percentage points in Qwen 3.5 4B, and twelve independently authored pressure templates showed substantial sensitivity to wording. The reported evaluation covered eight local checkpoint conditions, six hosted endpoints, and an audited 300-question rerun.

### Source excerpt

arXiv:2611.17763v1 Announce Type: new Abstract: When a language model accepts a false correction in a factual discussion, its next answer can depend on how the conversation continues. We introduce Knowledge Answer Divergence (KAD), an event-conditioned evaluation protocol that distinguishes initial false-answer adoption from subsequent persistence, recovery, and abstention. KAD freezes the conversation at the first explicit adoption of a designated false target and branches independent probes from that exact history. This enables matched comparisons of post-adoption behavior and history-management interventions that a single susceptibility score cannot capture. We evaluate eight local checkpoint conditions and six hosted endpoints, alongside an audited 300-question rerun. In corrected Llama 3.2 Instruct 3B controls, a neutral-history branch yields 42/46 correct responses, compared with 16/46 under full history. A matched cue-placement control changes false adoption by 27.3 percentage points in Qwen 3.5 4B. Twelve independently authored pressure templates demonstrate substantial sensitivity to wording. Together, these findings demonstrate the utility of KAD for evaluating factual reliability beyond initial susceptibility, enabling systematic comparisons of false-answer persistence, recovery conditions and conversational interventions.

## Before AI Agents Negotiate for Us: A Competitive Tournament of Rule-Based, Search, Optimization, Learned, and Language-Model Negotiators Under Scarcity

DevFeed: [Before AI Agents Negotiate for Us: A Competitive Tournament of Rule-Based, Search, Optimization, Learned, and Language-Model Negotiators Under Scarcity](<https://devfeed.tech/articles/before-ai-agents-negotiate-for-us-a-competitive-tournament-of-rule-based-search-optimization-learned-and-language-model-negotiators-under-scarcity-132179.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.15838>)

Author: Eduardo de Avila-Armenta, Ra\\'ul Alejandro Aquino-Ibarra, Jorge Octavio Nicol\\'as-D\\'iaz, Umberto Leon-Dominguez

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-gt](<https://devfeed.tech/tags/cs-gt.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [imitation-learning](<https://devfeed.tech/tags/imitation-learning.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [search](<https://devfeed.tech/tags/search.md>)

### AI overview

In a simulated Gale-Shapley matching market, no negotiation architecture led across every condition and role. Among nine rule-based, search, optimization, imitation-learning, and reinforcement-learning systems, ML2, RL, and OPT were statistically tied at the top in the balanced carrier market. RL beat all eight carrier rivals when carriers were scarce, but its match rate fell from 62% to 16% when shippers were scarce. RL2 maintained match rates above 91% in all conditions but had lower realized payoff than every shipper-side rival in two conditions, attributed to a confirmed mismatch between training and deployment features. Nine language models, ranging from 1 to 72 billion parameters, lost significantly to OPT2 and ML2 in 32 of 36 comparisons and won none. The abstract's results suggest that evaluations across multiple rivals and scarcity conditions can expose weaknesses hidden by isolated or fixed-market tests.

### Source excerpt

arXiv:2611.15838v1 Announce Type: new Abstract: Automated bilateral negotiation underlies two-sided matching markets such as freight brokering, where many shippers and carriers negotiate price before a matching mechanism decides who is paired with whom. Prior strategy comparisons typically use isolated one-on-one duels or a single fixed market, so they cannot show how a strategy behaves against several live rivals under a changing balance of supply and demand. We built nine negotiation architectures spanning rule-based, search-based, optimization-based, imitation-learning, and reinforcement-learning families, and ran all nine simultaneously on both sides of a Gale-Shapley matching market under a balanced condition and two scarcity conditions rotated across all 36 exclusion pairs. In the balanced carrier market, an imitation-learned engine (ML2), a reinforcement-learning policy trained only on isolated negotiations (RL), and an expected-utility optimization engine (OPT) formed a statistically tied top group, and no architecture led in every condition and role. RL beat all eight carrier rivals when carriers were scarce, but its match rate collapsed from 62% to 16% when shippers were scarce and it had to compete with the other carrier agents for demand. A second reinforcement-learning policy trained against rival agents (RL2) kept its match rate above 91% in every condition, yet lost on realized payoff to all eight shipper-side rivals in two of the three conditions, traced to a confirmed train/deployment feature mismatch. We also tested nine large language models (LLMs), from 1 to 72 billion parameters, under the same protocol: they lost significantly to the two strongest engines (OPT2 and ML2) in 32 of 36 comparisons and won none. These results show that isolated or single-condition evaluation can hide failure modes that a many-sided, scarcity-varying, statistically corrected tournament exposes directly.

[Next page](<https://devfeed.tech/topics/test.md?cursor=WyIyMDI2LTExLTEzVDA1OjAwOjAwKzAwOjAwIiwgImI0ZjBmODg2LThmZjQtNDNlNi04NjdiLTk4YmIyNGVmOTE1YyJd>)