# Large Language Model

A large language model (LLM) is a text-driven foundational machine-learning model trained on vast amounts of data for natural-language-processing tasks.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Clairic Hit 98% Accuracy on 72.8% of Questions via Confidence Scores

DevFeed: [Clairic Hit 98% Accuracy on 72.8% of Questions via Confidence Scores](<https://devfeed.tech/articles/clairic-hit-98-accuracy-on-72-8-of-questions-via-confidence-scores-146441.md>)

Original publisher: [Read original article](<https://towardsdatascience.com/clairic-hit-98-accuracy-on-728-of-questions-via-confidence-scores/>)

Author: Mariya Mansurova

Published: 2026-11-20T15:00:00Z

Content type: tutorial

Language: en

Sources: [Towards Data Science](<https://devfeed.tech/sources/towards-data-science.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Data Science](<https://devfeed.tech/topics/data-science.md>), [ai-experiments](<https://devfeed.tech/topics/ai-experiments.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [analysts](<https://devfeed.tech/tags/analysts.md>), [arithmetic](<https://devfeed.tech/tags/arithmetic.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [autocomplete](<https://devfeed.tech/tags/autocomplete.md>), [based](<https://devfeed.tech/tags/based.md>), [bike](<https://devfeed.tech/tags/bike.md>), [charts](<https://devfeed.tech/tags/charts.md>), [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [classification](<https://devfeed.tech/tags/classification.md>), [cloudflare](<https://devfeed.tech/tags/cloudflare.md>), [complex](<https://devfeed.tech/tags/complex.md>), [confidence](<https://devfeed.tech/tags/confidence.md>), [confidence-interval](<https://devfeed.tech/tags/confidence-interval.md>), [control](<https://devfeed.tech/tags/control.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [decision-models](<https://devfeed.tech/tags/decision-models.md>), [easy](<https://devfeed.tech/tags/easy.md>), [editor-s-picks](<https://devfeed.tech/tags/editor-s-picks.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [jev](<https://devfeed.tech/tags/jev.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-applications](<https://devfeed.tech/tags/llm-applications.md>), [luna](<https://devfeed.tech/tags/luna.md>), [model](<https://devfeed.tech/tags/model.md>), [science](<https://devfeed.tech/tags/science.md>), [system-one](<https://devfeed.tech/tags/system-one.md>)

### AI overview

Clairic is a fast, low-cost extraction model that accepts text and images and can return typed values with confidence scores. In tests against Luna, Clairic was faster but less accurate overall, especially on calculations; examples also included confidently incorrect extractions. It handled some chart-based extraction and returned null for missing revenue. On 1,360 questions across four languages, a 0.9 confidence cutoff yielded 98% accuracy while Clairic answered 72.8% of questions. The author suggests using it for high-confidence extraction and routing the rest to a larger model, with validation because confidence is imperfect. The tests did not establish a reason to prefer it for forecasting over traditional methods.

### Source excerpt

With a 0.9 confidence cutoff, Cloudflare's System One model handles the easy cases and routes the rest to a bigger, slower LLM like Luna. The post Clairic Hit 98% Accuracy on 72.8% of Questions via Confidence Scores appeared first on Towards Data Science.

## Datadog LLM Observability: Features, Pricing, and Alternatives

DevFeed: [Datadog LLM Observability: Features, Pricing, and Alternatives](<https://devfeed.tech/articles/datadog-llm-observability-features-pricing-and-alternatives-145962.md>)

Original publisher: [Read original article](<https://newrelic.com/blog/ai/datadog-llm-observability>)

Author: John Blust

Published: 2026-11-20T11:51:52.599709Z

Content type: article

Language: en

Sources: [New Relic](<https://devfeed.tech/sources/new-relic.md>)

Topics: [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [datadog](<https://devfeed.tech/topics/datadog.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>), [observability](<https://devfeed.tech/topics/observability.md>), [tracing](<https://devfeed.tech/topics/tracing.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [phoenix](<https://devfeed.tech/topics/phoenix.md>), [LangChain](<https://devfeed.tech/topics/langchain.md>), [dynatrace](<https://devfeed.tech/topics/dynatrace.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-monitoring](<https://devfeed.tech/tags/ai-monitoring.md>), [alternatives](<https://devfeed.tech/tags/alternatives.md>), [compare](<https://devfeed.tech/tags/compare.md>), [datadog](<https://devfeed.tech/tags/datadog.md>), [devops](<https://devfeed.tech/tags/devops.md>), [evaluations](<https://devfeed.tech/tags/evaluations.md>), [features](<https://devfeed.tech/tags/features.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-observability](<https://devfeed.tech/tags/llm-observability.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [new-relic](<https://devfeed.tech/tags/new-relic.md>), [observability](<https://devfeed.tech/tags/observability.md>), [pricing](<https://devfeed.tech/tags/pricing.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

### AI overview

Datadog LLM Observability, now called Agent Observability, traces model calls and agent workflows, evaluates quality and safety, supports prompt experiments, and tracks latency, tokens, errors, and costs. It is a standalone product billed by model-call spans; tool, retrieval, and workflow spans do not count. Both listed plans include core tracing and evaluation features, and annual Pro pricing starts at $160 per month, with retention add-ons affecting per-span rates. Alternatives differ in approach: New Relic connects AI telemetry to existing APM, including optional eBPF monitoring; Langfuse offers a self-hostable core; LangSmith emphasizes agent tracing and evaluation workflows; and Arize Phoenix supports local deployment with OpenTelemetry-based instrumentation. The text notes that Datadog can correlate with its other products through shared trace IDs and supports Python, Node.js, Java, OpenTelemetry, and HTTP API integration.

### Source excerpt

An objective look at Datadog LLM Observability: tracing, evaluations, how pricing works, and how alternatives like New Relic AI Monitoring compare.

## 21 ways Gemini can genuinely be useful on Android

DevFeed: [21 ways Gemini can genuinely be useful on Android](<https://devfeed.tech/articles/21-ways-gemini-can-genuinely-be-useful-on-android-146554.md>)

Original publisher: [Read original article](<https://www.computerworld.com/article/4007736/21-ways-gemini-can-genuinely-be-useful-on-android.html>)

Author: JR Raphael

Published: 2026-11-20T10:45:00Z

Content type: tutorial

Language: en

Sources: [Computerworld](<https://devfeed.tech/sources/computerworld.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-chatbots](<https://devfeed.tech/tags/ai-chatbots.md>), [android](<https://devfeed.tech/tags/android.md>), [android-artificial-intelligence-google-machine-learning-productivity-software](<https://devfeed.tech/tags/android-artificial-intelligence-google-machine-learning-productivity-software.md>), [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [chatbot](<https://devfeed.tech/tags/chatbot.md>), [context](<https://devfeed.tech/tags/context.md>), [device](<https://devfeed.tech/tags/device.md>), [edition-us](<https://devfeed.tech/tags/edition-us.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [google](<https://devfeed.tech/tags/google.md>), [google-assistant](<https://devfeed.tech/tags/google-assistant.md>), [google-s](<https://devfeed.tech/tags/google-s.md>), [graveyard](<https://devfeed.tech/tags/graveyard.md>), [industry](<https://devfeed.tech/tags/industry.md>), [language-en](<https://devfeed.tech/tags/language-en.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [like](<https://devfeed.tech/tags/like.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [productivity](<https://devfeed.tech/tags/productivity.md>), [productivity-software](<https://devfeed.tech/tags/productivity-software.md>), [rap](<https://devfeed.tech/tags/rap.md>), [search](<https://devfeed.tech/tags/search.md>), [storytype-tips](<https://devfeed.tech/tags/storytype-tips.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [technology](<https://devfeed.tech/tags/technology.md>)

### AI overview

Gemini is presented as most useful on Android for practical tasks rather than as a reliable answer engine: the excerpt says large language models can misunderstand context and give wrong answers, while Gemini still lacks some Google Assistant device basics. Examples include setting reminders, remembering details, saving notes to Google Keep, finding and labeling locations, navigating, checking or creating calendar events, making contacts from cards or visible information, summarizing on-screen documents, YouTube videos, and--when Gmail/Workspace integration is active--recent emails. It can also identify visible items and handle phone tasks such as checking battery, taking or showing screenshots, texting, calling, adjusting volume and settings, and opening system settings. These capabilities depend on the described integrations and on-screen context; the excerpt does not establish that every command works in all configurations.

### Source excerpt

Google's Gemini chatbot gets a bit of a bad rap as an Android assistant -- and, let's be honest: It's easy to see why. Google rushed Gemini out of the gates long before it was ready. It crippled the perfectly fine Google Assistant experience, and even with all the progress over time, Gemini still can't do everything Assistant did in terms of day-to-day device basics -- despite having now fully replaced Assistant and sent that service to the Google Graveyard. Beyond that, Gemini can be complicated and confusing, with its many models and cryptically coded options. It's being shoved in our faces at every possible opportunity in a move reminiscent of a certain other much-maligned Google service. And, most troubling of all, it's being positioned as an all-purpose answer machine when it gets answers wildly wrong anywhere from 5 to 20% of the time. Just like all the other generative AI chatbots of the moment, in other words, it's a steaming hot mess -- even if it's all anyone in the tech industry wants to talk about these days. But the problem with Gemini (and other similar systems) is less the technology itself and more the way its maker insists on framing it. These large-language model mechanisms just aren't good at understanding context or offering consistently reliable answers. They shouldn't be set up to serve as replacements for search. They get stuff wrong shockingly often, for cryin' out loud. It's par for the course with the underlying technology, and it's just plain silly to see the results. (Seriously: The hits keep comin' -- and comin', and comin', and...well, you get the idea.) They are, however, useful in other ways -- namely when it comes to more subjective, mundane sorts of tasks where context, judgment, and a lack of flat-out fabricating aren't required. Here, specifically, are 21 simple ways Gemini can genuinely be useful and worth using on Android -- with minimal "AI" (in the current sense of the term), no "creative brainstorming" ballyhoo, and absolutely no err

## From Geographic Memory to Spatial Generalization in Large Language Models

DevFeed: [From Geographic Memory to Spatial Generalization in Large Language Models](<https://devfeed.tech/articles/from-geographic-memory-to-spatial-generalization-in-large-language-models-145092.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25703>)

Author: Chen Chu, Zhaonan Wang, Cyrus Shahabi, Hengcai Zhang, Feng Lu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [geospatial](<https://devfeed.tech/topics/geospatial.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [geospatial](<https://devfeed.tech/tags/geospatial.md>), [grounding](<https://devfeed.tech/tags/grounding.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [location](<https://devfeed.tech/tags/location.md>), [london](<https://devfeed.tech/tags/london.md>), [spatial](<https://devfeed.tech/tags/spatial.md>)

### AI overview

The study separates geospatial memory--recalling observed location-description pairs--from spatial generalization to locations outside the training observations. It adds continuous location representations from a spatial encoder as tokens in LLMs and evaluates them on newly curated Los Angeles and London benchmarks. Captioning tests observed and unobserved locations, while a separate grounding task measures whether generated pairs help map text to geography. Across multiple LLM families, continuous representations improve captioning at unobserved locations compared with textual coordinates; generated descriptions bring modest but consistent grounding gains. Probing and clustering results are also consistent with spatially organized representations. The evidence is limited to the described benchmarks and assessments.

### Source excerpt

arXiv:2611.25703v1 Announce Type: cross Abstract: Large language models (LLMs) can associate places with geographic descriptions, but recovering an observed association does not establish an ability to generalize across geographic space. We distinguish geospatial memory, the recovery of location--description associations observed during training, from spatial generalization, the prediction of descriptions for locations in unobserved regions. We investigate whether continuous location representations improve this generalization and encourage geographic knowledge to become spatially organized within an LLM. We frame this organization as a latent spatial field: a learned mapping from locations to descriptions connected through spatial dependencies. Building on a continuous spatial encoder, we project location representations into the model's embedding space and supply them as spatial tokens. We evaluate the resulting models on two newly curated geo-text benchmarks for Los Angeles and London using two complementary assessments. Geospatial captioning tests both memory and spatial generalization by generating descriptions at observed locations and at locations beyond the spatial coverage of the training observations. A separate geospatial grounding model tests whether generated location--description pairs provide useful additional supervision for mapping text to geographic space. Across multiple LLM families, continuous location representations improve captioning at unobserved locations relative to textual coordinates, while their generated descriptions yield modest but consistent improvements in downstream grounding. Layer-wise probing and representation clustering provide additional evidence consistent with spatially organized geographic representations. Together, these findings show the value of evaluating spatial generalization separately from memory and of incorporating continuous spatial representations into language models.

## How Large Language Models Misrepresent American Climate Opinions

DevFeed: [How Large Language Models Misrepresent American Climate Opinions](<https://devfeed.tech/articles/how-large-language-models-misrepresent-american-climate-opinions-145311.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2512.23889>)

Author: Sola Kim, Jieshu Wang, Marco A. Janssen, John M. Anderies

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Hallucination detection](<https://devfeed.tech/topics/hallucination-detection.md>)

Tags: [climate](<https://devfeed.tech/tags/climate.md>), [compression](<https://devfeed.tech/tags/compression.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [error](<https://devfeed.tech/tags/error.md>), [gap](<https://devfeed.tech/tags/gap.md>), [gender](<https://devfeed.tech/tags/gender.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [opinions](<https://devfeed.tech/tags/opinions.md>), [race](<https://devfeed.tech/tags/race.md>), [survey](<https://devfeed.tech/tags/survey.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [variance](<https://devfeed.tech/tags/variance.md>)

### AI overview

In a comparison of six language models with 978 respondents to a nationally representative U.S. climate opinion survey across 20 questions, predictions were less varied than human responses: predicted variance averaged 29% lower, pulling less-concerned groups toward greater concern and more-concerned groups toward less. The models also predicted women as more pro-climate than men in every racial group. That pattern aligned with the human pattern for White, Hispanic, and Other Americans but reversed for Black Americans. The authors emphasize the direction, not the size, of this difference because ideological-extreme subgroups were small. They argue that auditing gender alone would miss how the error changes by race. These findings concern the studied models, profiles, survey, and questions.

### Source excerpt

arXiv:2512.23889v3 Announce Type: replace-cross Abstract: Large language models are increasingly used to simulate public opinion, standing in for survey respondents as "silicon samples." Whether they faithfully represent U.S. climate opinion, particularly at the intersections of race, gender, and ideology, remains largely untested. We prompted six large language models with the profiles of 978 respondents to a nationally representative U.S. climate opinion survey and compared the generated responses with those respondents' own answers across 20 questions. The models compress the diversity of American climate opinion, pulling predictions toward the population mean: less-concerned groups are predicted as more concerned, more-concerned groups as less, and the variance of predicted responses falls, on average, 29% below that of human responses. The compression is intersectional. The models apply one gender assumption, predicting women as more pro-climate than men in every racial group. The assumption matches the direction of the human pattern for White, Hispanic, and Other Americans and runs the wrong way for Black Americans. We claim the direction of that difference rather than its size, since it rests on small cells at the ideological extreme. An audit of gender alone would report that the models understate the gender gap, and would miss that the error changes direction by race, which is what single-variable audits cannot detect. What is at stake is not accuracy alone but representation: whose views a synthetic sample renders legible.

## BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection

DevFeed: [BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection](<https://devfeed.tech/articles/buggymle-scaling-machine-learning-engineering-difficulty-with-bug-injection-145007.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25088>)

Author: John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [AI Development](<https://devfeed.tech/topics/ai-development.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Training AI Models](<https://devfeed.tech/topics/training-ai-models.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>), [ai-systems](<https://devfeed.tech/tags/ai-systems.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [based](<https://devfeed.tech/tags/based.md>), [bench](<https://devfeed.tech/tags/bench.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bug](<https://devfeed.tech/tags/bug.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [complex](<https://devfeed.tech/tags/complex.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-ma](<https://devfeed.tech/tags/cs-ma.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [learning](<https://devfeed.tech/tags/learning.md>), [machine](<https://devfeed.tech/tags/machine.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

BuggyMLE creates harder repository-level machine-learning engineering tasks by injecting controlled bugs into working ML repositories. Its Code Complexity Index uses ten structural features, including repository scale, dependency depth, and patch size; across eight coding benchmarks, it predicts average solve rates with R² = 0.951. From 222 repositories, the framework produces over 6,600 variants and nearly 260,000 lines of injected code. On BuggyMLE-Bench-Lite, the highest solve rate among 14 models falls from 95% at the easiest tier to 9% at the hardest. A three-level curriculum built from 200 additional repositories fine-tuned Qwen3.5-9B to rival frontier models on that benchmark; the model also improved reported results on SWE-bench Lite (24% to 33%) and LiveCodeBench v6 pass@1 (69% to 71%).

### Source excerpt

arXiv:2611.25088v1 Announce Type: cross Abstract: Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks ($R^2 = 0.951$), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.

## GenSPARQL: A Generative Extension of SPARQL for Incomplete Knowledge Graphs

DevFeed: [GenSPARQL: A Generative Extension of SPARQL for Incomplete Knowledge Graphs](<https://devfeed.tech/articles/gensparql-a-generative-extension-of-sparql-for-incomplete-knowledge-graphs-145628.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26062>)

Author: Jingcheng Wu, Yi Wang, Hongkuan Zhou, Steffen Staab, Bo Xiong

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [база знаний](<https://devfeed.tech/topics/tag-c51cc4bf76e5.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Wikidata](<https://devfeed.tech/topics/wikidata.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [generative](<https://devfeed.tech/tags/generative.md>), [join](<https://devfeed.tech/tags/join.md>), [knowledge-graphs](<https://devfeed.tech/tags/knowledge-graphs.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>)

### AI overview

GenSPARQL extends SPARQL 1.1 with GENOP, which binds LLM-generated values to query variables while evaluating a query over incomplete knowledge graphs. Because LLMs return names rather than graph IRIs, a similarity join maps those names to graph entities. The query planner limits LLM calls to bindings selected by other graph patterns and filters, addressing the need for a separate call per variable binding. A web workbench lets users query two Wikidata-based graphs and inspect generated values, mapped entities, and calls. In evaluation with three open-weight LLMs, embedding similarity mapped 83% of named entities and produced one incorrect match.

### Source excerpt

arXiv:2611.26062v1 Announce Type: cross Abstract: Knowledge graphs on the Web are incomplete, and SPARQL answers a query only from the stored triples. Large language models (LLMs) can predict missing facts, but an LLM returns names instead of IRIs and requires one call per binding of a query variable. We present GenSPARQL, a generative extension of SPARQL 1.1 with an operator GENOP that binds values generated by an LLM to query variables during query evaluation. A similarity join maps generated names to entities of the graph, and the query planner invokes the LLM only for the bindings selected by the other graph patterns and filters. In a web workbench, attendees query two graphs built from Wikidata and inspect generated values, mapped entities and LLM calls. In our evaluation with three open-weight LLMs, the embedding similarity maps 83% of the entities named by the LLMs and yields only one incorrect match.

## SGLayout: Schema-Guided Closed-Loop Refinement for LLM-Driven Parameterized Layout Generation

DevFeed: [SGLayout: Schema-Guided Closed-Loop Refinement for LLM-Driven Parameterized Layout Generation](<https://devfeed.tech/articles/sglayout-schema-guided-closed-loop-refinement-for-llm-driven-parameterized-layout-generation-145510.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25410>)

Author: Siyang Chen, Yuan Pu, Yuhan Qin, Zhaotan Lin, Siyuan Liang, Zhuolun He, Lining Zhang, Bei Yu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [code-execution](<https://devfeed.tech/tags/code-execution.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ar](<https://devfeed.tech/tags/cs-ar.md>), [drc](<https://devfeed.tech/tags/drc.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [generation](<https://devfeed.tech/tags/generation.md>), [layout](<https://devfeed.tech/tags/layout.md>), [llm](<https://devfeed.tech/tags/llm.md>), [repair](<https://devfeed.tech/tags/repair.md>), [schema](<https://devfeed.tech/tags/schema.md>), [tasks](<https://devfeed.tech/tags/tasks.md>)

### AI overview

SGLayout uses a structured schema to link code instances to geometric attributes, then maps layout violations back to precise code targets for repair. It checks intent through quantitative structural checks and qualitative vision-language assessment, and checks rule compliance with DRC; this loop repeats until layout convergence. On 112 GDSFactory tasks, it achieved a 76.79% pass rate, versus 50.89% for the strongest baseline.

### Source excerpt

arXiv:2611.25410v1 Announce Type: cross Abstract: Large Language Models (LLMs) have shown strong potential in parameterized layout generation via tool-augmented code synthesis. However, LLMs lack spatial awareness, making one-shot generation unreliable. While layout quality can be evaluated with established techniques, a critical gap remains: geometric-domain evaluation outcomes cannot be directly translated into actionable code-level fixes, trapping iterative refinement in blind refactoring. We propose SGLayout, a self-refining framework that closes this gap via a schema-guided mechanism. Upon code execution, a structured schema captures the correspondence between each code-level instance and its geometric attributes, establishing a bidirectional link between code space and geometry space. Emulating engineer review practices, SGLayout evaluates layouts along two complementary dimensions: intent compliance through quantitative structural checking and qualitative vision-language assessment, and rule compliance through DRC. All identified violations are then back-projected through the schema onto precise code targets, enabling surgical repair. This closed-loop process iterates until layout convergence. On a curated benchmark of 112 parameterized layout generation tasks using GDSFactory, SGLayout achieves a 76.79% pass rate, compared with 50.89% for the strongest baseline.

## Why Trust Your Agent? Empirical Security Gains from TRiSM-Guided Agentic Workflows in Healthcare

DevFeed: [Why Trust Your Agent? Empirical Security Gains from TRiSM-Guided Agentic Workflows in Healthcare](<https://devfeed.tech/articles/why-trust-your-agent-empirical-security-gains-from-trism-guided-agentic-workflows-in-healthcare-145354.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.28666>)

Author: Liam Kearns

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>)

Topics: [LLM security](<https://devfeed.tech/topics/llm-security.md>), [ai security](<https://devfeed.tech/topics/ai-security.md>), [least privilege](<https://devfeed.tech/topics/least-privilege.md>), [Security](<https://devfeed.tech/topics/security.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-workflows](<https://devfeed.tech/tags/agentic-workflows.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-trust](<https://devfeed.tech/tags/ai-trust.md>), [attack](<https://devfeed.tech/tags/attack.md>), [automation](<https://devfeed.tech/tags/automation.md>), [breaches](<https://devfeed.tech/tags/breaches.md>), [claude-haiku-4-5](<https://devfeed.tech/tags/claude-haiku-4-5.md>), [client](<https://devfeed.tech/tags/client.md>), [construction](<https://devfeed.tech/tags/construction.md>), [covid-19](<https://devfeed.tech/tags/covid-19.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [data](<https://devfeed.tech/tags/data.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [disclosure](<https://devfeed.tech/tags/disclosure.md>), [gemini](<https://devfeed.tech/tags/gemini.md>), [generation](<https://devfeed.tech/tags/generation.md>), [healthcare](<https://devfeed.tech/tags/healthcare.md>), [security](<https://devfeed.tech/tags/security.md>), [trust](<https://devfeed.tech/tags/trust.md>)

### AI overview

In a medical report-generation application tested with five LLMs, a TRiSM-guided workflow reduced attack success for RAG poisoning (31% to 10%) and data-field injection (42% to 25%), eliminated client-side network injection by construction, and increased report accuracy from 72.5% to 86.5%. In an exploratory grouping, attacks aimed beyond the reporting task, including data disclosure and tool abuse, fell from 21.7% to 2.5%, with no patient records disclosed under the redesign. However, attacks that changed the Covid-19 diagnosis remained effective, making up 32 of 35 successful attacks against the redesign. The findings suggest least privilege limits what injections can reach but cannot ensure the model resists injected content within its task; clinical outputs still need independent verification. No tested model resisted all three attack vectors.

### Source excerpt

arXiv:2606.28666v2 Announce Type: replace-cross Abstract: Agent-based AI has enabled the automation of tasks by exposing application tools and resources to large language models (LLMs). However, to improve scope and accuracy, agents are often given access rights that exceed those of ordinary users, introducing significant security risks and potential regulatory breaches. This paper applies the AI Trust, Risk, and Security Management (TRiSM) framework to a medical report-generation application to demonstrate how an insecure baseline workflow can be transformed into a security-conscious agentic workflow. Both designs were evaluated with five LLMs (Claude Haiku 4.5, GPT-4.1-nano, GPT-4.1-mini, GPT-5.4-mini and Gemini 2.5 Flash) across 800 report generations and 500 attack trials covering RAG poisoning, data-field injection and client-side network injection. The redesign reduced attack success from 31% to 10% for RAG poisoning and from 42% to 25% for data-field injection (McNemar p < 0.001 for both), removed network injection by construction, and raised report accuracy from 72.5% to 86.5%. In an exploratory post-hoc grouping by attack goal, success against attacks aimed outside the report task, such as data disclosure and tool abuse, fell from 21.7% to 2.5% and no patient record was disclosed under the redesign. Attacks altering the reported Covid-19 diagnosis remained effective, accounting for 32 of the 35 successful attacks against the redesign. Least-privilege architecture limits what an injection can reach, but not whether a model follows injected content within its task, so clinical outputs need verification independent of the LLM. No model resisted all three attack vectors, so model selection cannot substitute for secure workflow design.

## Decoupled Knowledge Retrieval and Compact Memory Enable Reliable Lifelong Model Editing

DevFeed: [Decoupled Knowledge Retrieval and Compact Memory Enable Reliable Lifelong Model Editing](<https://devfeed.tech/articles/decoupled-knowledge-retrieval-and-compact-memory-enable-reliable-lifelong-model-editing-145596.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25861>)

Author: Yue Che, Jingze Wang, Chen Guoyi, Yujie Wu, Xuming Ran, Mingkun Xu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>)

Tags: [artificial-intelligence](<https://devfeed.tech/tags/artificial-intelligence.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>)

### AI overview

BiSpace addresses growing storage needs and declining retrieval performance in memory-based lifelong model editing while leaving the base LLM parameters untouched. It separates retrieval and injection across representation spaces: queries locate edits in an external semantic space, and sparse, storage-efficient updates inject the relevant knowledge into the model's internal representation space. The abstract reports that BiSpace reduces storage per entry to about 10.2% of the original, remains resilient through edit sequences of up to 16K edits, and improves Overall Performance by more than 23% over existing methods on a more challenging benchmark.

### Source excerpt

arXiv:2611.25861v1 Announce Type: new Abstract: Lifelong model editing requires continuously incorporating new facts into deployed large language models (LLMs) without expensive retraining or compromising existing capabilities. Memory-based editors offer an appealing solution by keeping the base parameters untouched, yet their practical use is hindered by linearly growing storage requirements and deteriorating retrieval performance as edits accumulate. To tackle these problems, we introduce BiSpace, a lifelong model editing framework that decouples memory retrieval from knowledge injection across two distinct representation spaces. BiSpace routes queries and identifies relevant edits in an external semantic space, while injecting the corresponding knowledge into the internal representation space of the LLM through sparse and storage-efficient updates. Extensive experiments show that BiSpace shrinks each entry's storage to only about 10.2% of the original and shows strong resilience to long edit sequences of up to 16K edits. On a more challenging benchmark, BiSpace further improves Overall Performance by more than 23% over existing methods. These results advance memory-based editing toward reliable long-term model maintenance at scale.

## The ROI of Small Examples: Optimizing In-Context Learning Under Token Budgets

DevFeed: [The ROI of Small Examples: Optimizing In-Context Learning Under Token Budgets](<https://devfeed.tech/articles/the-roi-of-small-examples-optimizing-in-context-learning-under-token-budgets-145450.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24971>)

Author: Anxo Perez, Jorge Gab\\'in, Javier Parapar

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [few-shot](<https://devfeed.tech/topics/few-shot.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM optimization](<https://devfeed.tech/topics/llm-optimization.md>)

Tags: [budget](<https://devfeed.tech/tags/budget.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [examples](<https://devfeed.tech/tags/examples.md>), [in-context-learning](<https://devfeed.tech/tags/in-context-learning.md>), [inference](<https://devfeed.tech/tags/inference.md>), [model-performance](<https://devfeed.tech/tags/model-performance.md>), [prompt](<https://devfeed.tech/tags/prompt.md>), [qa](<https://devfeed.tech/tags/qa.md>)

### AI overview

Under a fixed prompt token budget, the study finds that using more short demonstrations generally improves in-context learning for question answering compared with spending the budget on a few long demonstrations. The authors grouped examples by length and tested six instruction-tuned models on Beat the AI and Natural Questions. The effect was strongest on Beat the AI; it also appeared on Natural Questions, where gains were smaller and saturated earlier with long contexts. In an ANLI R3 inference experiment, one of the two shorter length groups performed best for five of six models, though longer examples remained competitive. The results suggest demonstration length is a relevant prompt design choice alongside example count.

### Source excerpt

arXiv:2611.24971v1 Announce Type: cross Abstract: In-context learning (ICL) improves large language model performance at inference time without updating model parameters. However, these gains are not free: in practical deployments, longer prompts directly increase inference cost and latency. Under a fixed token budget, prompt construction therefore becomes a resource-allocation problem. While prior work has often focused on how many examples to include in ICL, we study the complementary question of whether the available budget is better spent on a few long demonstrations or on a larger number of shorter ones. We study this aspect in question answering (QA), by organizing candidate demonstrations into length-based bins and constructing prompts from examples within the same length range under fixed budgets. Experiments on two QA benchmarks (\textit{Beat the AI} and \textit{Natural Questions}), using six instruction-tuned LLMs, show that shorter demonstrations are generally the most effective use of prompt space. This effect is especially strong on \textit{Beat the AI}, and remains visible on NQ, although gains are smaller and saturate earlier in the long-context setting. An additional natural language inference experiment on ANLI R3 finds that one of the two shorter bins performs best for five of six models, although longer demonstrations remain competitive. Overall, our results highlight demonstration granularity as an important factor in prompt design.

## HapBO: Harnessed $\\alpha$-$\\pi$ Bayesian Optimization via Expert Natural Language Priors

DevFeed: [HapBO: Harnessed $\\alpha$-$\\pi$ Bayesian Optimization via Expert Natural Language Priors](<https://devfeed.tech/articles/hapbo-harnessed-alpha-pi-bayesian-optimization-via-expert-natural-language-priors-145008.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25100>)

Author: Jean-S\\'ebastien Giroux, Rose Guay Hottin, Marco Bonizzato

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [Operations research and optimization](<https://devfeed.tech/topics/operations-research-and-optimization.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [adversarial](<https://devfeed.tech/tags/adversarial.md>), [applications](<https://devfeed.tech/tags/applications.md>), [based](<https://devfeed.tech/tags/based.md>), [bayesian](<https://devfeed.tech/tags/bayesian.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [family](<https://devfeed.tech/tags/family.md>), [function](<https://devfeed.tech/tags/function.md>), [gaussian](<https://devfeed.tech/tags/gaussian.md>), [improvement](<https://devfeed.tech/tags/improvement.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [probability](<https://devfeed.tech/tags/probability.md>), [queries](<https://devfeed.tech/tags/queries.md>), [real-world](<https://devfeed.tech/tags/real-world.md>), [solar](<https://devfeed.tech/tags/solar.md>), [stat-ml](<https://devfeed.tech/tags/stat-ml.md>), [synthetic](<https://devfeed.tech/tags/synthetic.md>), [token-cost](<https://devfeed.tech/tags/token-cost.md>), [token-costs](<https://devfeed.tech/tags/token-costs.md>)

### AI overview

HapBO turns natural-language expert or LLM knowledge into a structured, reusable belief function, elicited once and used as a bounded positive multiplier on nonnegative Bayesian optimization acquisitions. Its expressive representation is not limited to Gaussian priors, and optimization requires no further LLM calls. For Expected Improvement, the authors prove the standard simple-regret rate is preserved up to a bounded factor, even with adversarial beliefs. Across 40 HPOBench tasks, a synthetic suite in 6-15 dimensions, and applications in cortical neurostimulation and constrained solar design, HapBO reports the lowest aggregate HPOBench regret and strong real-world performance. The abstract says it improves real-world outcomes at budgets of 12-20 queries, with lower wall-clock and token costs than other LLM-based methods.

### Source excerpt

arXiv:2611.25100v1 Announce Type: new Abstract: Bayesian Optimization (BO) can be substantially more sample-efficient when informed by prior knowledge about the problem. In practice, however, such knowledge is often available only in natural language, e.g., expert intuitions or documentation, rather than as the explicit probability distribution landscape that prior-guided BO methods may require. Leveraging it effectively demands both an expressive mapping from language to priors and robustness to information that may be misleading. We propose HapBO (Harnessed $\alpha$-$\pi$ Bayesian Optimization), which elicits beliefs once from a human expert, a large language model (LLM), or both, and deterministically compiles a structured specification into a reusable belief function ($\pi: \mathcal{X} \to[0,1]$), with a rich representation not restricted to a Gaussian family. Building on $\alpha$-$\pi$BO, HapBO applies this belief function through a bounded positive multiplier on any nonnegative acquisition, preserving positive acquisition support and requiring no LLM calls during optimization. We prove that, for the Expected Improvement acquisition function (EI), HapBO preserves the standard simple-regret rate up to a bounded factor, even under adversarial beliefs. We evaluate HapBO on 40 HPOBench tasks, a synthetic suite in 6-15 dimensions and two real-world applications: cortical neurostimulation and constrained solar design. HapBO achieves the lowest aggregate HPOBench regret and strong real-world task performance, with zero in-loop token cost. In real-world applications it improves the optimization outcome even at budgets of 12-20 queries, and at a small fraction of the wall-clock and token costs of other LLM-based methods.

## What the model knows but does not say: monitoring medical LLMs through verbalized thoughts

DevFeed: [What the model knows but does not say: monitoring medical LLMs through verbalized thoughts](<https://devfeed.tech/articles/what-the-model-knows-but-does-not-say-monitoring-medical-llms-through-verbalized-thoughts-145455.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24994>)

Author: Frederik Hauke, Jan Clusmann, Hanna Kreutzer, Christiane Kuhl, Tianyu Han, Felix Barajas Ordonez, Sven Nebelung, Jakob Nikolas Kather, Daniel Truhn

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM observability](<https://devfeed.tech/topics/llm-observability.md>), [LLM security](<https://devfeed.tech/topics/llm-security.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [autoencoder](<https://devfeed.tech/tags/autoencoder.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [gemma-3](<https://devfeed.tech/tags/gemma-3.md>), [llama-3-3](<https://devfeed.tech/tags/llama-3-3.md>), [llms](<https://devfeed.tech/tags/llms.md>), [medical](<https://devfeed.tech/tags/medical.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [thoughts](<https://devfeed.tech/tags/thoughts.md>)

### AI overview

The study reports that verbalized internal activations can reveal clinical prompt injections that ordinary answers conceal. A natural-language autoencoder converts Gemma-3-27B activations into text; an input-blind judge assesses that "thoughts" text and the model's answer separately. Across two attack objectives (omitting pathology or covertly recommending a drug), two input modalities, and two delivery mechanisms, the thoughts channel detected injections significantly more often in all eight conditions (p<0.05), while both channels had near-zero false positives on clean inputs. The omission finding also carried over to Llama-3.3-70B. The channel flagged unsafe requests while the model was still reading them. The approach used an off-the-shelf verbalizer and general-purpose judge, without retraining; the excerpt does not establish performance beyond these reported settings.

### Source excerpt

arXiv:2611.24994v1 Announce Type: cross Abstract: Efforts to keep clinical language models safe have concentrated on what enters and leaves the model, but a manipulated model can return an answer that looks entirely ordinary. Here we show that the model's own internal state, rendered in plain language, exposes manipulations the output conceals. Using a natural-language autoencoder, we translate the activations of a multimodal LLM (Gemma-3-27B) into text and let an input-blind judge read this verbalized internal state (the thoughts channel) and the model's answer separately. Across two clinical attack objectives (omitting a pathology, covertly recommending a drug), two input modalities, and two delivery mechanisms, the thoughts channel caught the injection significantly more often than the answer in all eight injected conditions (p<0.05), while both channels held near-zero false-positive rates on clean inputs; the omission result carried over to Llama-3.3-70B. The thoughts channel also registered unsafe clinical requests as harmful while the model was still reading them, before it decided whether to refuse. None of this requires retraining: an off-the-shelf verbalizer and a general-purpose judge are enough.

## Can LLMs Resolve Entity Conflicts in Domain Specific NLU Pipelines?

DevFeed: [Can LLMs Resolve Entity Conflicts in Domain Specific NLU Pipelines?](<https://devfeed.tech/articles/can-llms-resolve-entity-conflicts-in-domain-specific-nlu-pipelines-145095.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25733>)

Author: Akansha Bhardwaj, Mingbo Cui, Victor Guimaraes, Dominik Kohler, Diana Nicoleta Popa, Meghana M. Reddy, Paolo Rosso

Published: 2026-11-20T05:00:00Z

Content type: comparison

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Entity resolution](<https://devfeed.tech/topics/entity-resolution.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [lora](<https://devfeed.tech/topics/lora.md>), [Enhancing AI model performance](<https://devfeed.tech/topics/enhancing-ai-model-performance.md>)

Tags: [cases](<https://devfeed.tech/tags/cases.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [complex](<https://devfeed.tech/tags/complex.md>), [conflict-resolution](<https://devfeed.tech/tags/conflict-resolution.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [flexibility](<https://devfeed.tech/tags/flexibility.md>), [llama3](<https://devfeed.tech/tags/llama3.md>), [llms](<https://devfeed.tech/tags/llms.md>), [lora](<https://devfeed.tech/tags/lora.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [prompting](<https://devfeed.tech/tags/prompting.md>), [strategies](<https://devfeed.tech/tags/strategies.md>)

### AI overview

The study compares rule-based entity conflict resolution with two Llama3.1-8B-Instruct approaches in domain-specific NLU pipelines: multi-shot prompting and LoRA fine-tuning. It evaluates how well the models match existing pipeline outputs and, separately, their accuracy on human-labeled data. The abstract reports that prompting is easy to adapt and allows rapid behavioral changes, while fine-tuning performs better on complex, ambiguous cases. It suggests combining the approaches to pair prompting's flexibility with fine-tuning's broader accuracy gains.

### Source excerpt

arXiv:2611.25733v1 Announce Type: cross Abstract: Resolving entity conflicts is a critical step in Pipeline-based NLU architectures, especially when multiple extractors provide overlapping or conflicting outputs. Traditional rule-based entity conflict resolution methods often lack flexibility and are unscalable. In this work, we aim to study if the broad knowledge of LLMs can be leveraged to resolve entity conflicts in domain specific NLU pipelines. We present an experimental comparison between a traditional rule-based conflict resolution method, and two LLM-based approaches: a multi-shot prompting method using the Llama3.1-8B-Instruct model and a LoRA fine-tuned Llama3.1-8B-Instruct model. The evaluation was conducted under two setups: (1) a broad assessment of LLM's capability to match existing NLU pipeline output, and (2) a more controlled scenario assessing the LLM accuracy on human-labeled data. Results show that while multi-shot prompting offers ease of adaptation and rapid behavioral changes, the fine-tuned model delivers superior performance on complex, ambiguous cases. Ultimately, the best approach appears to be a combination of both strategies, leveraging multi-shot prompting's flexibility alongside the broad accuracy gains of LoRA fine-tuning for a more comprehensive performance.

## ForeSeek: Foresight-Driven Reasoning for Long-Form Video Understanding

DevFeed: [ForeSeek: Foresight-Driven Reasoning for Long-Form Video Understanding](<https://devfeed.tech/articles/foreseek-foresight-driven-reasoning-for-long-form-video-understanding-145624.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.26024>)

Author: Yuchen Sun, Siyuan Cheng, Shi-Zhe Chen, Yixin Qin, Jinwen Luo, Zheng Wei

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [object-detection](<https://devfeed.tech/topics/object-detection.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [global](<https://devfeed.tech/tags/global.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [training](<https://devfeed.tech/tags/training.md>), [video](<https://devfeed.tech/tags/video.md>)

### AI overview

ForeSeek replaces reactive clip retrieval in long-form video agents with predictive exploration: it forms visually grounded expectations about unseen video segments, checks them through targeted observations, and refines its search as evidence emerges. Failed expectations guide backtracking. A two-stage training framework teaches multimodal language models foresight-guided reasoning and efficient decisions. Across five long-video benchmarks, the authors report competitive or better accuracy than the strongest search-based agent while observing 2.5-4.4 times fewer frames.

### Source excerpt

arXiv:2611.26024v1 Announce Type: new Abstract: Long-form video understanding, as a fundamental yet crucial vision task, is being revolutionized by multimodal large language model (MLLM)-based video agents. However, the retrieval-driven nature inherent to these agents creates a significant bottleneck, forcing them to reactively fetch clips based on local saliency without a global temporal plan. To address this, we present ForeSeek, a foresight-driven video agent that transforms long-video reasoning from reactive retrieval into predictive exploration. Rather than blindly probing the timeline, ForeSeek forms visually grounded expectations over unseen temporal regions, verifies them through targeted observations, and continuously refines its search based on emerging evidence, where failed expectations serve as informative signals for principled backtracking. Furthermore, we develop a two-stage training framework that equips MLLMs with foresight-guided reasoning and efficient decision making. Extensive experiments on five long-video benchmarks demonstrate that ForeSeek consistently improves the efficiency--accuracy trade-off of long-video reasoning, achieving competitive or superior accuracy while observing $2.5$--$4.4\times$ fewer frames than the strongest search-based agent. Code is available at https://github.com/harry0021/ForeSeek

## The Post-Training Dilemma: Why We Should Rethink the Sequential SFT-RL Paradigm

DevFeed: [The Post-Training Dilemma: Why We Should Rethink the Sequential SFT-RL Paradigm](<https://devfeed.tech/articles/the-post-training-dilemma-why-we-should-rethink-the-sequential-sft-rl-paradigm-144871.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2601.07389>)

Author: Xueyan Niu, Bo Bai, Wei Han, Weixi Zhang

Published: 2026-11-20T05:00:00Z

Content type: opinion

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [post-training](<https://devfeed.tech/topics/post-training.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [condition](<https://devfeed.tech/tags/condition.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-it](<https://devfeed.tech/tags/cs-it.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [gradient](<https://devfeed.tech/tags/gradient.md>), [hh](<https://devfeed.tech/tags/hh.md>), [language](<https://devfeed.tech/tags/language.md>), [language-model](<https://devfeed.tech/tags/language-model.md>), [large-language-model](<https://devfeed.tech/tags/large-language-model.md>), [learning](<https://devfeed.tech/tags/learning.md>), [llm](<https://devfeed.tech/tags/llm.md>), [math-500](<https://devfeed.tech/tags/math-500.md>), [math-it](<https://devfeed.tech/tags/math-it.md>), [mechanism](<https://devfeed.tech/tags/mechanism.md>), [misalignment](<https://devfeed.tech/tags/misalignment.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [rl](<https://devfeed.tech/tags/rl.md>), [sft](<https://devfeed.tech/tags/sft.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

The paper argues that sequential SFT-then-RL training can raise the SFT loss because the two objectives have nearly orthogonal gradients, a conflict the authors say hyperparameter tuning or a KL penalty cannot resolve. Under the Polyak-Lojasiewicz condition, it gives a lower bound on this increase based on distance from the SFT-optimal parameter set. Experiments with Qwen3-0.6B report higher SFT loss after RL on MATH-500, Alpaca, HH-RLHF, and SST-2; the authors also describe SFT as flattening and RL as re-sharpening the loss landscape. They propose joint optimization with gradient surgery, landscape-aware RL stopping, and studying the model-scale threshold where SFT may be unnecessary.

### Source excerpt

arXiv:2601.07389v3 Announce Type: replace Abstract: This position paper argues that the sequential supervised fine-tuning followed by reinforcement learning (SFT-then-RL) pipeline---the de facto paradigm for large language model (LLM) post-training---has a fundamental property that cannot be fixed by hyperparameter tuning: SFT and RL optimize over nearly orthogonal parameter subspaces, so RL inevitably degrades the capabilities acquired during the preceding SFT phase. Under the Polyak-Lojasiewicz (PL) condition, which is empirically observed for over-parameterized networks, we show that RL increases the SFT loss by at least $\frac{\mu}{2}\mathrm{dist}(\theta_{\mathrm{RL}}, \Theta^*_{\mathrm{SFT}})^2$ for some $\mu>0$. Although the KL penalty can constrain distributional shift, it cannot correct the underlying gradient misalignment between the two objectives. Experiments on Qwen3-0.6B across four tasks confirm the theoretical predictions: on MATH-500, Alpaca, HH-RLHF, and SST-2, the SFT loss increases after the RL phase. We identify the underlying mechanism as persistent near-orthogonality between the gradients of the two objectives, paired with opposing landscape dynamics: SFT flattens the loss landscape while RL re-sharpens it. Based on our analysis, we propose three promising research directions: (1) joint optimization of the two objectives with gradient surgery to resolve conflicting updates; (2) landscape-aware scheduling that terminates RL training at the theoretically optimal point; and (3) characterizing the model scale threshold above which SFT becomes unnecessary.

## LLM-Based Threat Detection and Prevention Framework for IoT Ecosystems

DevFeed: [LLM-Based Threat Detection and Prevention Framework for IoT Ecosystems](<https://devfeed.tech/articles/llm-based-threat-detection-and-prevention-framework-for-iot-ecosystems-144852.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2505.00240>)

Author: Yazan Otoum, Arghavan Asad, Ishtiaq Ahmad, Amiya Nayak

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Cryptography and Security](<https://devfeed.tech/sources/arxiv-cryptography-and-security.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [threat detection](<https://devfeed.tech/topics/threat-detection.md>), [Internet of things](<https://devfeed.tech/topics/iot.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [actions](<https://devfeed.tech/tags/actions.md>), [adaptive](<https://devfeed.tech/tags/adaptive.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [bert](<https://devfeed.tech/tags/bert.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cr](<https://devfeed.tech/tags/cs-cr.md>), [cs-et](<https://devfeed.tech/tags/cs-et.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cybersecurity](<https://devfeed.tech/tags/cybersecurity.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [decision-making](<https://devfeed.tech/tags/decision-making.md>), [detection](<https://devfeed.tech/tags/detection.md>), [docker](<https://devfeed.tech/tags/docker.md>), [ecosystems](<https://devfeed.tech/tags/ecosystems.md>), [edge-cloud](<https://devfeed.tech/tags/edge-cloud.md>), [experimental](<https://devfeed.tech/tags/experimental.md>), [f1](<https://devfeed.tech/tags/f1.md>), [framework](<https://devfeed.tech/tags/framework.md>), [iot](<https://devfeed.tech/tags/iot.md>), [llm](<https://devfeed.tech/tags/llm.md>), [threat](<https://devfeed.tech/tags/threat.md>), [threat-detection](<https://devfeed.tech/tags/threat-detection.md>)

### AI overview

The proposed IoT security framework pairs a detection component with a prevention component: fine-tuned models trained on IoT-23 and TON_IoT identify anomalous network behavior, while a Prevention LLM selects mitigation actions. Prompt management, local adaptation, and global refinement support context-aware decisions and ongoing model-guided adaptation. In Docker-based edge-cloud experiments, BERT Small reportedly achieved 99.75% accuracy, F1-score, precision, and recall for threat detection; the Prevention LLM achieved 98.3% accuracy in selecting suitable mitigations. These are reported experimental results, and the supplied abstract does not describe broader deployment or validation.

### Source excerpt

arXiv:2505.00240v3 Announce Type: replace-cross Abstract: The rapid expansion of the Internet of Things (IoT) has introduced complex and evolving cybersecurity challenges. This paper presents a Large Language Model (LLM)-based framework for intelligent threat detection and prevention in IoT networks. The proposed architecture integrates two core components: a Detection Component that leverages fine-tuned models trained on IoT-specific datasets (IoT-23 and TON_IoT) to identify anomalous network behaviours, and a Prevention Component that generates adaptive response strategies to mitigate detected threats. The framework incorporates prompt management, local adaptation, and global refinement modules to support context-aware decision-making and continuous model-guided adaptation. Experimental validation in a Docker-based edge-cloud environment shows that BERT Small achieves 99.75% accuracy, F1-score, precision, and recall for threat detection, while the Prevention LLM achieves 98.3% accuracy in selecting suitable mitigation actions. These results highlight the potential of LLM-driven architectures to provide explainable and adaptive security solutions for next-generation IoT ecosystems.

## PosteriorBench: Evaluating Behavior Posterior Sampling in Language Models

DevFeed: [PosteriorBench: Evaluating Behavior Posterior Sampling in Language Models](<https://devfeed.tech/articles/posteriorbench-evaluating-behavior-posterior-sampling-in-language-models-145096.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25735>)

Author: Zhiyang Xun, Daniel D. Johnson, Sarah Schwettmann, Neil Chowdhury

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [LLM Techniques](<https://devfeed.tech/topics/llm-techniques.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [qwen](<https://devfeed.tech/topics/qwen.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [compute](<https://devfeed.tech/tags/compute.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [fine-tuning](<https://devfeed.tech/tags/fine-tuning.md>), [kl-divergence](<https://devfeed.tech/tags/kl-divergence.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [prompting](<https://devfeed.tech/tags/prompting.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [responses](<https://devfeed.tech/tags/responses.md>), [sampling](<https://devfeed.tech/tags/sampling.md>), [tasks](<https://devfeed.tech/tags/tasks.md>), [training](<https://devfeed.tech/tags/training.md>), [transcripts](<https://devfeed.tech/tags/transcripts.md>)

### AI overview

PosteriorBench evaluates methods for sampling language model transcripts conditioned on rare behaviors, measuring compute cost and fidelity to the true posterior with forward and reverse KL divergence. Its 21 tasks use Qwen3.5-0.8B and cover behavior probabilities from 10⁻⁵ to 10⁻¹, with exact posterior samples drawn from a corpus of one billion labeled responses. In the evaluated methods, prompting is cheap but remains far from the posterior, while fine-tuning on posterior samples is often too costly; several prior training methods do no better. No evaluated method combines efficiency and fidelity, leaving cheap, faithful posterior sampling an open problem.

### Source excerpt

arXiv:2611.25735v1 Announce Type: new Abstract: Rare language model behaviors can be difficult to study because obtaining them through ordinary sampling can require a very large number of samples. This motivates the behavior posterior sampling problem: given a language model and a behavior of interest, is there an algorithm that efficiently samples from the model's distribution over transcripts that exhibit this behavior? We introduce PosteriorBench, a benchmark for comparing algorithms by their compute cost and by how close their samples are to the true posterior in forward and reverse KL divergence. PosteriorBench contains 21 tasks on Qwen3.5-0.8B with behavior probabilities from $10^{-5}$ to $10^{-1}$, each with exact posterior samples from a corpus of one billion labeled responses. We find that no method we evaluate is both efficient and faithful: simple methods like prompting are cheap but plateau at a high KL divergence from the posterior, fine-tuning on posterior samples is often cost-prohibitive, and several training methods from prior work do not perform better. Building a posterior sampler that can cheaply approximate the true posterior remains an open problem, and we hope PosteriorBench can measure progress towards this objective. Code and data are available at https://github.com/TransluceAI/PosteriorBench.

## ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

DevFeed: [ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining](<https://devfeed.tech/articles/roam-grpo-enforcing-reasoning-diversity-in-grpo-via-routed-off-mode-approach-mining-145610.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25926>)

Author: Jeff Duan, Matt Thomson

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [grpo](<https://devfeed.tech/topics/grpo.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [rlvr](<https://devfeed.tech/topics/rlvr.md>), [post-training](<https://devfeed.tech/topics/post-training.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [creativity](<https://devfeed.tech/tags/creativity.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [diversity](<https://devfeed.tech/tags/diversity.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [grpo](<https://devfeed.tech/tags/grpo.md>), [math](<https://devfeed.tech/tags/math.md>), [mining](<https://devfeed.tech/tags/mining.md>), [post-training](<https://devfeed.tech/tags/post-training.md>), [problems](<https://devfeed.tech/tags/problems.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [reasoning](<https://devfeed.tech/tags/reasoning.md>), [reinforcement-learning](<https://devfeed.tech/tags/reinforcement-learning.md>), [rlvr](<https://devfeed.tech/tags/rlvr.md>), [sft](<https://devfeed.tech/tags/sft.md>), [strategies](<https://devfeed.tech/tags/strategies.md>), [supervised-fine-tuning](<https://devfeed.tech/tags/supervised-fine-tuning.md>), [well](<https://devfeed.tech/tags/well.md>)

### AI overview

ROAM-GRPO aims to counter reasoning-mode collapse in GRPO by routing well-solved problems to an "off-mode" track. The model generates distinct solution strategies, an LLM judge checks their logical validity and distinctness, and accepted strategies are self-distilled into the policy. In the reported experiment, training Qwen3.5-2b on 1,000 MATH level-5 problems increased peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set. The supplied text does not detail the evaluation setup or broader limitations.

### Source excerpt

arXiv:2611.25926v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.

## From Industrial Product Documentation to Automated Digital Product Passport Generation: An Experimental Study of Large Language Models

DevFeed: [From Industrial Product Documentation to Automated Digital Product Passport Generation: An Experimental Study of Large Language Models](<https://devfeed.tech/articles/from-industrial-product-documentation-to-automated-digital-product-passport-generation-an-experimental-study-of-large-language-models-145733.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.24998>)

Author: Marco Buecheler, Grant Richards, Nathan Hartman

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [Vectors & LLMs](<https://devfeed.tech/topics/vectors-llms.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [hybrid-search](<https://devfeed.tech/topics/hybrid-search.md>), [Factuality scoring in AI](<https://devfeed.tech/topics/factuality-scoring-in-ai.md>)

Tags: [automated](<https://devfeed.tech/tags/automated.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [digital](<https://devfeed.tech/tags/digital.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [experimental](<https://devfeed.tech/tags/experimental.md>), [generation](<https://devfeed.tech/tags/generation.md>), [industrial](<https://devfeed.tech/tags/industrial.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [passport](<https://devfeed.tech/tags/passport.md>), [product](<https://devfeed.tech/tags/product.md>), [study](<https://devfeed.tech/tags/study.md>)

### AI overview

The study tests a RAG pipeline for generating Digital Product Passports as Asset Administration Shell instances from scattered industrial documentation. It chunks documents, stores embeddings in ChromaDB, combines semantic and lexical retrieval against AAS submodel requirements, and supplies retrieved content to five LLMs. Across 4,560 executions covering 76 product-submodel combinations, 97.4% of outputs were valid JSON, but 44.9% passed full AAS validation. Deterministic scoring found mean completeness of 49.6%, conditional accuracy of 58.3%, and overall field correctness of 30.8%; an LLM judge rated the same outputs higher at 66.5%, 88.9%, and 60.4%. The abstract says four prompt-guidance levels were evaluated but does not report their comparative effects.

### Source excerpt

arXiv:2611.24998v1 Announce Type: cross Abstract: Manufacturing companies that are planning to sell certain products in the European Union (EU) will be required to provide mandatory digital product passports (DPPs). The use of DPPs is to provide standardized product information throughout multiple product life cycles in a circular economy. Required passport information, such as technical specifications, material composition, and environmental product footprint, is available in manufacturers' data sources but is scattered across different heterogeneous systems, documents, and data formats. The given paper investigates using large language models (LLMs) to automate the generation of DPPs based on the asset administration shell (AAS). A retrieval-augmented generation (RAG) approach is used to extract product information from industrial documentation and generate populated AAS instances. Product documentation is processed and divided into text chunks. The text chunks are then stored in a persistent ChromaDB vector database using vector embeddings. The retrieval process combines semantic and lexical search to identify relevant information based on standardized AAS submodel template requirements. The retrieved information is provided to five LLMs to generate structured AAS instances. The experiments cover different industrial products and AAS submodels. Four levels of prompt guidance are investigated to assess their influence on generation quality. The generated passports are evaluated against a ground truth. The experimental evaluation includes 4,560 executions across 76 product-submodel combinations. Of the generated passport outputs, 97.4% are valid JSON, while 44.9% pass the full AAS validation procedure. Under deterministic scoring, mean completeness, conditional accuracy, and overall field correctness are 49.6%, 58.3%, and 30.8%, respectively. An LLM-as-a-judge yields better scores of 66.5%, 88.9%, and 60.4% for the same outputs.

## Moving Generality from Model to Modeler: LLM Agents for PHM on a Deployment-Style Benchmark

DevFeed: [Moving Generality from Model to Modeler: LLM Agents for PHM on a Deployment-Style Benchmark](<https://devfeed.tech/articles/moving-generality-from-model-to-modeler-llm-agents-for-phm-on-a-deployment-style-benchmark-145028.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25252>)

Author: Zhenling Chen, Zhiguo Zeng, Anne Barros

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [generalization in machine learning](<https://devfeed.tech/topics/generalization-in-machine-learning.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [benchmark overfitting machine learning](<https://devfeed.tech/topics/benchmark-overfitting-machine-learning.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [agents](<https://devfeed.tech/tags/agents.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [automl](<https://devfeed.tech/tags/automl.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [design](<https://devfeed.tech/tags/design.md>), [domain](<https://devfeed.tech/tags/domain.md>), [equipment](<https://devfeed.tech/tags/equipment.md>), [error](<https://devfeed.tech/tags/error.md>), [health](<https://devfeed.tech/tags/health.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [language](<https://devfeed.tech/tags/language.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [model](<https://devfeed.tech/tags/model.md>)

### AI overview

On NextUnit-Bench, frontier LLM coding agents outperform AutoML and most faithfully implemented published methods at building predictors for unseen equipment. The benchmark covers 24 public datasets across 58 unit- or condition-holdout folds, with two hours to produce an executable predictor. The reported advantage comes mainly from learning each machine's data: a short plain-text plan captures much of the gap between frontier agents, while the strongest agent performs at a weaker agent's level when given its plan. Pre-data choice of plan author and domain toolkits appear to matter little. A locally deployable open-weight model executes the strongest agent's designs only half as often and scores no better than without them. Reported published-method accuracies depend on each paper's conventions; capping test engines' remaining life can halve reported error.

### Source excerpt

arXiv:2611.25252v1 Announce Type: new Abstract: Coding agents built on large language models (LLMs) now help develop many models published as general solutions. Should generality instead reside in the agent that builds a model for each new task? We examine this question in prognostics and health management (PHM), where every model must work on unseen equipment. Our deployment-style benchmark, NextUnit-Bench, covers 24 public datasets split into 58 folds that hold out units or operating conditions; each agent has two hours to deliver an executable predictor, then run on the hidden units. First, frontier agents outperform AutoML, which reaches less than half the best agent's score, and faithful implementations of published methods on four datasets, of which only 6 of 30 beat the strongest agent's own design; published accuracies, where reproducible, rest on the papers' own conventions, and a single convention, capping the remaining life of test engines, can halve the reported error. Second, the agents' advantage is a design made on each machine's data; once made, a plain-text plan of a few hundred words carries most of the gap between two frontier agents, and given a weaker agent's plan, the strongest agent falls to that agent's level, often to identical predictions. Before touching the data, however, it matters little whose plan an agent carries out, and domain toolkits appear to help neither: what sets the strongest apart is how much it learns from the data. Third, such designs reach only implementers that can execute them: given the strongest agent's designs, a locally deployable open-weight model delivers half as often, even when they are lightened, and scores no higher than without them. Generality, we argue, is not disappearing but moving, from the model to the modeler.

## AutoRAGBench: A Benchmark for LLM Agents Automating RAG System Development

DevFeed: [AutoRAGBench: A Benchmark for LLM Agents Automating RAG System Development](<https://devfeed.tech/articles/autoragbench-a-benchmark-for-llm-agents-automating-rag-system-development-145541.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25582>)

Author: Hang Zhang, Yuzhi Pan, Ziyao Zhong, Chaokun Wang, Shuo Cao, Zeyu Huang, Yue Xue, Fang Niu, Bofan Xie, Guanchen Ge, Zecheng Li, Leqi Zheng, Ziyang Liu, Wei Wang, Miao Gu

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llm-agents](<https://devfeed.tech/tags/llm-agents.md>), [rag](<https://devfeed.tech/tags/rag.md>), [system-development](<https://devfeed.tech/tags/system-development.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

AutoRAGBench evaluates whether LLM agents can build and improve retrieval-augmented generation systems across nine development tasks. Agents worked in an open code space without starter code or prescribed strategies, with six hours and one H800 GPU per task; they analyzed data, implemented pipelines, and refined them using experimental feedback. Across nine configurations, GPT-6.1 Sol with Codex CLI had the highest average score (56.1), compared with 44.4 for the composite Human-SOTA baseline. Starting from human-designed systems on seven tasks, it improved scores by an average of 16.6 points. These results support the benchmark's finding that agents can automate effective RAG development under the tested conditions.

### Source excerpt

arXiv:2611.25582v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) is a well-established paradigm for augmenting large language models (LLMs) with information retrieved from external corpora. However, developing effective RAG systems requires substantial engineering effort and iterative experimentation, as effective design choices vary across tasks and domains. With rapid advances in LLM agent capabilities, we ask how well these agents can autonomously build RAG systems and how effectively they can improve them through iteration. In this paper, we propose AutoRAGBench, the first benchmark to evaluate how well LLM agents develop RAG systems from scratch in an open code space. It comprises nine RAG system development tasks. For each task, agents build RAG systems from scratch without starter code or prescribed strategies, under a six-hour budget on a single H800 GPU. They analyze task data, implement RAG pipelines, and iteratively refine their designs based on experimental feedback. We evaluate nine agent configurations and analyze both the effectiveness of the resulting RAG systems and their development trajectories. GPT-6.1 Sol with Codex CLI achieves the highest average score of 56.1. The composite Human-SOTA baseline averages 44.4. These results show that LLM agents can automate effective RAG system development. When starting from a human-designed system on each of seven tasks, GPT-6.1 Sol with Codex CLI improves the score by 16.6 points on average.

## SPACE: Source-free Proxy Anchor Concept Erasure for MLLMs

DevFeed: [SPACE: Source-free Proxy Anchor Concept Erasure for MLLMs](<https://devfeed.tech/articles/space-source-free-proxy-anchor-concept-erasure-for-mllms-144904.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.09868>)

Author: Zhijing Zhang, Jiaqi Ding, Qianshan Wei, Nan Zhou, Jiaqi Li, Yongliang Wu, Tongxin Zhu, Xiaolin Fang

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [Model Development](<https://devfeed.tech/topics/model-development.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>)

Tags: [code](<https://devfeed.tech/tags/code.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-retention](<https://devfeed.tech/tags/data-retention.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [entropy](<https://devfeed.tech/tags/entropy.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [knowledge](<https://devfeed.tech/tags/knowledge.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [machine-unlearning](<https://devfeed.tech/tags/machine-unlearning.md>), [multimodal](<https://devfeed.tech/tags/multimodal.md>), [privacy](<https://devfeed.tech/tags/privacy.md>), [proxy](<https://devfeed.tech/tags/proxy.md>), [regulatory](<https://devfeed.tech/tags/regulatory.md>)

### AI overview

SPACE is a source-free machine unlearning framework for multimodal large language models that aims to erase target concepts without access to their visual data. Text-Guided Proxy Anchor Selection retrieves semantically aligned proxy anchors from the shared feature space; Dual-Constraint Semantic Isolation then optimizes those anchors to indirectly erase the concepts while confining updates to the null space of retained knowledge. The authors state that this bounds perturbations to retained knowledge and maximizes feature spectral entropy. Experiments across six datasets reportedly achieve performance comparable to state-of-the-art methods that use target data.

### Source excerpt

arXiv:2606.09868v2 Announce Type: replace Abstract: As Multimodal Large Language Models (MLLMs) face growing privacy risks and regulatory constraints, machine unlearning (MU) has emerged as a crucial solution for removing sensitive data while preserving model performance. However, existing MU methods typically rely on visual data of the target concepts, which is often unavailable due to strict data retention policies, thus creating a demand for source-free unlearning approaches that operate without access to the target data. In this work, we propose Source-free Proxy Anchor Concept Erasure (SPACE), the first source-free unlearning framework specialized for MLLMs. SPACE consists of two stages: (1) Text-Guided Proxy Anchor Selection (TPAS), which retrieves semantically aligned proxy anchors from the shared feature space. (2) Dual-Constraint Semantic Isolation (DCSI), which optimizes these anchors to indirectly erase target concepts. DCSI confines updates to the null space of retained knowledge, ensuring structural integrity. We theoretically prove that SPACE strictly bounds the perturbation on retained knowledge and maximizes feature spectral entropy, thereby maintaining the model's performance. Furthermore, extensive experiments across six datasets show that SPACE achieves performance comparable to that of state-of-the-art data-dependent methods, validating its effectiveness in source-free MU scenarios. The source code will be released.

## A four-level executable validation protocol for evaluating LLM-generated topology specifications of physics-based HVAC digital twins

DevFeed: [A four-level executable validation protocol for evaluating LLM-generated topology specifications of physics-based HVAC digital twins](<https://devfeed.tech/articles/a-four-level-executable-validation-protocol-for-evaluating-llm-generated-topology-specifications-of-physics-based-hvac-digital-twins-145061.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25471>)

Author: Younghwan Joo, Jeonghoon Han, Sanghyun Oh, Soosik Bang, Sung-il Kim

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Robotics Simulation](<https://devfeed.tech/topics/robotics-simulation.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [cs-sy](<https://devfeed.tech/tags/cs-sy.md>), [digital-twins](<https://devfeed.tech/tags/digital-twins.md>), [eess-sy](<https://devfeed.tech/tags/eess-sy.md>), [llm](<https://devfeed.tech/tags/llm.md>), [physics](<https://devfeed.tech/tags/physics.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [specifications](<https://devfeed.tech/tags/specifications.md>), [topology](<https://devfeed.tech/tags/topology.md>), [validation](<https://devfeed.tech/tags/validation.md>)

### AI overview

The four-level protocol distinguishes executable HVAC digital twins from physically valid ones by checking static validity, model build, simulation convergence, and behavioral equivalence to a hidden reference within bands calibrated using ±20% sizing perturbations. Across eight LLMs and fourteen topologies, seven models built in 98.6-100% of trials after up to three repairs and simulations converged in at least 80%, but behavioral equivalence ranged from 50.0-98.6%. About half of failed trials still converged to a steady state different from the reference, mostly for desiccant rotor systems. Pooled pass rates differed by under five percentage points across description authors. The abstract argues that difficult domain physics should be represented explicitly in validation levels; it proposes adaptation to other fields that generate executable digital twins.

### Source excerpt

arXiv:2611.25471v1 Announce Type: cross Abstract: Physics-based digital twins of air conditioning, a principal energy consumer in manufacturing and buildings, can be generated automatically from a declarative specification of its equipment, connections, and sizing. Large language models (LLMs) now write such specifications from plain-language descriptions. Evaluations of these specifications stop at parse success, completed runs, or textual similarity, and cannot reject a model that converges in simulation but is physically wrong. In this study, a four-level protocol is proposed that tests whether an LLM-written specification not only executes but also yields a physically valid digital twin. Each specification is executed through static validity, model build, simulation convergence, and behavioral equivalence with a hidden reference model within bands calibrated by $\pm$20% sizing perturbations. The protocol was applied to eight LLMs and fourteen HVAC topologies, five describing an existing low-humidity test chamber. To test for authorship bias, engineers and a second model family also described six topologies. Within three repairs, seven of eight models built in 98.6 to 100% of trials, with simulations converging in at least 80.0%, but reached behavioral equivalence in only 50.0 to 98.6%. About half of the failed trials converged, but not to the reference steady state, mostly in desiccant rotor systems. Pooled pass rates varied by under five percentage points across description authors. These results imply that the physics hardest to model in each field, here the desiccant rotor, must enter the generation framework as validation levels. The protocol can be adapted to other fields that generate executable digital twins from declarative specifications.

[Next page](<https://devfeed.tech/topics/llm.md?cursor=WyIyMDI2LTExLTIwVDA1OjAwOjAwKzAwOjAwIiwgImJlOGI2YTcyLTQyNjQtNDQwYS1hOTRmLTM5MmM2NzU2ZWNhZCJd>)