# search-engines

Search engines are information retrieval systems that crawl and index documents, then return ranked results for user queries.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## LINEAGE: A Benchmark for Empirical Lineage Retrieval in AI Scientific Literature

DevFeed: [LINEAGE: A Benchmark for Empirical Lineage Retrieval in AI Scientific Literature](<https://devfeed.tech/articles/lineage-a-benchmark-for-empirical-lineage-retrieval-in-ai-scientific-literature-145594.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25845>)

Author: Pangjing Wu, Xu Yuan, Wenqi Fan, Qing Li

Published: 2026-11-20T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Graphs](<https://devfeed.tech/topics/graphs.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [2018](<https://devfeed.tech/tags/2018.md>), [2025](<https://devfeed.tech/tags/2025.md>), [abstract](<https://devfeed.tech/tags/abstract.md>), [academic](<https://devfeed.tech/tags/academic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [components](<https://devfeed.tech/tags/components.md>), [constructor](<https://devfeed.tech/tags/constructor.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [experiment](<https://devfeed.tech/tags/experiment.md>), [experiments](<https://devfeed.tech/tags/experiments.md>), [graph](<https://devfeed.tech/tags/graph.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [literature](<https://devfeed.tech/tags/literature.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>)

### AI overview

Lineage benchmarks retrieval of prior AI research methods explicitly used as empirical comparison baselines, a relation that topical relevance and citation links alone do not capture. Its lineage graphs provide multi-hop retrieval ground truth, and Lineage-Constructor builds calibrated graphs from heterogeneous paper records and full-paper baseline evidence. The benchmark covers 67,502 papers, 164,948 lineage edges, and 80,000 multi-facet queries, drawn from 121,048 papers across 17 AI venues from 2018-2025. Tests of 19 retrieval systems found that performance drops sharply with lineage distance, and isolated hits often fail to recover closely connected comparison neighborhoods.

### Source excerpt

arXiv:2611.25845v1 Announce Type: cross Abstract: Current research agents are moving beyond academic search toward literature synthesis and experiment planning. These workflows increasingly rely on literature retrieval systems, which serve as standalone systems or as components of research agents. However, existing scientific literature retrieval benchmarks mainly evaluate these methods using topical relevance or citation links as supervision, which cannot capture which prior methods are used for empirical comparison. To address this gap, we introduce Lineage, a benchmark for empirical lineage retrieval in AI scientific literature. An empirical lineage relation links a prior paper to a source paper that explicitly uses its method as an empirical comparison baseline. These relations form lineage graphs whose multi-hop neighborhoods provide retrieval ground truth for structural evaluation. To construct these graphs at scale, we develop Lineage-Constructor, which converts heterogeneous paper records and full-paper empirical baseline evidence into calibrated lineage graphs. Built from 121,048 papers across 17 top AI venues from 2018 to 2025, Lineage contains 3,950 graph components, 67,502 paper nodes, 164,948 lineage edges, and 80,000 multi-facet retrieval queries. Experiments with 19 representative retrieval systems show that empirical lineage retrieval remains challenging. Performance drops sharply with lineage distance, while structural metrics reveal that isolated retrieval hits often fail to recover closely connected empirical comparison neighborhoods.

## Web search in Latin America: language models use less local and diverse news than search engines

DevFeed: [Web search in Latin America: language models use less local and diverse news than search engines](<https://devfeed.tech/articles/web-search-in-latin-america-language-models-use-less-local-and-diverse-news-than-search-engines-145613.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.25955>)

Author: Francisco W. Kerche, Franziska Sofia Hafner, Felix M. Simon, Luc Rocher

Published: 2026-11-20T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [amplify](<https://devfeed.tech/tags/amplify.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [information-systems](<https://devfeed.tech/tags/information-systems.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [latin-america](<https://devfeed.tech/tags/latin-america.md>), [news](<https://devfeed.tech/tags/news.md>), [search](<https://devfeed.tech/tags/search.md>), [search-engines](<https://devfeed.tech/tags/search-engines.md>)

### AI overview

The study finds that language models return less diverse news sources than search engines when answering queries about Latin America. Researchers tested three search engines and four language models on 1,901 current events across nine countries, varying query language and user location. For local users, search engines returned 41.3% domestic news, while language models returned 3.6 to 11.6 percentage points less; domestic-news shares varied across user types. The findings suggest LLM-based search may reproduce or sometimes amplify geographical imbalances in news visibility. The researchers released an open dataset for multilingual and geographically diverse evaluation.

### Source excerpt

arXiv:2611.25955v1 Announce Type: cross Abstract: AI platforms are becoming a major gateway to news and current events. This shift risks amplifying existing geographical biases that favour narratives from wealthier countries over those from the Global South. This is particularly relevant in Latin America, which accounts for the largest share of AI usage in the Global South. We examine how information about Latin America changes when users shift from conventional search engines to LLM-based search. We construct a dataset of 1,901 current events across nine Latin American countries, craft realistic queries, and submit them to three search engines and four language models. We vary query language and user location to represent local, diaspora, tourist, and foreign user scenarios. Across 49,207 requests and 8,834 unique domains, language models return a less diverse set of news sources than search engines. Across user types, the share of domestic news returned by language models is inconsistent. For local users, while search engines already return only 41.3\% local news, language models return a further 3.6 to 11.6 percentage points less. Taken together, our findings suggest that LLM-based search can reproduce and, in some settings, amplify geographical hierarchies in the reach and visibility of news sources. We release an open dataset to support multilingual and geographically diverse evaluation of LLM-based search, and highlight implications for more equitable information systems design.

## I Found My Own Website Blocking ChatGPT. Then I Checked 640 Others.

DevFeed: [I Found My Own Website Blocking ChatGPT. Then I Checked 640 Others.](<https://devfeed.tech/articles/i-found-my-own-website-blocking-chatgpt-then-i-checked-640-others-143906.md>)

Original publisher: [Read original article](<https://hackernoon.com/i-found-my-own-website-blocking-chatgpt-then-i-checked-640-others?source=rss>)

Author: Sean Garschi (Growthr)

Published: 2026-11-19T12:34:16Z

Content type: tutorial

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Website](<https://devfeed.tech/topics/website.md>), [Cloudflare](<https://devfeed.tech/topics/cloudflare.md>), [hosting](<https://devfeed.tech/topics/hosting.md>), [proxy](<https://devfeed.tech/topics/proxy.md>), [Search engine optimization (SEO)](<https://devfeed.tech/topics/seo.md>), [ChatGPT](<https://devfeed.tech/topics/chatgpt.md>)

Tags: [403](<https://devfeed.tech/tags/403.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-bot-management](<https://devfeed.tech/tags/ai-bot-management.md>), [ai-bots](<https://devfeed.tech/tags/ai-bots.md>), [ai-crawler-blocking](<https://devfeed.tech/tags/ai-crawler-blocking.md>), [ai-crawlers](<https://devfeed.tech/tags/ai-crawlers.md>), [ai-search](<https://devfeed.tech/tags/ai-search.md>), [ai-search-engine-optimization](<https://devfeed.tech/tags/ai-search-engine-optimization.md>), [automated](<https://devfeed.tech/tags/automated.md>), [blocking](<https://devfeed.tech/tags/blocking.md>), [bot-management](<https://devfeed.tech/tags/bot-management.md>), [bots](<https://devfeed.tech/tags/bots.md>), [browser](<https://devfeed.tech/tags/browser.md>), [cache](<https://devfeed.tech/tags/cache.md>), [cdn](<https://devfeed.tech/tags/cdn.md>), [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [claudebot](<https://devfeed.tech/tags/claudebot.md>), [cloudflare-ai-bots](<https://devfeed.tech/tags/cloudflare-ai-bots.md>), [crawler](<https://devfeed.tech/tags/crawler.md>), [gptbot](<https://devfeed.tech/tags/gptbot.md>), [robots](<https://devfeed.tech/tags/robots.md>), [robots-txt](<https://devfeed.tech/tags/robots-txt.md>), [sites](<https://devfeed.tech/tags/sites.md>), [technical-seo-audit](<https://devfeed.tech/tags/technical-seo-audit.md>), [web-crawlers](<https://devfeed.tech/tags/web-crawlers.md>), [website](<https://devfeed.tech/tags/website.md>), [website-crawler-access](<https://devfeed.tech/tags/website-crawler-access.md>)

### AI overview

The author found two independent layers blocking AI crawlers: Cloudflare's user-agent setting and Hostinger's origin firewall, which intermittently rejected PerplexityBot IPs on cache misses. In a hand-picked scan, 100 of 591 measurable sites returned 403 to GPTBot or ClaudeBot, and only four documented the choice in robots.txt; the sample does not establish industry-wide rates. External curl checks can reveal user-agent differences, but may misread fingerprinting or IP-based rules, so the article recommends comparing CDN and origin logs and checking Security Events. Sites that choose to block crawlers should document the decision and distinguish training crawlers such as GPTBot from user-requested fetchers such as ChatGPT-User, since blocking the latter also keeps pages out of answers.

### Source excerpt

Our CDN was silently returning 403 to GPTBot and ClaudeBot. Across 640 sites we tested, 100 refused an AI crawler and only 4 said so in robots.txt.

## Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

DevFeed: [Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips](<https://devfeed.tech/articles/exa-boosts-embedding-throughput-by-50-to-index-billions-of-documents-on-aws-ai-chips-141408.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/machine-learning/exa-boosts-embedding-throughput-by-50-to-index-billions-of-documents-on-aws-ai-chips/>)

Author: Summer Devlin

Published: 2026-11-18T23:58:22Z

Content type: news

Language: en

Sources: [Artificial Intelligence](<https://devfeed.tech/sources/artificial-intelligence.md>)

Topics: [AWS Inferentia](<https://devfeed.tech/topics/aws-inferentia.md>), [AWS AI chips](<https://devfeed.tech/topics/aws-ai-chips.md>), [neuron](<https://devfeed.tech/topics/neuron.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [AWS Trainium](<https://devfeed.tech/topics/aws-trainium.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-chips](<https://devfeed.tech/tags/ai-chips.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-ai-chips](<https://devfeed.tech/tags/aws-ai-chips.md>), [aws-inferentia](<https://devfeed.tech/tags/aws-inferentia.md>), [aws-neuron](<https://devfeed.tech/tags/aws-neuron.md>), [aws-trainium](<https://devfeed.tech/tags/aws-trainium.md>), [cost](<https://devfeed.tech/tags/cost.md>), [custom](<https://devfeed.tech/tags/custom.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [database](<https://devfeed.tech/tags/database.md>), [economics](<https://devfeed.tech/tags/economics.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [index](<https://devfeed.tech/tags/index.md>), [inference](<https://devfeed.tech/tags/inference.md>), [kernels](<https://devfeed.tech/tags/kernels.md>), [neuron](<https://devfeed.tech/tags/neuron.md>), [optimized](<https://devfeed.tech/tags/optimized.md>), [performance](<https://devfeed.tech/tags/performance.md>), [search](<https://devfeed.tech/tags/search.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [unit-economics](<https://devfeed.tech/tags/unit-economics.md>), [vector](<https://devfeed.tech/tags/vector.md>)

### AI overview

Exa reports a 47% improvement in performance and cost per million documents embedded on AWS Neuron hardware versus its initial baseline. Starting with FP16 data parallelism on Inferentia2, it optimized weight layout and precision, used a drop-in attention kernel, and built a fused SwiGLU MLP kernel to reduce memory spill. Custom attention kernels tailored to the model's shapes then cut padding work and instruction steps. Each change passed a correctness gate of 0.999 cosine similarity to the reference; performance was measured with a 30-minute saturation benchmark. The workload uses one NeuronCore per pod, and the same approach scales through data parallelism on Trainium2. Exa says tuning across hardware and Regions helps its scheduler match shifting demand and availability. The excerpt describes these results and methods but does not provide absolute throughput or cost figures.

### Source excerpt

Exa's neural search engine is backed by an in-house vector database, so the unit economics of embedding tens of billions of web documents drive the business. This post shares how Exa optimized embedding inference on AWS Neuron hardware (Inferentia2 and Trainium2) with custom NKI kernels, reaching a 47% gain in performance and cost per million documents.

## Microsoft says Bing serves 1B people. Its next search engine isn't for them.

DevFeed: [Microsoft says Bing serves 1B people. Its next search engine isn't for them.](<https://devfeed.tech/articles/microsoft-says-bing-serves-1b-people-its-next-search-engine-isn-t-for-them-141166.md>)

Original publisher: [Read original article](<https://thenewstack.io/microsoft-web-iq-agents/>)

Author: Frederic Lardinois

Published: 2026-11-18T20:38:31Z

Content type: news

Language: en

Sources: [The New Stack](<https://devfeed.tech/sources/the-new-stack.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Microsoft](<https://devfeed.tech/topics/microsoft.md>)

Tags: [advanced](<https://devfeed.tech/tags/advanced.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-strategy](<https://devfeed.tech/tags/ai-strategy.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [azure](<https://devfeed.tech/tags/azure.md>), [bing](<https://devfeed.tech/tags/bing.md>), [build](<https://devfeed.tech/tags/build.md>), [cost](<https://devfeed.tech/tags/cost.md>), [costs](<https://devfeed.tech/tags/costs.md>), [crawler](<https://devfeed.tech/tags/crawler.md>), [deepseek](<https://devfeed.tech/tags/deepseek.md>), [design](<https://devfeed.tech/tags/design.md>), [firecrawl](<https://devfeed.tech/tags/firecrawl.md>), [index](<https://devfeed.tech/tags/index.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [search](<https://devfeed.tech/tags/search.md>), [search-engine](<https://devfeed.tech/tags/search-engine.md>)

### AI overview

Microsoft's Web IQ reuses Bing's index but is designed for AI agents: it returns relevant passages rather than ranked links or entire documents, which can reduce the text models must process. Its Auto mode runs query variations in parallel; Auto Advanced also checks for information gaps and searches again. Microsoft says it follows sites' robots.txt instructions and identifies itself as a crawler. Web IQ leaves answer generation to the agent's chosen model, unlike competitors described as producing full answers or research reports. Listed prices per 1,000 searches are $7 for standard, $8 for Auto, and $9 for Auto Advanced; Browse costs $1 when the URL is already known. The excerpt presents a tradeoff between direct, fast API calls and automated search that may improve quality or cost less than repeated agent calls. Claims about speed and quality are attributed to Microsoft or its customer, not independently established here.

### Source excerpt

Microsoft says its Bing search engine now has one billion monthly users. The company says those are humans -- not The post Microsoft says Bing serves 1B people. Its next search engine isn't for them. appeared first on The New Stack.

## Meet the Writer: Alena Belova on Reputation, AI Search and the Numbers Nobody Publishes

DevFeed: [Meet the Writer: Alena Belova on Reputation, AI Search and the Numbers Nobody Publishes](<https://devfeed.tech/articles/meet-the-writer-alena-belova-on-reputation-ai-search-and-the-numbers-nobody-publishes-141289.md>)

Original publisher: [Read original article](<https://hackernoon.com/meet-the-writer-alena-belova-on-reputation-ai-search-and-the-numbers-nobody-publishes?source=rss>)

Author: HackerNoon Writers Spotlight

Published: 2026-11-18T19:42:59Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Search engine optimization (SEO)](<https://devfeed.tech/topics/seo.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Finance](<https://devfeed.tech/topics/finance.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-search](<https://devfeed.tech/tags/ai-search.md>), [ai-search-optimization](<https://devfeed.tech/tags/ai-search-optimization.md>), [data-journalism](<https://devfeed.tech/tags/data-journalism.md>), [economics](<https://devfeed.tech/tags/economics.md>), [generative-engine-optimization](<https://devfeed.tech/tags/generative-engine-optimization.md>), [geo](<https://devfeed.tech/tags/geo.md>), [hidden](<https://devfeed.tech/tags/hidden.md>), [llms](<https://devfeed.tech/tags/llms.md>), [meet-the-writer](<https://devfeed.tech/tags/meet-the-writer.md>), [numbers](<https://devfeed.tech/tags/numbers.md>), [online](<https://devfeed.tech/tags/online.md>), [online-reputation](<https://devfeed.tech/tags/online-reputation.md>), [search](<https://devfeed.tech/tags/search.md>), [testing](<https://devfeed.tech/tags/testing.md>), [writer](<https://devfeed.tech/tags/writer.md>)

### AI overview

Alena Belova's work connects AI search, reputation and hidden economics through a recurring question: how do people and organizations become visible, and what supports the picture people trust? At BARS Agency, she works on how companies appear in AI search; Geometrika began as a tool to track how assistants describe brands and which sources they use. Her research starts with a table of unanswered questions, then checks primary documents and reconciles datasets against their stated totals. She treats a number's absence from public records as a finding, while emphasizing restraint in what makes it into a story. She says AI answers can vary between rounds and across countries, and plans to ask major assistants who she is and trace any errors to their sources. She also wants smaller cultural institutions to be easier to find online.

### Source excerpt

Alena Belova on AI search, online reputation and hidden economics: how she researches the numbers nobody publishes and what she is testing next.

## Evaluating AST-Derived Knowledge Graph Retrieval for AIOps Incident Diagnosis in Apache Kafka

DevFeed: [Evaluating AST-Derived Knowledge Graph Retrieval for AIOps Incident Diagnosis in Apache Kafka](<https://devfeed.tech/articles/evaluating-ast-derived-knowledge-graph-retrieval-for-aiops-incident-diagnosis-in-apache-kafka-139584.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.21625>)

Author: Shashank Mishra, Shweta Mishra

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [AIOps](<https://devfeed.tech/topics/aiops.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [graph-database](<https://devfeed.tech/topics/graph-database.md>), [Parsing](<https://devfeed.tech/topics/parsing.md>)

Tags: [aiops](<https://devfeed.tech/tags/aiops.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [ast](<https://devfeed.tech/tags/ast.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [diagnosis](<https://devfeed.tech/tags/diagnosis.md>), [graph](<https://devfeed.tech/tags/graph.md>), [incident](<https://devfeed.tech/tags/incident.md>), [knowledge-graph](<https://devfeed.tech/tags/knowledge-graph.md>), [log](<https://devfeed.tech/tags/log.md>), [queries](<https://devfeed.tech/tags/queries.md>), [reranking](<https://devfeed.tech/tags/reranking.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [source](<https://devfeed.tech/tags/source.md>)

### AI overview

In Apache Kafka incident diagnosis, vector-only retrieval achieved Recall@20 of 0.956, and adding static or confidence-guided staged AST ownership-graph expansion did not significantly improve recall. Staging did reduce cross-encoder pair workload and measured reranking time by about 52% compared with static expansion. As an external control, HippoRAG scored 0.308 versus 0.803 for its ColBERTv2-only backbone; a post-hoc document ensemble raised its score to 0.648 but did not close the gap. In a manually evaluated 100-query study, AST-derived structural evidence showed no significant advantage in root-cause correctness. The findings are based on 1,600 source-linked log records and a 315-query benchmark covering 103 targets.

### Source excerpt

arXiv:2611.21625v1 Announce Type: new Abstract: Incident diagnosis requires retrieving source-linked operational evidence from large codebases. We evaluate graph-augmented retrieval for Apache Kafka using an abstract-syntax-tree (AST)-derived ownership graph over 1,600 source-linked log records and an audited benchmark of 315 incident-style queries covering 103 unique targets. Vector-only retrieval is a strong baseline at Recall@20 of 0.956, and neither static nor confidence-guided staged AST expansion significantly improves Recall@20 over it. As an external control we evaluate HippoRAG, a published text-derived graph-retrieval system, against its own ColBERTv2 retrieval backbone. Against ColBERTv2-only (Recall@20 0.803), HippoRAG attained 0.308; a post-hoc document-ensemble sensitivity raised Recall@20 to 0.648 but did not eliminate the deficit. Relative to static expansion, staging reduces cross-encoder pair workload and measured reranking time by about 52%. A manually evaluated 100-query downstream study finds no significant root-cause-correctness advantage for AST-derived structural evidence.

## ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents

DevFeed: [ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents](<https://devfeed.tech/articles/forecastgym-a-live-controlled-and-replayable-benchmark-for-forecasting-agents-140109.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.23045>)

Author: Yicheng Tao, Yuante Li, Zipei Fan

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [brier-score](<https://devfeed.tech/tags/brier-score.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [forecasting](<https://devfeed.tech/tags/forecasting.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [live](<https://devfeed.tech/tags/live.md>), [prediction-market](<https://devfeed.tech/tags/prediction-market.md>)

### AI overview

ForecastGym evaluates forecasting agents against fixed, time-stamped web evidence, helping separate search quality from model reasoning. It records results from four commercial search APIs every two days for open prediction-market questions, then replays the evidence offline after outcomes are known. Its first release contains 150 questions, 614 evidence states, and 131,000 search results. Retrospective searches after settlement returned outcome-revealing results for as many as 92% of questions despite tested date filters. On frozen evidence, all tested LLMs trailed the market; diagnostic analysis suggests missed public information mattered more than misreading in high-gap states. Changing the search API shifted a model's Brier score by up to 0.043, exceeding the spread across models. Persistent notes performed similarly to rereading all evidence while using half the input tokens.

### Source excerpt

arXiv:2611.23045v1 Announce Type: new Abstract: Agents built on large language models (LLMs) increasingly forecast real-world events by searching the Web and reasoning over what they find. How well such an agent forecasts depends on both the retrieved evidence and the model, making control over that evidence essential for evaluation. Live evaluations can mix retrieval with reasoning when each system searches on its own, while retrospective search can leak later information through date filters or today's ranking of historical pages. Comparing models, search sources, and memory strategies requires both fixed evidence and a record of how it evolved before resolution. We introduce ForecastGym, which records what four commercial web search APIs return for open prediction-market questions every two days and replays this frozen evidence offline once outcomes are known, comparing models, memory, and captured search sources under a fixed evidence-acquisition policy. Its first release covers 150 questions, 614 evidence states, one per question and collection time, and 131k search results. Rerunning the same queries after settlement returns results that all but state the outcome for up to 92% of the questions, despite the tested date filters. On the frozen evidence, all tested LLMs trail the market in forecasting accuracy. Our diagnostic analysis suggests that missed public information contributes more to the gap in high-gap states than misreading does. Switching the search API changes a model's Brier score by up to 0.043, more than the spread across models, while persistent notes score similarly to rereading all evidence at half the input tokens. The evidence a forecaster sees can matter as much as the model that reads it.

## Parallel web search is free on AI Gateway and eve through January 31

DevFeed: [Parallel web search is free on AI Gateway and eve through January 31](<https://devfeed.tech/articles/parallel-web-search-is-free-on-ai-gateway-and-eve-through-january-31-141198.md>)

Original publisher: [Read original article](<https://vercel.com/changelog/parallel-web-search-is-free-on-ai-gateway-and-eve-through-january-31>)

Author: Mark Pariente

Published: 2026-11-18T05:00:00Z

Content type: release

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-gateway](<https://devfeed.tech/tags/ai-gateway.md>), [ai-sdk](<https://devfeed.tech/tags/ai-sdk.md>), [api](<https://devfeed.tech/tags/api.md>), [api-key](<https://devfeed.tech/tags/api-key.md>), [automatically](<https://devfeed.tech/tags/automatically.md>), [built-in](<https://devfeed.tech/tags/built-in.md>), [data-retention](<https://devfeed.tech/tags/data-retention.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [eve](<https://devfeed.tech/tags/eve.md>), [fast](<https://devfeed.tech/tags/fast.md>), [filter](<https://devfeed.tech/tags/filter.md>), [free](<https://devfeed.tech/tags/free.md>), [links](<https://devfeed.tech/tags/links.md>), [model](<https://devfeed.tech/tags/model.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [search](<https://devfeed.tech/tags/search.md>), [web](<https://devfeed.tech/tags/web.md>), [web-search](<https://devfeed.tech/tags/web-search.md>)

### AI overview

Parallel fast mode is free on AI Gateway through January 31, 2027, and is the default web search for eve agents when their model is served through AI Gateway. It works with any AI Gateway model without a separate Parallel API key, returning links with short excerpts and allowing filters by website or publication date. With the AI SDK, enable it using `gateway.tools.parallelSearch({ mode: 'fast' })` in the tools configuration. The offer applies to fast mode only, with Zero Data Retention enabled by default; other modes use standard pricing.

### Source excerpt

Parallel web search in fast mode is now free on AI Gateway through January 31, 2027, and Parallel is now the default web search for eve agents. The tool works with any AI Gateway model, with no separate Parallel API key. Parallel finds relevant web pages and returns links with short excerpts. You can filter results by website or publication date. In eve, the built-in web_search tool uses Parallel fast mode automatically when the model is served through AI Gateway. See the eve documentation for other search configurations. To use Parallel fast mode with the AI SDK, add gateway.tools.parallelSearch({ mode: 'fast' }) to your tools: The offer covers fast mode only, with Zero Data Retention enabled by default. Other modes are billed at standard pricing. Learn more in the web search documentation. Read more

## Deal With It: On Better Schedules for Listwise Reranking

DevFeed: [Deal With It: On Better Schedules for Listwise Reranking](<https://devfeed.tech/articles/deal-with-it-on-better-schedules-for-listwise-reranking-140023.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.22392>)

Author: Benjamin Clavi\\'e, Herumb Shandilya, Hanna Lichtenberg, Aamir Shakir, Makoto P. Kato

Published: 2026-11-18T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [round robin](<https://devfeed.tech/topics/round-robin.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [quota](<https://devfeed.tech/tags/quota.md>), [random](<https://devfeed.tech/tags/random.md>), [rank](<https://devfeed.tech/tags/rank.md>), [repair](<https://devfeed.tech/tags/repair.md>), [reranking](<https://devfeed.tech/tags/reranking.md>), [round-robin](<https://devfeed.tech/tags/round-robin.md>), [schedules](<https://devfeed.tech/tags/schedules.md>), [split](<https://devfeed.tech/tags/split.md>)

### AI overview

Listwise LLM rerankers can lose top passages when a parallel schedule advances a fixed quota from each candidate group, triggering costly repair calls. The abstract reports that dealing candidates round-robin by retriever rank minimizes expected cuts under a simple retriever model; contiguous chunks cut more as the retriever improves. DealRank is within 0.1 nDCG@10 points of the best schedules on average across six LLMs, and is 1.1-1.4 times faster than BlitzRank with concurrent first calls up to 1,000 candidates. In existing schedules such as BlitzRank, dealing reduces queries requiring repair from 59% to 14% with no significant quality change. Random partitioning appears equivalent at quota four, but depends on the seed and trails by 1.3 nDCG@10 points at quota one over a dense retriever. These are abstract-level claims; the supplied text does not include the full paper.

### Source excerpt

arXiv:2611.22392v1 Announce Type: cross Abstract: Listwise LLM rerankers cannot reliably rank hundreds of documents in a single call, and rely on schedules to orchestrate their calls. Parallel schedules split candidates into groups and advance a fixed quota from each. When a group holds more of the top passages than its quota, some are cut, and expensive repair calls become required. We show that the way candidates are split matters where such a quota can bind, and that, in expected cuts, the best split there deals them round-robin by retriever rank, like cards. Although not conceptually new, dealing has remained unanalyzed. We prove that, under a simple retriever model, no split with equal quotas cuts fewer of the true top ten on average, and that contiguous chunks cut more as the retriever improves. DealRank, the simplest schedule our analysis allows, is on average within 0.1 nDCG@10 points of the best schedules over six LLMs, in two rounds. It is consistently 1.1 to 1.4 times faster than BlitzRank with concurrent first calls up to a thousand candidates. There, it needs three rounds and is eleven times faster than the sliding window. In existing schedules such as BlitzRank, dealing reduces repair calls from 59% of queries to 14%, improving latency with no significant change in quality. Random partitioning appears equivalent at a quota of four, as predicted, but ties results to the seed and trails by 1.3 nDCG@10 points at a quota of one over a dense retriever.

## The $50 Per Hour Search Engine

DevFeed: [The $50 Per Hour Search Engine](<https://devfeed.tech/articles/the-50-per-hour-search-engine-137773.md>)

Original publisher: [Read original article](<https://hackaday.com/2026/11/17/the-50-per-hour-search-engine/>)

Author: Al Williams

Published: 2026-11-17T15:00:16Z

Content type: article

Language: en

Sources: [Hackaday](<https://devfeed.tech/sources/hackaday.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Google Search](<https://devfeed.tech/topics/google-search.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [boolean](<https://devfeed.tech/tags/boolean.md>), [command-line](<https://devfeed.tech/tags/command-line.md>), [cost](<https://devfeed.tech/tags/cost.md>), [database](<https://devfeed.tech/tags/database.md>), [dialog](<https://devfeed.tech/tags/dialog.md>), [featured](<https://devfeed.tech/tags/featured.md>), [history](<https://devfeed.tech/tags/history.md>), [search](<https://devfeed.tech/tags/search.md>), [search-engine](<https://devfeed.tech/tags/search-engine.md>)

### AI overview

DIALOG brought interactive searching of specialized research collections to users decades before the Web. Developed at Lockheed and commercialized in 1972, it let searchers query indexed databases through command-line operations such as BEGIN, EXPAND, SELECT, and TYPE, refining results with Boolean logic. Unlike broad web search, its collections were carefully organized professional information, often providing bibliographic records and abstracts rather than full articles; saved searches could also run when databases were updated. Access required a terminal or modem and network connection, with database and communications charges adding up--commonly about $50 per hour for databases in 1981--so experienced searchers planned queries to limit paid time. DIALOG offered more precise, repeatable searches but demanded specialist skill and was costly. The excerpt says the service continues as Clarivate Dialog with specialized research databases.

### Source excerpt

Imagine doing a Google search and getting a bill for $50 an hour -- or even $300 an hour. Even worse, that's $50 for every hour you were even connected. ...read more

## StaleBench: Isolating Stale Retrieval in Agent Memory, and Pricing the Fix

DevFeed: [StaleBench: Isolating Stale Retrieval in Agent Memory, and Pricing the Fix](<https://devfeed.tech/articles/stalebench-isolating-stale-retrieval-in-agent-memory-and-pricing-the-fix-137491.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.19752>)

Author: S M Mahbub Murshed

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Entity resolution](<https://devfeed.tech/topics/entity-resolution.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-memory](<https://devfeed.tech/tags/agent-memory.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [memory](<https://devfeed.tech/tags/memory.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>)

### AI overview

StaleBench evaluates stale retrieval across 892 real updating facts, using validity labels from Wikidata via TempLAMA rather than retrieval scores. It finds that retrieval metrics can misrank memory policies: a retention policy and a retain-everything control returned identical rank-1 statements, yet reader accuracy differed by 29.5 and 65.6 points when dates were hidden. The new Contamination@5 metric ranks these policies correctly where current@1 and current@5 do not. Removing statements' lexical anchors reduced freshness for deployed keying mechanisms by tens of points; entity resolution during both writing and querying recovered most of that loss. Mem0, Graphiti, LangMem, and a reimplemented ERASE handle supersession differently, with no single scoring convention describing all four. The authors release the data, frozen evaluator, and reference baselines.

### Source excerpt

arXiv:2611.19752v1 Announce Type: cross Abstract: An agent that stores facts over time eventually stores several versions of the same fact. Asked a value-agnostic question - "which team does X play for now?" - a similarity retriever cannot prefer the current version, because every version matches the entity the query names. StaleBench isolates this stale-retrieval failure on 892 real updating facts, with currency labels from the source (Wikidata validity years via TempLAMA), never from a retrieval score. Our central result is that retrieval metrics can rank memory policies backwards: a retention policy and its retain-everything control return the identical statement at rank 1 for every test query, yet differ by 29.5 and 65.6 points in reader accuracy on two readers when no date is shown. Contamination@5, the metric we add, ranks them correctly where current@1 and current@5 do not. Withdrawing the lexical anchor that statements carry costs deployed keying mechanisms tens of points of freshness; entity resolution on both the write path and the query recovers most of the loss. Three deployed systems (Mem0, Graphiti, LangMem) and a reimplemented ERASE each handle supersession differently, and no single scoring convention describes all four. We release the data, a frozen evaluator, and reference baselines.

## Airbnb Introduces CASTLE for Training Natural Language Search Models with Synthetic Data

DevFeed: [Airbnb Introduces CASTLE for Training Natural Language Search Models with Synthetic Data](<https://devfeed.tech/articles/airbnb-introduces-castle-for-training-natural-language-search-models-with-synthetic-data-135326.md>)

Original publisher: [Read original article](<https://www.infoq.com/news/2026/11/airbnb-castle/>)

Author: Robert Krzaczyński

Published: 2026-11-16T12:25:00Z

Content type: news

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [distilabel](<https://devfeed.tech/topics/distilabel.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [airbnb](<https://devfeed.tech/tags/airbnb.md>), [airbnb-castle](<https://devfeed.tech/tags/airbnb-castle.md>), [cold-start](<https://devfeed.tech/tags/cold-start.md>), [data](<https://devfeed.tech/tags/data.md>), [framework](<https://devfeed.tech/tags/framework.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [natural-language-processing](<https://devfeed.tech/tags/natural-language-processing.md>), [news](<https://devfeed.tech/tags/news.md>), [queries](<https://devfeed.tech/tags/queries.md>), [relevance](<https://devfeed.tech/tags/relevance.md>), [search](<https://devfeed.tech/tags/search.md>), [search-traffic](<https://devfeed.tech/tags/search-traffic.md>), [synthetic-data](<https://devfeed.tech/tags/synthetic-data.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

CASTLE trains natural language search models by turning de-identified booking sessions into contrastive listing pairs: the booked property is positive, a viewed but unbooked property is negative, and an LLM generates a query that distinguishes them. This embeds relevance labels in query construction, unlike InPars and Promptagator's separate query-generation and labeling steps. CASTLE combines template prompts, few-shot prompts based on about 500 research seed queries, attribute-grounded prompts, and query variation. For pairs lacking booking history, its Virtual Judge labels additional query-listing pairs; CASTLE-R is intended for production settings with emerging search traffic. In the reported evaluation, query-distribution KL divergence was 1.01 versus 9.33 for Promptagator, and attribute-level divergence was 0.08 versus 0.09 for manually curated survey queries. Three experts agreed with relevance assignments 91-93% of the time across 200 triplets. The team says it evaluated on untouched real queries and used leakage guardrails, but did not report a controlled comparison of random splits against stricter splits disjoint in seed templates and listing IDs.

### Source excerpt

Airbnb researchers have introduced CASTLE, a framework designed to address the cold-start problem in natural language search. The approach uses de-identified booking sessions and large language models (LLMs) to jointly generate synthetic search queries and relevance labels, enabling model training before sufficient real search traffic is available. By Robert Krzaczyński

## Engineering Zero-State Search

DevFeed: [Engineering Zero-State Search](<https://devfeed.tech/articles/engineering-zero-state-search-135514.md>)

Original publisher: [Read original article](<https://hackernoon.com/engineering-zero-state-search?source=rss>)

Author: Archana Atreya

Published: 2026-11-16T09:36:21Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>), [Model Development](<https://devfeed.tech/topics/model-development.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>)

Tags: [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [affinity](<https://devfeed.tech/tags/affinity.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [all](<https://devfeed.tech/tags/all.md>), [app](<https://devfeed.tech/tags/app.md>), [article](<https://devfeed.tech/tags/article.md>), [autocomplete](<https://devfeed.tech/tags/autocomplete.md>), [background](<https://devfeed.tech/tags/background.md>), [balancing](<https://devfeed.tech/tags/balancing.md>), [based](<https://devfeed.tech/tags/based.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [engineering-zero-state-search](<https://devfeed.tech/tags/engineering-zero-state-search.md>), [first](<https://devfeed.tech/tags/first.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [pre-query-recommendations](<https://devfeed.tech/tags/pre-query-recommendations.md>), [product-design](<https://devfeed.tech/tags/product-design.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [ranking-algorithm](<https://devfeed.tech/tags/ranking-algorithm.md>), [recommendation-systems](<https://devfeed.tech/tags/recommendation-systems.md>), [search](<https://devfeed.tech/tags/search.md>), [search-latency-optimization](<https://devfeed.tech/tags/search-latency-optimization.md>), [search-ux](<https://devfeed.tech/tags/search-ux.md>), [state](<https://devfeed.tech/tags/state.md>), [user](<https://devfeed.tech/tags/user.md>), [zero](<https://devfeed.tech/tags/zero.md>), [zero-state-search](<https://devfeed.tech/tags/zero-state-search.md>)

### AI overview

Zero-state search ranks suggestions before a user types, so fast results depend on precomputation but can become stale. The article recommends pairing a slow batch or nearline candidate layer with a lightweight edge or client reranker that accounts for recent actions and context. To avoid repetitive screens, it advises clustering and deduplicating intents, penalizing repeatedly ignored suggestions, and reserving some slots for exploration. The interface should let users begin typing immediately and yield to autocomplete without friction. Evaluate the full search session--downstream task completion, time to success, reformulation or abandonment, and how suggestions change across sessions--rather than zero-state click-through rate alone.

### Source excerpt

The engineering trade-offs of recommending before the first keystroke. Learn how to balance predictive ranking, data freshness, and user control.

## LEGAL-UQA: Testing Constitutional Question-Answering Systems for Urdu Speakers with a Parallel English-Urdu Benchmark

DevFeed: [LEGAL-UQA: Testing Constitutional Question-Answering Systems for Urdu Speakers with a Parallel English-Urdu Benchmark](<https://devfeed.tech/articles/legal-uqa-testing-constitutional-question-answering-systems-for-urdu-speakers-with-a-parallel-english-urdu-benchmark-134944.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2410.13013>)

Author: Faizan Faisal, Umair Yousaf, Muhammad Hassnain, Zeerak Babar

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Language Engineering](<https://devfeed.tech/topics/language-engineering.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cl](<https://devfeed.tech/tags/cs-cl.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [english](<https://devfeed.tech/tags/english.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [human-review](<https://devfeed.tech/tags/human-review.md>), [legal](<https://devfeed.tech/tags/legal.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [qa](<https://devfeed.tech/tags/qa.md>), [questions](<https://devfeed.tech/tags/questions.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

LEGAL-UQA provides 619 parallel English-Urdu question-answer pairs linked to passages from Pakistan's Constitution. Its evaluation separates answering Urdu questions with a passage supplied from retrieving the relevant passage. Model rankings varied with the answer-similarity measure, and simple word matching retrieved passages more effectively than the tested embedding methods. At the evaluated cutoff, the automated judge JEV agreed with human grades less often than a baseline that gave every answer full credit. An audit also found generated reference answers that omitted source-supported information. The authors recommend evaluating retrieval and answering separately and using human review to validate judges and references.

### Source excerpt

arXiv:2410.13013v2 Announce Type: replace-cross Abstract: Legal information assistants could help Urdu speakers understand their constitutional rights, but building reliable systems requires test data and evaluation methods suited to the language. We present LEGAL-UQA, a parallel English-Urdu benchmark of 619 question-answer pairs linked to passages from Pakistan's Constitution. Each example includes the question, reference answer, and source passage in both languages. We evaluate two components separately: answering Urdu questions when the source passage is supplied, and retrieving the passage from which each question was generated. We also compare JEV, a decision model used as an automated judge, with human grades. The findings show that model rankings depend on how answer similarity is measured, while a simple word-matching method retrieves source passages more effectively than the embedding methods tested. At the evaluated cutoff, JEV agrees with human grades less often than a baseline that gives every answer full credit. An audit also identifies cases where generated references omit information supported by the source. Together, these findings highlight the need to test retrieval and answering separately, include simple retrieval baselines, and validate automated judges and reference answers with human review when engineering legal QA software for Urdu speakers.

## Who sets the consent signal? Platform defaults and crawl coverage in measuring opt-outs from AI training

DevFeed: [Who sets the consent signal? Platform defaults and crawl coverage in measuring opt-outs from AI training](<https://devfeed.tech/articles/who-sets-the-consent-signal-platform-defaults-and-crawl-coverage-in-measuring-opt-outs-from-ai-training-134923.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18213>)

Author: Marcin Lawenda

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Networking](<https://devfeed.tech/sources/arxiv-networking.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [hosting](<https://devfeed.tech/topics/hosting.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [ai-training](<https://devfeed.tech/tags/ai-training.md>), [consent](<https://devfeed.tech/tags/consent.md>), [coverage](<https://devfeed.tech/tags/coverage.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [opt-out](<https://devfeed.tech/tags/opt-out.md>), [platform](<https://devfeed.tech/tags/platform.md>), [web](<https://devfeed.tech/tags/web.md>)

### AI overview

The study finds that AI-training opt-outs are often set by platforms and hosts rather than site owners, so consent measurements should account for delegated signals and crawl coverage. Researchers crawled 3,445,751 reachable sites from Common Crawl's Polish, Czech, Slovak, Hungarian, and German domain lists in October 2026, plus a random sample of 10,000 Slovak registry domains. They matched signals to vendor templates and identical files and tested four exploratory hypotheses with logistic models. Opt-outs appeared on 0.70%-3.82% of listed sites; platform and hosting defaults set 15%-78% of opt-outs. Default settings reached 60%-95% of a vendor's sites, while owner-enabled settings reached at most 2.1%. Platforms statistically explained most of the higher Czech and Slovak rates relative to Germany. Common Crawl-based lists missed about a third of Slovak opt-outs because they exclude sites that block its crawler.

### Source excerpt

arXiv:2611.18213v1 Announce Type: new Abstract: Web data used in social research and in the training of AI models increasingly carry consent signals, but who sets these signals is rarely measured. This paper attributes opt-outs from AI training to the party that set them. The complete lists of Polish, Czech, Slovak, Hungarian and German domains recorded by Common Crawl, with 3,445,751 reachable sites, were crawled in October 2026 together with a random sample of 10,000 Slovak registry domains. Each signal was matched against vendor templates and identical files. Four exploratory hypotheses were examined with logistic models. Between 0.70% and 3.82% of the listed sites opt out. Platform and hosting defaults set 15% to 78% of the opt-outs. Settings active by default reach 60% to 95% of a vendor's sites, while settings that owners switch on reach at most 2.1%. Most of the higher Czech and Slovak rates compared with Germany are statistically attributable to platforms. Lists derived from Common Crawl miss about a third of the Slovak opt-outs because they exclude sites that block its crawler. The results describe delegated consent, a signal that a platform sets on behalf of its users, which measurements of consent on the web must consider.

## GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers

DevFeed: [GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers](<https://devfeed.tech/articles/gentus-table-union-search-via-constrained-generation-of-semantic-table-identifiers-135437.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.00833>)

Author: Shulun Zhang, Linting Wang, Yuwei Xu, Yingli Zhou, Jieming Shi, Chenhao Ma

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [beam](<https://devfeed.tech/tags/beam.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [data-discovery](<https://devfeed.tech/tags/data-discovery.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [enrichment](<https://devfeed.tech/tags/enrichment.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generative](<https://devfeed.tech/tags/generative.md>), [identifier](<https://devfeed.tech/tags/identifier.md>), [latency](<https://devfeed.tech/tags/latency.md>), [quality](<https://devfeed.tech/tags/quality.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [search](<https://devfeed.tech/tags/search.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [storage](<https://devfeed.tech/tags/storage.md>), [supervision](<https://devfeed.tech/tags/supervision.md>), [table](<https://devfeed.tech/tags/table.md>), [union](<https://devfeed.tech/tags/union.md>), [vector](<https://devfeed.tech/tags/vector.md>), [vector-index](<https://devfeed.tech/tags/vector-index.md>)

### AI overview

GenTUS retrieves unionable tables by generating their semantic identifiers directly from a query table. It learns compact identifiers with pairwise unionability supervision, encouraging compatible tables to share prefixes, then uses prefix-constrained beam search so every generated identifier maps to an existing table. The generated order provides the ranking without searching or reranking a candidate pool, which avoids the candidate-recall limitation described for encode-search-refine methods. Across seven public benchmarks, the abstract reports the best overall retrieval quality, with the largest gains on larger and more diverse benchmarks, plus substantially lower online retrieval time and retrieval-artifact storage than prior TUS methods.

### Source excerpt

arXiv:2607.00833v2 Announce Type: replace Abstract: Modern data lakes contain heterogeneous tables, and the information needed for a task is often scattered across many tables with different schemas, sources, and naming conventions. Table union search (TUS) retrieves tables whose rows are semantically compatible with a query table for union, supporting data discovery, enrichment, and downstream analytics. Existing TUS methods follow an encode-search-refine pipeline: they embed tables or columns, retrieve a candidate pool from a vector index, and refine it by query-candidate matching or reranking. As a result, a unionable table missed by candidate retrieval cannot be recovered later, and searching and refining a pool for every query adds latency and storage costs that grow with the data lake. We propose GenTUS, a generative retrieval framework that reformulates TUS as constrained generation over discrete semantic table identifiers. GenTUS assigns each data-lake table a compact identifier learned under pairwise unionability supervision, which encourages unionable tables to share identifier prefixes, and trains a generator to produce the identifiers of unionable tables directly from the query table. At query time, prefix-constrained beam search guarantees that every generated identifier maps to an existing table, and its output order serves directly as the ranking, with no candidate pool to search or rerank. Experiments on seven public TUS benchmarks show that GenTUS achieves the best overall retrieval quality, with the largest gains on the larger and more diverse benchmarks, while requiring substantially lower online retrieval time and retrieval-artifact storage than prior TUS methods.

## StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval

DevFeed: [StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval](<https://devfeed.tech/articles/streamembed-real-time-indexing-from-pixels-for-screenshot-retrieval-135913.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.18516>)

Author: Dayoon Ko, Yichuan Wang, Soyeong Jeong, Joseph E. Gonzalez, Gunhee Kim, Sewon Min

Published: 2026-11-16T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [screenshot](<https://devfeed.tech/topics/screenshot.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Vectors & LLMs](<https://devfeed.tech/topics/vectors-llms.md>)

Tags: [adapter](<https://devfeed.tech/tags/adapter.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [blocks](<https://devfeed.tech/tags/blocks.md>), [computer-use](<https://devfeed.tech/tags/computer-use.md>), [cs-cv](<https://devfeed.tech/tags/cs-cv.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [encoding](<https://devfeed.tech/tags/encoding.md>), [exploits](<https://devfeed.tech/tags/exploits.md>), [fast](<https://devfeed.tech/tags/fast.md>), [faster](<https://devfeed.tech/tags/faster.md>), [foundation](<https://devfeed.tech/tags/foundation.md>), [frontier](<https://devfeed.tech/tags/frontier.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [indexing](<https://devfeed.tech/tags/indexing.md>), [pixels](<https://devfeed.tech/tags/pixels.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [screenshot](<https://devfeed.tech/tags/screenshot.md>)

### AI overview

StreamEmbed speeds up visual indexing of incoming screenshots by exploiting repeated pixels: it skips spatial blocks that duplicate a neighbor and reuses cached keys and values for blocks unchanged between screenshots. A learned adapter helps preserve fidelity to embeddings computed from full screenshots. On ScreenMemEval, with 10,426 questions over 184 hours of histories, it encoded up to 3.7x faster than full-screenshot encoding on an M2 Pro GPU without sacrificing accuracy, and improved Recall@10 by up to 6.3 points over OCR indexing at matched latency. Its vector index used 0.13 GB of RAM for a simulated year of screenshots. These are reported benchmark results; the supplied text does not describe evaluation beyond that setup.

### Source excerpt

arXiv:2611.18516v1 Announce Type: cross Abstract: Screen history can serve as searchable visual memory for people and agents, but unlike an offline corpus, it must be encoded and indexed as each screenshot arrives. OCR-based indexing is fast but loses visual information; visual encoding takes over two seconds. Yet screens change incrementally: most pixels repeat within and across screenshots. This raises a question: do visual encoders need to recompute their representations from scratch for every new screenshot? We present StreamEmbed, a system for incremental visual embedding that exploits this redundancy. Guided solely by pixel differences, it combines spatial token pruning, which skips blocks that repeat a neighbor, with temporal KV reuse, which reuses cached keys and values for blocks unchanged across screenshots. A small learned adapter compensates for pruning and reuse to preserve fidelity to full-screenshot embeddings. On ScreenMemEval, our benchmark of 10,426 questions over 184 hours of screen histories from two humans and a computer-use agent, StreamEmbed sets a new accuracy-latency Pareto frontier on an M2 Pro GPU: it encodes up to 3.7x faster than full-screenshot encoding without sacrificing accuracy and improves accuracy over OCR-based indexing by up to 6.3 Recall@10 points at matched latency. A memory-efficient vector index stores a simulated year of screenshots in 0.13 GB of RAM. Together, these results make real-time indexing of screen history in its native visual form practical, laying a foundation for fully local visual assistants.

## Web search for agents is here: Why ranking still decides the answer

DevFeed: [Web search for agents is here: Why ranking still decides the answer](<https://devfeed.tech/articles/web-search-for-agents-is-here-why-ranking-still-decides-the-answer-134078.md>)

Original publisher: [Read original article](<https://thenewstack.io/ai-agent-search-ranking/>)

Author: Bonnie Chase

Published: 2026-11-14T16:00:00Z

Content type: article

Language: en

Sources: [The New Stack](<https://devfeed.tech/sources/the-new-stack.md>)

Topics: [information retrieval](<https://devfeed.tech/topics/information-retrieval.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [hybrid-search](<https://devfeed.tech/topics/hybrid-search.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Web](<https://devfeed.tech/topics/web.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [22](<https://devfeed.tech/tags/22.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [access-controls](<https://devfeed.tech/tags/access-controls.md>), [account](<https://devfeed.tech/tags/account.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [but](<https://devfeed.tech/tags/but.md>), [first](<https://devfeed.tech/tags/first.md>), [here](<https://devfeed.tech/tags/here.md>), [information](<https://devfeed.tech/tags/information.md>), [needs](<https://devfeed.tech/tags/needs.md>), [post-contributed](<https://devfeed.tech/tags/post-contributed.md>), [ranking](<https://devfeed.tech/tags/ranking.md>), [search](<https://devfeed.tech/tags/search.md>), [sponsor-vespa-ai](<https://devfeed.tech/tags/sponsor-vespa-ai.md>), [sponsored-post-contributed](<https://devfeed.tech/tags/sponsored-post-contributed.md>), [url](<https://devfeed.tech/tags/url.md>), [vespa-ai](<https://devfeed.tech/tags/vespa-ai.md>), [web](<https://devfeed.tech/tags/web.md>), [web-search](<https://devfeed.tech/tags/web-search.md>)

### AI overview

Live web search helps agents find current pages, but a retrieval layer must decide which web and internal evidence to trust and show. The article warns that simply concatenating results can leave conflicts unresolved, favor stale but similar documents, expose restricted content if permissions are checked too late, and increase cost and latency across repeated agent calls. It recommends combining keyword, vector, and structured filtering; ranking with explicit freshness, source authority, and business rules; enforcing permissions at query time; and sending only relevant passages to the model. External results can be indexed with internal data when permitted or kept separate and fused or reranked later; these choices trade off latency, freshness, score comparability, and complexity. The article also advises measuring retrieval quality separately from answer generation and load-testing at realistic agent call volumes.

### Source excerpt

AI agents have had a URL problem for a while. When an agent needs current information but has no search The post Web search for agents is here: Why ranking still decides the answer appeared first on The New Stack.

## How to Evaluate Snowflake Cortex Embedding Models for Enterprise Search

DevFeed: [How to Evaluate Snowflake Cortex Embedding Models for Enterprise Search](<https://devfeed.tech/articles/how-to-evaluate-snowflake-cortex-embedding-models-for-enterprise-search-133909.md>)

Original publisher: [Read original article](<https://hackernoon.com/how-to-evaluate-snowflake-cortex-embedding-models-for-enterprise-search?source=rss>)

Author: Akshay Sharma

Published: 2026-11-14T03:45:02Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [hybrid-search](<https://devfeed.tech/topics/hybrid-search.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Vectors & LLMs](<https://devfeed.tech/topics/vectors-llms.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [8k](<https://devfeed.tech/tags/8k.md>), [account](<https://devfeed.tech/tags/account.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-assistant](<https://devfeed.tech/tags/ai-assistant.md>), [arctic](<https://devfeed.tech/tags/arctic.md>), [cortex](<https://devfeed.tech/tags/cortex.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [dimensions](<https://devfeed.tech/tags/dimensions.md>), [embedding](<https://devfeed.tech/tags/embedding.md>), [embedding-models](<https://devfeed.tech/tags/embedding-models.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [enterprise-ai](<https://devfeed.tech/tags/enterprise-ai.md>), [enterprise-search](<https://devfeed.tech/tags/enterprise-search.md>), [models](<https://devfeed.tech/tags/models.md>), [performance](<https://devfeed.tech/tags/performance.md>), [rag](<https://devfeed.tech/tags/rag.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [search](<https://devfeed.tech/tags/search.md>), [semantic-search](<https://devfeed.tech/tags/semantic-search.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [snowflake-cortex](<https://devfeed.tech/tags/snowflake-cortex.md>), [text](<https://devfeed.tech/tags/text.md>), [vector](<https://devfeed.tech/tags/vector.md>), [vector-databases](<https://devfeed.tech/tags/vector-databases.md>)

### AI overview

Snowflake Cortex makes embedding generation through AI_EMBED straightforward, but choosing among its eight text models requires testing retrieval on the enterprise workload. The article recommends starting with a compact English model for short English documents, evaluating multilingual candidates on cross-language queries, and comparing long-context models with chunking for lengthy or mixed-topic documents. Larger vectors and longer context windows do not guarantee better retrieval; dimensions alone are not a quality measure. Semantic similarity can miss exact error codes and identifiers, so hybrid lexical and vector retrieval may help. Evaluate candidates on representative queries and relevant documents, weigh failure impact, check account and regional availability and current pricing, and record model and chunking versions because changing models may require re-embedding the corpus. No comparative benchmark winner is claimed.

### Source excerpt

Snowflake Cortex offers eight text embedding models, but which one should you choose? Explore vector dimensions, multilingual retrieval, performance, and cost.

## Bing now has an "-ai" operator for search results without the AI

DevFeed: [Bing now has an "-ai" operator for search results without the AI](<https://devfeed.tech/articles/bing-now-has-an-ai-operator-for-search-results-without-the-ai-133449.md>)

Original publisher: [Read original article](<https://www.techspot.com/news/114557-bing-now-has-ai-operator-search-results-without.html>)

Author: Alfonso Maruccia

Published: 2026-11-13T23:24:00Z

Content type: news

Language: en

Sources: [TechSpot](<https://devfeed.tech/sources/techspot.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [Chrome extension](<https://devfeed.tech/topics/chrome-extension.md>), [Extension](<https://devfeed.tech/topics/extension.md>), [Google Search](<https://devfeed.tech/topics/google-search.md>), [Google Chrome](<https://devfeed.tech/topics/google-chrome.md>), [Edge](<https://devfeed.tech/topics/edge.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-answers](<https://devfeed.tech/tags/ai-answers.md>), [ai-features](<https://devfeed.tech/tags/ai-features.md>), [ai-industry](<https://devfeed.tech/tags/ai-industry.md>), [ai-mode](<https://devfeed.tech/tags/ai-mode.md>), [ai-search](<https://devfeed.tech/tags/ai-search.md>), [ai-summaries](<https://devfeed.tech/tags/ai-summaries.md>), [bing](<https://devfeed.tech/tags/bing.md>), [browser-extension](<https://devfeed.tech/tags/browser-extension.md>), [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [choice](<https://devfeed.tech/tags/choice.md>), [chrome](<https://devfeed.tech/tags/chrome.md>), [chromium](<https://devfeed.tech/tags/chromium.md>), [custom](<https://devfeed.tech/tags/custom.md>), [data-center](<https://devfeed.tech/tags/data-center.md>), [extensions](<https://devfeed.tech/tags/extensions.md>), [generative-ai](<https://devfeed.tech/tags/generative-ai.md>), [google](<https://devfeed.tech/tags/google.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [microsoft-edge](<https://devfeed.tech/tags/microsoft-edge.md>), [operator](<https://devfeed.tech/tags/operator.md>), [search](<https://devfeed.tech/tags/search.md>), [the-web](<https://devfeed.tech/tags/the-web.md>), [users](<https://devfeed.tech/tags/users.md>), [web](<https://devfeed.tech/tags/web.md>)

### AI overview

Microsoft's Bing offers two ways to suppress its AI summaries: install the Microsoft Bing AI Search Choice browser extension, which works with Chrome and Edge and lets users toggle AI summaries, or add the "-ai" operator to a query. Users can configure a custom search engine to append the operator automatically. The text says the operator is an option for users who do not want to install an add-on; it does not establish that either method removes AI features beyond Bing's service. It also cites a study estimating frequent errors in Google AI Overviews, which Google disputes, and reports disagreement over Google's AI-related policies in schools.

### Source excerpt

After being among the first to incorporate ChatGPT-style AI answers on Bing, Microsoft has acknowledged that some people don't want to deal with AI's answers in their web searches. The company has now unveiled a browser extension designed to give users a choice between "full" AI-powered search results and traditional... Read Entire Article

## Understanding span queries in Amazon OpenSearch Service

DevFeed: [Understanding span queries in Amazon OpenSearch Service](<https://devfeed.tech/articles/understanding-span-queries-in-amazon-opensearch-service-133390.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/understanding-span-queries-in-amazon-opensearch-service/>)

Author: Jon Handler

Published: 2026-11-13T22:00:29Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [opensearch](<https://devfeed.tech/topics/opensearch.md>), [Amazon OpenSearch Service](<https://devfeed.tech/topics/amazon-opensearch-service.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-opensearch-service](<https://devfeed.tech/tags/amazon-opensearch-service.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-cloud](<https://devfeed.tech/tags/aws-cloud.md>), [edit-distance](<https://devfeed.tech/tags/edit-distance.md>), [expert-400](<https://devfeed.tech/tags/expert-400.md>), [opensearch](<https://devfeed.tech/tags/opensearch.md>), [queries](<https://devfeed.tech/tags/queries.md>), [query](<https://devfeed.tech/tags/query.md>), [span](<https://devfeed.tech/tags/span.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Span queries let Amazon OpenSearch Service users express term-order, position, proximity, and containment constraints that phrase queries cannot reliably capture. Phrase queries match contiguous terms; `match_phrase` slop allows positional swaps, which can produce matches with terms in an unexpected order. Span queries combine span-only clauses such as `span_term`, `span_near`, `span_first`, `span_not`, `span_containing`, and `span_within`; `in_order` controls clause ordering, while containment queries differ in whether they return the larger or smaller span. The examples show anchoring a term near a field's start and requiring an exact phrase to occur inside a broader span. This precision can help legal, regulatory, patent, scientific, and compliance searches where placement matters, and offers a query-time option when text lacks a structured model. Span queries are more verbose and require careful composition.

### Source excerpt

Among OpenSearch query types, span queries give you precise control over where terms appear in a document and how they relate to one another. Learn how span queries work, when to use them instead of phrase queries, and how to compose them to enforce exact term order, position, and containment that phrase queries cannot express.

## Too much of a good thing: Why 20 candidates beat 200 when reranking search results

DevFeed: [Too much of a good thing: Why 20 candidates beat 200 when reranking search results](<https://devfeed.tech/articles/too-much-of-a-good-thing-why-20-candidates-beat-200-when-reranking-search-results-132453.md>)

Original publisher: [Read original article](<https://www.elastic.co/search-labs/blog/jina-reranker-candidate-pool-size>)

Author: Scott Martens,Sofia Vasileva

Published: 2026-11-13T16:15:00Z

Content type: article

Language: en

Sources: [Elasticsearch Labs](<https://devfeed.tech/sources/elasticsearch-labs.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [elasticsearch](<https://devfeed.tech/topics/elasticsearch.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [all](<https://devfeed.tech/tags/all.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bm25](<https://devfeed.tech/tags/bm25.md>), [but](<https://devfeed.tech/tags/but.md>), [elasticsearch](<https://devfeed.tech/tags/elasticsearch.md>), [good](<https://devfeed.tech/tags/good.md>), [help](<https://devfeed.tech/tags/help.md>), [jina-ai](<https://devfeed.tech/tags/jina-ai.md>), [just](<https://devfeed.tech/tags/just.md>), [ml-research](<https://devfeed.tech/tags/ml-research.md>), [more](<https://devfeed.tech/tags/more.md>), [relevance](<https://devfeed.tech/tags/relevance.md>), [reranker](<https://devfeed.tech/tags/reranker.md>), [reranking](<https://devfeed.tech/tags/reranking.md>), [results](<https://devfeed.tech/tags/results.md>), [search](<https://devfeed.tech/tags/search.md>), [semantic](<https://devfeed.tech/tags/semantic.md>), [semantic-search](<https://devfeed.tech/tags/semantic-search.md>), [thing](<https://devfeed.tech/tags/thing.md>), [v3-5](<https://devfeed.tech/tags/v3-5.md>)

### AI overview

Across 25 retrieval benchmarks, jina-reranker-v3.5 performed best with different candidate pool sizes depending on first-stage retrieval: about 100 candidates for Elasticsearch BM25 and 20 for jina-embeddings-v5-text-small semantic search. With BM25, 50 candidates retained nearly all the accuracy of 100 while halving token use; with semantic search, 200 candidates scored worse on average than using no reranker. The article attributes this to listwise rerankers processing candidates together, making them vulnerable to noise and attention dilution. Better initial retrieval leaves less for reranking to improve. Results varied by benchmark, so the authors recommend measuring on the target data. For Elasticsearch, they outline adding a text_similarity_reranker and tuning rank_window_size: start at 100 for BM25 or 20 for semantic search, then compare performance.

### Source excerpt

We expected more candidates to help, but across 25 benchmarks in Elasticsearch jina-reranker-v3.5 peaked at 100 candidates for BM25 and just 20 for semantic search, where 200 scored worse than no reranker at all.

## AI Search as an Information Intermediary: Clickstream Evidence on Web Traffic

DevFeed: [AI Search as an Information Intermediary: Clickstream Evidence on Web Traffic](<https://devfeed.tech/articles/ai-search-as-an-information-intermediary-clickstream-evidence-on-web-traffic-131947.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2607.07652>)

Author: Qiaoni Shi, Kai Gu, Kai Zhu

Published: 2026-11-13T05:00:00Z

Content type: news

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>), [information retrieval](<https://devfeed.tech/topics/information-retrieval.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Google Search](<https://devfeed.tech/topics/google-search.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-search](<https://devfeed.tech/tags/ai-search.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [econ-gn](<https://devfeed.tech/tags/econ-gn.md>), [q-fin-ec](<https://devfeed.tech/tags/q-fin-ec.md>), [web-traffic](<https://devfeed.tech/tags/web-traffic.md>)

### AI overview

Using U.S. desktop clickstream data during ChatGPT Search's rollout, the study finds that referrals accompanied 5.4% of ChatGPT sessions, compared with 31.1% of Google queries. Ad-supported sites received a smaller share of ChatGPT referrals, while reference and technical sites received larger shares. Conventional searches on Google, Bing, and Yahoo declined after ChatGPT adoption, with declines occurring sooner when adoption followed the expansion of ChatGPT Search to users without accounts. That expansion increased ChatGPT use among existing users; conventional search showed no detectable overall change, though it fell among households with lower prior search activity. These findings describe how AI search may affect open-web traffic through both referrals and changes in conventional search.

### Source excerpt

arXiv:2607.07652v2 Announce Type: replace-cross Abstract: AI search retrieves web information to ground generated answers. We study this new information intermediary using U.S. desktop clickstream data observing use alongside referrals during ChatGPT Search's rollout. Referrals accompany 5.4% of ChatGPT sessions versus 31.1% of Google queries. Ad-supported sites receive a smaller share of ChatGPT than Google referrals, while reference and technical sites receive larger shares. Conventional searches (Google, Bing, Yahoo) decline after ChatGPT adoption, sooner when adoption follows ChatGPT Search's expansion to users without accounts. Among existing users, that expansion raises ChatGPT use; conventional searches show no detectable overall change but fall among households with lower prior search activity. These findings suggest AI search can reshape open-web traffic through its own referrals and changes in conventional search.

[Next page](<https://devfeed.tech/topics/search-engines.md?cursor=WyIyMDI2LTExLTEzVDA1OjAwOjAwKzAwOjAwIiwgImQwMjZjYzNhLTA3MDYtNDQxOS04Nzc0LTY5YTY5YjJiOTNhNCJd>)