# search-engines Published articles for search-engines. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers DevFeed: [Cloudflare Adds Setting to Block AI Training Crawlers While Allowing Search Crawlers]() Original publisher: [Read original article]() Author: Alex Harper Published: 2026-09-16T17:18:57Z Content type: news Language: en Sources: [Web Designer Depot]() Topics: [Cloudflare](), [AI Bots](), [Crawler](), [Google Search](), [Artificial Intelligence]() Tags: [ai](), [ai-crawlers](), [ai-tech](), [ai-training](), [anthropic](), [artificial-intelligence](), [cloudflare](), [content-protection](), [future-of-the-web](), [google-extended](), [google-search](), [googlebot](), [openai](), [publishers](), [robots-txt](), [search](), [search-engines](), [web](), [web-design](), [web-development](), [web-publishing](), [web-scraping](), [website-traffic]() ### AI overview Cloudflare launched a Disallow AI Training setting that lets website owners allow traditional search crawlers while blocking training-only crawlers from companies including Amazon, Anthropic, Meta, and OpenAI. The article notes that robots.txt depends on crawler compliance and that blocking Google-Extended does not remove content from Google Search features such as AI Overviews or AI Mode. ### Source excerpt Cloudflare just gave website owners a new weapon against AI crawlers: keep the search traffic, block the AI training. After years of watching bots consume the web's content, publishers finally have an easier way to tell AI companies where to go. ## Schema markup for SEO: What it is and how it works DevFeed: [Schema markup for SEO: What it is and how it works]() Original publisher: [Read original article]() Author: Webflow Team Published: 2026-06-24T00:00:00Z Content type: tutorial Language: en Sources: [Webflow Blog]() Topics: [Structured-data](), [Search engine optimization (SEO)](), [Website](), [AI search]() Tags: [ai-search](), [schema](), [search](), [search-engines](), [seo](), [strategy](), [web]() ### AI overview This tutorial explains schema markup, also called structured data, and how to add relevant Schema.org properties to web pages. It describes how structured data can support rich search results, SEO visibility, user experience, and discovery in AI search, while noting that it does not guarantee higher rankings. ### Source excerpt Learn how schema markup for SEO can drive more traffic to your site by giving search engines the right information for rich search results. ## From Static Catalogue to Smart Discovery: Engineering Lightning-Fast Search at DreamSetGo DevFeed: [From Static Catalogue to Smart Discovery: Engineering Lightning-Fast Search at DreamSetGo]() Original publisher: [Read original article]() Author: Dream Blog Published: 2026-03-09T13:50:30Z Content type: article Language: en Sources: [Dream11 Engineering]() Topics: [PostgreSQL](), [Microservice](), [systems](), [elasticsearch]() Tags: [architecture](), [dream-sports](), [elasticsearch](), [engineering](), [open](), [performance](), [postgresql](), [search](), [search-engines](), [sports](), [tech](), [travel]() ### AI overview DreamSetGo's tech team explains how its growing sports-travel catalogue created a need for faster, more intuitive product discovery. The article describes rebuilding the catalogue as a microservice, considering PostgreSQL with Elasticsearch, and choosing PostgreSQL based on architectural and operational trade-offs. ### Source excerpt By the DreamSetGo Tech Team If you're excited about building high-performance systems that power premium sports travel experiences, explore our open roles at DreamSetGo. Why Search Became a Product Imperative Over the past couple of years, DreamSetGo's catalogue didn't just grow, it accelerated. We were introducing access to new tournaments, adding destinations across continents, and were expanding into different hospitality formats. What began as a few structured products quickly evolved into a layered catalogue of packages and experiences. And that's when we started noticing something subtle but important. Adding products was straightforward. Making them discoverable wasn't. Every new experience meant longer descriptions and richer metadata. On paper, this was definitely progress. But from a user perspective, it introduced friction. If a traveller already knew exactly where to click, they could navigate their way through the catalogue. But if they came in with intent like: "Wimbledon finals hospitality" "Hot air balloons in Cappadocia" "Scuba diving in the Andamans" ...there was no fast, intuitive way to show them the most relevant experience. Once we acknowledged this gap, the next step became clear: we needed a search system that was scalable, resilient, and fast enough to keep up with our growth. This blog takes you through how we designed that search, from architectural trade-offs to production optimisations, and why PostgreSQL turned out to be the best choice for this. The Obvious Architecture (That We Didn't Choose) We started revamping our architecture six months back and rebuilt our travel product catalogue as a dedicated microservice, one that would eventually power search, and filter it by date, price and more. At that point, we had roughly 1,000 products. But we weren't building for 1,000. We had a clear vision of significantly expanding the catalogue within the near future. Like most teams, our first instinct was predictable: PostgreSQL -> Source of Truth ## Hybrid Search -- Where Keywords Meet Vectors, Enabling Classifieds Discovery DevFeed: [Hybrid Search -- Where Keywords Meet Vectors, Enabling Classifieds Discovery]() Original publisher: [Read original article]() Author: Inês Soveral Published: 2025-09-09T15:01:44Z Content type: article Language: en Sources: [OLX]() Topics: [AI search](), [Query (disambiguation)]() Tags: [article](), [classifieds](), [discovery](), [hybrid-search](), [retrieval](), [search](), [search-engines](), [vector-search]() ### AI overview OLX describes its transition from keyword matching to hybrid search, combining keyword and vector retrieval for classifieds discovery. The article explains the motivations, implementation challenges, solutions, and observed benefits, including improved handling of vague, misspelled, or differently phrased queries. ### Source excerpt Hybrid Search -- Where Keywords Meet Vectors, Enabling Classifieds Discovery In the midst of a fast-paced technological revolution, where user expectations grow increasingly sophisticated, the quality of search in the classifieds space has never been more critical. In fact, users expect the search box to understand what they are looking for and produce relevant search results; if this is not the case, they will easily move on to any competitor who provides this experience. At OLX, our search system traditionally relied on keyword matching between user queries and ad titles and descriptions -- a straightforward but rigid approach. While functional, it often led to low recall or even zero results pages (ZRPs). To mitigate this, we gradually developed an extension chain logic -- which will be explained in detail in later sections -- to address specific edge cases and expand recall. Over time, however, this logic became increasingly complex and difficult to improve upon, calling for a disruptive solution to further evolve our search system. In this article, we share why and how we transitioned to Hybrid Search as a retrieval strategy for our double-sided marketplace. We walk through the key changes required to support semantic search, highlight the challenges we faced, and detail the solutions put in place to overcome them. Finally, we reflect on the tangible improvements and practical benefits observed after this shift. What is Hybrid Search and what value does it bring? Despite its lack of flexibility, keyword matching remains highly effective in e-commerce. It returns results that exactly match the user's query terms, offering clear traceability and helping users understand why specific ads appear. This is particularly useful when users know precisely what they're looking for, and when combined with structured filters, a feature of the OLX marketplace. However, keyword matching falls short when queries are vague, misspelled, or phrased differently from how sellers descri ## Optimizing Hugo Sitemaps to prioritize posts crawling over taxonomies DevFeed: [Optimizing Hugo Sitemaps to prioritize posts crawling over taxonomies]() Original publisher: [Read original article]() Author: Tom Published: 2024-05-17T00:00:00Z Content type: tutorial Language: en Sources: [Tomasz Gągor]() Topics: [Hugo](), [Search engine optimization (SEO)](), [Protocol (disambiguation)]() Tags: [google](), [guide](), [hugo](), [hugo-changefreq-settings](), [hugo-site-seo](), [hugo-sitemap-optimization](), [robots](), [search](), [search-engine-optimization](), [search-engines](), [seo](), [seo-sitemap-tuning](), [sitemap-priority-settings](), [structure](), [taxonomy]() ### AI overview A practical guide to customizing Hugo sitemaps so posts receive higher crawl priority than taxonomy pages. It explains adjusting change frequency and priority while avoiding sitemap and breadcrumb issues that can appear in Google Search Console. ### Source excerpt Learn how to optimize your Hugo site's sitemap by adjusting priority and change frequency settings to improve SEO and guide search engine crawlers effectively.