# Data Infrastructure Published articles for Data Infrastructure. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Seagate and WD AI Storage Research Finds Enterprises Rank Storage Above Compute as the AI Bottleneck DevFeed: [Seagate and WD AI Storage Research Finds Enterprises Rank Storage Above Compute as the AI Bottleneck]() Original publisher: [Read original article]() Author: Lyle Smith Published: 2026-09-15T17:23:54Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [Data Infrastructure](), [AI Infrastructure](), [idc](), [Inference](), [Retrieval-Augmented Generation](), [synthetic-data]() Tags: [ai](), [compute](), [data-infrastructure](), [datasets](), [enterprise](), [genai](), [hdd](), [idc](), [inference](), [reports](), [research](), [retrieval](), [storage](), [synthetic-data]() ### AI overview Seagate and WD published separate studies indicating that AI is increasing enterprise storage requirements and extending data retention. Although their headline percentages differ because they asked different questions, both reports point to storage becoming a larger part of AI infrastructure planning alongside growing archive and retrieval needs. ### Source excerpt Seagate and WD published separate AI storage studies within days of each other; the headline numbers: Seagate says 99% of enterprises expect AI to increase their storage requirements over the next three years, while WD's IDC research puts the comparable figure at 74%. Read the fine print, and both reports land in the same directional The post Seagate and WD AI Storage Research Finds Enterprises Rank Storage Above Compute as the AI Bottleneck appeared first on StorageReview.com. ## Streamhouse Working Group proposes a vendor-neutral architecture for real-time data infrastructure DevFeed: [Streamhouse Working Group proposes a vendor-neutral architecture for real-time data infrastructure]() Original publisher: [Read original article]() Author: Alexander Gallego Published: 2026-09-15T00:00:00Z Content type: opinion Language: en Sources: [Redpanda]() Topics: [Data Infrastructure](), [real-time](), [Streaming](), [Artificial Intelligence](), [Redpanda-Connect]() Tags: [ai](), [announcement](), [data-infrastructure](), [real-time](), [redpanda-connect](), [streaming]() ### AI overview Redpanda says it has joined Aiven, Confluent, StreamNative, and Ververica to form the Streamhouse Working Group, which proposes a vendor-neutral architecture for real-time operational data infrastructure. The article distinguishes Streamhouse from the lakehouse by focusing on continuously processing current data for production applications and AI agents. ### Source excerpt Redpanda has joined Aiven, Confluent, StreamNative, and Ververica to form the Streamhouse Working Group. Here's what that means for the future of real-time data infrastructure. ## VDURA Deploys High-Performance Storage Platform for AI and HPC at New Mexico State University DevFeed: [VDURA Deploys High-Performance Storage Platform for AI and HPC at New Mexico State University]() Original publisher: [Read original article]() Author: Harold Fritts Published: 2026-09-02T10:00:00Z Content type: news Language: en Sources: [StorageReview.com]() Topics: [Artificial Intelligence](), [High-Performance Computing](), [Data Infrastructure](), [InfiniBand](), [Post-quantum cryptography](), [Deployment](), [Training AI Models](), [Hardware](), [Low Latency](), [Quantum Computing]() Tags: [ai](), [data-infrastructure](), [deployment](), [enterprise](), [enterprise-storage](), [hardware](), [high-performance-computing](), [infiniband](), [low-latency](), [post-quantum-cryptography-pqc](), [quantum](), [storage]() ### AI overview VDURA has moved its storage platform at New Mexico State University into full production to support AI and high-performance computing research. The deployment combines NVMe flash and high-density HDD tiers through a global namespace over InfiniBand, allowing separate scaling of performance and capacity. It also supports NMSU's post-quantum cryptography research and large-scale data pipeline projects. ### Source excerpt VDURA has completed the deployment of its data platform at New Mexico State University (NMSU), moving the system into full production. The infrastructure is designed to serve the university's research community with a high-durability, high-throughput storage environment tailored specifically for artificial intelligence and high-performance computing (HPC) workloads. NMSU, which holds Carnegie R1 status and manages The post VDURA Deploys High-Performance Storage Platform for AI and HPC at New Mexico State University appeared first on StorageReview.com. ## MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet DevFeed: [MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet]() Original publisher: [Read original article]() Author: Arvind Srinivasan; Neil Spring; Omar Baldonado; Rajiv Krishnamurthy Published: 2026-08-24T18:02:29Z Content type: article Language: en Sources: [Engineering at Meta]() Topics: [Ethernet](), [Networks](), [AI Infrastructure](), [GPU](), [data centers](), [Low Latency](), [telemetry](), [Inference]() Tags: [ai-infrastructure](), [data-center-engineering](), [data-centers](), [data-infrastructure](), [ethernet](), [gpu](), [low-latency](), [networking-traffic](), [open-source](), [performance](), [real-time](), [telemetry]() ### AI overview Meta introduces MetaRoCE, an RDMA transport protocol designed for AI workloads on commodity Ethernet at million-GPU scale. The article describes its release through the Open Compute Project and explains how endpoint intelligence, packet spraying, fine-grained logical paths, and real-time telemetry aim to provide high throughput, low tail latency, and operational simplicity for distributed training and inference. ### Source excerpt Training and serving frontier AI models depends on fast, reliable networks that move data between GPUs without wasting compute cycles. To meet this challenge at scale, Meta designed MetaRoCE - a clean-sheet RDMA transport protocol purpose-built for AI workloads on commodity Ethernet. We're releasing the MetaRoCE specification, a reference software implementation and a compliance test [...] Read More... The post MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet appeared first on Engineering at Meta. ## MTIA 300: Meta's First Training Chip with Built-in NICs and Communication-Offloading Engines DevFeed: [MTIA 300: Meta's First Training Chip with Built-in NICs and Communication-Offloading Engines]() Original publisher: [Read original article]() Author: Rajiv Krishnamurthy; Wes Bland Published: 2026-08-24T17:45:52Z Content type: article Language: en Sources: [Engineering at Meta]() Topics: [recommendation systems](), [Language models]() Tags: [chip-design](), [communication](), [data-infrastructure](), [devinfra](), [gpu](), [hardware](), [inference](), [infrastructure](), [meta](), [networking-traffic](), [performance](), [production-engineering](), [training]() ### AI overview Meta describes MTIA 300, an in-house accelerator for training ranking and recommendation models, with built-in network chiplets and a co-designed HCCL communication library. The design targets communication-heavy distributed training by integrating RDMA NICs into the chip package and offloading communication work. ### Source excerpt MTIA 300 is the first of Meta's family of in-house training and inference accelerators optimized for training ranking and recommendation models. We're sharing how MTIA 300's built-in NIC chiplets allow it to meet the communication needs associated with training recommendation models with superior performance over general-purpose GPUs. By co-designing MTIA's communication library, HCCL, alongside the [...] Read More... The post MTIA 300: Meta's First Training Chip with Built-in NICs and Communication-Offloading Engines appeared first on Engineering at Meta. ## Core Banking Modernization with Temenos Core and CockroachDB DevFeed: [Core Banking Modernization with Temenos Core and CockroachDB]() Original publisher: [Read original article]() Author: Nanda Badrappan,Muruga Balakrishnan Published: 2026-08-24T00:00:00Z Content type: article Language: en Sources: [Cockroach Labs]() Topics: [CockroachDB](), [Cockroach Labs](), [Data Infrastructure](), [Availability](), [real-time](), [Resilience](), [Transactions](), [legacy](), [SQL](), [Cloud](), [systems](), [on-prem]() Tags: [availability](), [cloud](), [cockroach-labs](), [cockroachdb](), [data-infrastructure](), [legacy](), [on-prem](), [real-time](), [resilience](), [sql](), [transactions]() ### AI overview This article explains why banks are modernizing legacy core banking infrastructure. It describes how Temenos Core and CockroachDB are positioned to support real-time transaction processing, continuous availability, distributed operations, resilience, and regulatory requirements. ### Source excerpt Banking has become an always-on business. Customers expect instant payments, accurate balances, and continuous access to financial services... ## From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta's Ads Ranking DevFeed: [From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta's Ads Ranking]() Original publisher: [Read original article]() Author: Steven De Gryze; Parshva Doshi; Sean O'Byrne; Arnold Overwijk; Dinesh Ramasamy; Lee Xiong Published: 2026-08-05T19:20:20Z Content type: article Language: en Sources: [Engineering at Meta](), [Meta ML Applications]() Topics: [recommendation systems](), [Temporal data]() Tags: [ads](), [architecture](), [data-infrastructure](), [meta](), [ml-applications](), [production-engineering](), [recommendation-systems](), [scaling-laws](), [tokenization]() ### AI overview Meta describes a multi-stage sequence-model architecture for ads ranking that separates offline user modeling from lightweight online ranking. It also uses dense tokenization and target-aware attention to learn feature interactions, with reported conversion and ad-click lifts across Instagram and Facebook. ### Source excerpt Every day, Meta's recommendation platforms handle billions of user interactions, generating rich temporal signals that capture individual preferences and intent across products, ads, and content. In our 2024 post on sequence learning for ads recommendations, we showed how modeling the order and timing of user actions (rather than relying on static, manually engineered sparse features) [...] Read More... The post From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta's Ads Ranking appeared first on Engineering at Meta. ## GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model DevFeed: [GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model]() Original publisher: [Read original article]() Author: Darren Liu; Huayu Li; Raghav Boinepalli; Yuzhen Huang; Jackie (Jiaqi) Xu; Richard Qiu; Chunzhi Yang; Rich Zhu; Dev (Devashish) Shankar; Huaqing Xiong Published: 2026-08-03T18:00:17Z Content type: article Language: en Sources: [Engineering at Meta](), [Meta AI Research](), [Meta ML Applications]() Topics: [recommendation systems](), [networking]() Tags: [ads](), [ai-infrastructure](), [ai-research](), [data-infrastructure](), [gpu](), [hardware](), [kernels](), [llm](), [meta](), [ml-applications](), [networking](), [recommendation-systems](), [training]() ### AI overview Meta describes training its GEM ads recommendation foundation model at LLM scale. The article covers recommendation-specific kernels, ultra-low-precision training, and topology-aware parallelism that doubled end-to-end training efficiency to 20-25% MFU while increasing training FLOPs fourfold. ### Source excerpt Meta's Generative Ads Recommendation Model (GEM), the foundation model behind ads recommendations across Instagram and Facebook, now trains at LLM scale on several thousand of the latest-generation GPUs. This post goes into the details on how we achieved: doubling end-to-end (E2E) training efficiency to 20-25% Model FLOPs Utilization (MFU) while scaling training FLOPs 4x in [...] Read More... The post GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model appeared first on Engineering at Meta. ## Data Engineering Weekly #280 DevFeed: [Data Engineering Weekly #280]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-07-27T03:37:19Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [Data Infrastructure](), [Open Source](), [Data Quality](), [DuckDB](), [Database]() Tags: [data](), [data-engineering](), [data-infrastructure](), [data-quality](), [database](), [duckdb](), [engineering](), [open-source](), [python]() ### AI overview Data Engineering Weekly #280 is a newsletter roundup covering updates to leetdata.ai and aidataengineer.io, agent-oriented data infrastructure, open-source modern data stack tools, data-tool landscapes, metric certification, and data quality in the AI era. ### Source excerpt The Weekly Data Engineering Newsletter ## The data platform 1Password needed didn't exist. So we built it. DevFeed: [The data platform 1Password needed didn't exist. So we built it.]() Original publisher: [Read original article]() Author: info@1password.com (Wayne Duso; Mandy Gu; and Amie Bright) Published: 2026-07-22T00:00:00Z Content type: article Language: en Sources: [Blog on 1Password Blog]() Topics: [data](), [Data Infrastructure](), [data-governance](), [real-time](), [Unified Access](), [Artificial Intelligence]() Tags: [agentic](), [ai](), [building-1password](), [data](), [data-governance](), [data-infrastructure](), [developers](), [real-time](), [sql](), [systems](), [unified-access](), [workflows]() ### AI overview 1Password describes building an internal data platform to make business data accessible, trustworthy, and available in real time across product, finance, engineering, analytics, SQL, APIs, dashboards, and AI workflows. The effort addresses bottlenecks caused by a centralized data lake, bespoke pipelines, and tightly coupled storage, governance, and compute. ### Source excerpt Consider a few tasks that take place across every business, every day: A product team ships a feature and wants to know if customers are using it successfully. A finance team needs customer and account information for planning. An analyst needs definitions to create a report. An AI assistant needs operational context to answer a business question. Those sound like different workflows, but they all rely on the same underlying data. And each one of these actors, across each of these teams, needs that data to be both accessible and trustworthy. At 1Password, trust is at the center of everything we build. Millions of people and businesses rely on us to protect the credentials, secrets, and access workflows that power modern work. The same principle applies to our own internal data. As our products, systems, and use of AI evolved, data became a shared dependency across the business. It powers everything from Unified Access andsecure agentic access patterns for customers, to the workflows used by product, finance, and engineering teams. As those systems grew, so did the number of people and applications that depended on our internal data. But our data infrastructure did not respond well to this. We had built a centralized data lake supported by a growing collection of bespoke data pipelines and one-off solutions. Each new use case required another integration or transformation. Over time, the data platform became a bottleneck: teams turned to CSVs to move faster, and data engineers spent more time maintaining pipelines than enabling new capabilities. What we learned was that manually moving data was no longer enough. Data needs to be available in real time, accessible wherever it's needed, and trusted through the forms our customers need, whether that is through SQL, APIs, dashboards or AI workflows. Privacy and governance need to be built into data the moment it's created so every downstream use remains secure and unambiguous by design. This is the foundation we're build ## ClickHouse announced as Principal Partner and Front of Shirt Sponsor of Fulham Football Club DevFeed: [ClickHouse announced as Principal Partner and Front of Shirt Sponsor of Fulham Football Club]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-07-21T08:09:34Z Content type: news Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [real-time](), [data](), [Artificial Intelligence](), [Open Source]() Tags: [analytics](), [applications](), [artificial-intelligence](), [clickhouse](), [company](), [data-infrastructure](), [global](), [innovation](), [open-source](), [partnership](), [performance](), [real-time](), [technology]() ### AI overview ClickHouse has become Fulham Football Club's Principal Partner and Front of Shirt Sponsor of the Men's First Team in a multi-year partnership. The collaboration combines sponsorship, supporter-facing experiences, and opportunities to develop skills in data and artificial intelligence, while highlighting ClickHouse's role in real-time analytics and open-source database technology. ### Source excerpt Fulham Football Club is proud to announce ClickHouse, a global technology company specialising in real-time analytics, data infrastructure and AI applications, as its newest Principal Partner and Front of Shirt Sponsor of the Men's First Team. ## One Driver, One Format, Every Language: ADBC DevFeed: [One Driver, One Format, Every Language: ADBC]() Original publisher: [Read original article]() Author: Luke Gannon Published: 2026-07-10T15:00:55Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Databases](), [data](), [Data Science](), [R](), [Ruby](), [Go Language](), [JavaScript](), [Python]() Tags: [apache](), [api](), [c](), [clickhouse](), [data](), [data-infrastructure](), [data-science](), [database](), [databases](), [go](), [java](), [javascript](), [python](), [ruby]() ### AI overview ClickHouse has introduced an official ADBC driver, providing Arrow-native, zero-conversion access to ClickHouse across multiple programming languages. Built with Rust and distributed through the ADBC Driver Foundry, it gives languages such as Ruby, R, and C standardized database access without separate ClickHouse drivers. ### Source excerpt ClickHouse now has an official ADBC driver, giving Ruby, R, C, and every other ADBC-aware tool zero-conversion, Arrow-native access to ClickHouse without a dedicated client for each language. ## 10 Years of Meta's Commitment to Python DevFeed: [10 Years of Meta's Commitment to Python]() Original publisher: [Read original article]() Author: Chris Wiltz Published: 2026-06-30T16:00:46Z Content type: opinion Language: en Sources: [Meta AI Research](), [Meta ML Applications]() Topics: [Meta](), [Python](), [Open Source](), [Programming language](), [Software](), [PyTorch]() Tags: [ai-research](), [culture](), [data-infrastructure](), [devinfra](), [engineering](), [meta](), [ml-applications](), [open-source](), [production-engineering](), [programming](), [python](), [pytorch](), [software]() ### AI overview Meta reflects on its 10th consecutive year sponsoring the Python Software Foundation and explains Python's importance across its engineering stack, including products, infrastructure, and AI research. ### Source excerpt This year marks Meta's 10th consecutive year as a sponsor of the Python Software Foundation (PSF), the charitable organization dedicated to advancing, supporting, and protecting the open-source Python programming language and the community that sustains it. Python is one of the world's most influential programming languages, and we use it across our engineering stack, from [...] Read More... The post 10 Years of Meta's Commitment to Python appeared first on Engineering at Meta. ## Core dump epidemiology: fixing an 18-year-old bug DevFeed: [Core dump epidemiology: fixing an 18-year-old bug]() Original publisher: [Read original article]() Published: 2026-06-30T00:00:00Z Content type: article Language: en Sources: [OpenAI News]() Topics: [bug](), [OpenAI](), [Rockset](), [C++](), [Data Infrastructure](), [Azure](), [Hardware](), [race-condition](), [Open Source](), [Streaming]() Tags: [azure](), [bug](), [c-plus-plus](), [data-infrastructure](), [engineering](), [hardware](), [open-source](), [openai](), [race-condition](), [software]() ### AI overview OpenAI engineers analyzed a large population of core dumps to investigate rare crashes in the Rockset service. The investigation uncovered two unrelated causes: silent CPU corruption on an Azure host and an 18-year-old race condition in GNU libunwind. ### Source excerpt OpenAI engineers used large-scale core dump analysis to debug rare infrastructure crashes, uncovering both a hardware fault and a long-standing software bug. ## Customer Analysis for SaaS: A Framework for Real Decisions DevFeed: [Customer Analysis for SaaS: A Framework for Real Decisions]() Original publisher: [Read original article]() Author: Ayush Agarwal Published: 2026-06-13T00:00:00Z Content type: tutorial Language: en Sources: [Dodo Payments Blog]() Topics: [Software as a service](), [data](), [Data Infrastructure](), [Framework]() Tags: [analytics](), [churn](), [data](), [data-infrastructure](), [growth](), [guide](), [how-to](), [retention](), [saas]() ### AI overview This guide presents a framework for SaaS customer analysis, covering segmentation, customer behavior, willingness to pay, retention drivers, and the data infrastructure needed to turn customer data into actionable business decisions. ### Source excerpt Customer analysis framework for SaaS founders. Segmentation, behavior, willingness-to-pay, retention drivers, and the data infrastructure to make it actionable. ## Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world DevFeed: [Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world]() Original publisher: [Read original article]() Author: Patrick Lam Published: 2026-06-09T17:01:02Z Content type: article Language: en Sources: [The Airbnb Tech Blog - Medium]() Topics: [data-architecture](), [Data Infrastructure](), [data-modeling](), [data]() Tags: [analytics](), [architecture](), [data](), [data-architecture](), [data-engineering](), [data-infrastructure](), [data-modeling](), [engineering](), [offline](), [post](), [technology]() ### AI overview Airbnb's data and analytics engineering teams evolved a decade-old offline data warehouse to support Homes, Experiences, and Services. The article examines the trade-offs between separate product-specific data models and a unified monolithic model while describing the need for a consistent, flexible, and scalable data foundation. ### Source excerpt How Airbnb's data engineers and analytics engineers built a consistent and flexible data modeling framework to support the expansion into Homes, Experiences, and Services. By: Patrick Lam, Namrata Lamba, Jamie Stober With the May 2025 Summer Release, Airbnb redesigned its app, relaunched Experiences, and debuted Services, pushing us beyond our traditional Homes focus. For the data teams, this meant rapidly evolving a decade-old infrastructure to integrate two brand-new product pillars. Our data engineers and analytics engineers rose to the challenge by building a consistent and flexible framework to serve as a robust and scalable data foundation for the next decade of growth. But getting there wasn't straightforward. This fundamental shift surfaced a critical question for our data organization: How do you evolve your offline data architecture to support new product lines without introducing disorder in vital analytics services? We knew the approach we took would have long-lasting implications. A fragmented strategy risked creating data silos, inconsistent analytics, and a tangled web of technical debt that would likely slow down future innovation. In this post, we'll take you behind the scenes to share key decisions that we made, the framework that emerged, and the lessons that helped reshape our offline data warehouse for the future. Note that we focus specifically on our offline data warehouse (the analytics-oriented data infrastructure owned by our data engineers and analytics engineers) rather than the online data systems that serve the app directly, as the two domains have fundamentally different requirements, constraints, and design philosophies that warrant separate treatment. The core dilemma: separate vs. monolithic The first and most critical question was how to structure offline data for the new, three-product world, with Homes, a refreshed Experiences product, and the new Services offering. This involved a trade-off between two main approaches: Separate ## Real-time streaming for the agentic era with NVIDIA DevFeed: [Real-time streaming for the agentic era with NVIDIA]() Original publisher: [Read original article]() Author: Melissa Czapiga Published: 2026-06-01T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Vera CPU](), [streaming-data-processing](), [Latency](), [Data Infrastructure](), [agent observability](), [systems]() Tags: [agentic](), [agents](), [announcement](), [compute](), [data-infrastructure](), [latency](), [nvidia](), [nvidia-vera](), [performance](), [streaming](), [systems]() ### AI overview Redpanda describes its collaboration with NVIDIA to run streaming workloads on NVIDIA Vera CPUs, reporting 5.5x lower latency for AI agents in mission-critical environments. The article discusses testing on workloads processing billions of messages daily, real-time data infrastructure, and potential benefits for enterprise AI and agentic applications. ### Source excerpt NVIDIA Vera launches today with Redpanda as part of the ecosystem, delivering 5.5x lower latencies for agents running in mission-critical environments. ## AI Agents Need Context to Reason, Not Just Data DevFeed: [AI Agents Need Context to Reason, Not Just Data]() Original publisher: [Read original article]() Author: Quentin Packard Published: 2026-05-28T00:00:00Z Content type: article Language: en Sources: [Cockroach Labs]() Topics: [Artificial Intelligence](), [context](), [AI Agent](), [Agentic AI Architecture](), [Data Infrastructure]() Tags: [agentic-ai](), [agents](), [ai](), [ai-architecture](), [context](), [data-infrastructure](), [database]() ### AI overview The article argues that production failures in AI agents often stem from inadequate context management rather than the model itself. Reliable agent behavior requires current data, memory, permissions, observability, and awareness of system constraints, making context management a data infrastructure problem beyond basic retrieval or prompt engineering. ### Source excerpt When your AI agent makes a bad decision in production, what do you blame? ## ClickHouse tops $250M ARR and 4,000 customers, launches Claude-powered agents at Open House 2026 DevFeed: [ClickHouse tops $250M ARR and 4,000 customers, launches Claude-powered agents at Open House 2026]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-05-27T17:57:17Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [AI Chat](), [AI Bots]() Tags: [agents](), [analytics](), [announcements](), [anthropic](), [benchmark](), [claude](), [clickhouse](), [cloud](), [data-infrastructure](), [serverless](), [skills]() ### AI overview ClickHouse announced that its serverless cloud offering exceeded $250 million in annual run-rate revenue and reached 4,000 customers. It also introduced Claude-powered ClickHouse Agents and published the CostBench cloud data warehouse benchmark. ### Source excerpt ClickHouse today opened Open House 2026, its second annual user conference, with a set of announcements that mark one of the company's most active quarters since founding. ## Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use DevFeed: [Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-05-21T16:01:00Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [data](), [recommendation systems](), [datasets](), [Machine learning](), [Embeddings](), [Feature Engineering](), [Inference](), [Latency](), [systems]() Tags: [data](), [data-infrastructure](), [datasets](), [embeddings](), [engineering](), [feature-engineering](), [inference](), [latency](), [machine-learning](), [pinterest](), [production](), [recommendation-system](), [recommendation-systems](), [retrieval](), [systems](), [train](), [transformers]() ### AI overview Pinterest describes a redesign of its user-sequence platform for ranking, retrieval, and recommendation workloads. The article explains how enriched event sequences support training datasets, offline analysis, online inference, and latency-sensitive production use cases, with goals of reducing cost, improving extensibility, and simplifying debugging. ### Source excerpt Authors (listed alphabetically) Ads Feature Engineering Infra team: Ajay Venkatakrishnan, Le Zhang Core ML Infra team: Eric Shang, Pihui Wei ML Data team: Connor Votroubek, Yi He User Understanding team: Camilo Munoz, Simin Li If you work on ranking, retrieval, or recommendation systems, you've probably asked for some version of the same thing: "Give me the last N meaningful actions this user took, with the right enrichments, in a format that's easy to train and serve ML models." On paper, that sounds simple. In practice, "user sequences" often become one of the most expensive and fragile parts of the ML data stack. They end up powering everything from training datasets to offline analysis and online inference, so they need to be fresh and complete at the same time. They must remain consistent as you add new events and enrichments. And they have to do all of this while serving latency-sensitive production workloads. This article walks through how we redesigned our user-sequence platform to make these sequences cheaper to run, faster to extend, and easier to debug, while still supporting demanding production use cases. What We Mean by "User Sequence" In this context, a user sequence is an ordered list of recent, relevant events for a user, along with the enrichments (signals) attached to each event. Here, enrichments mean all the extra signals we attach to raw events, so they're useful for models: embeddings (for example, Pin or query representations), contextual features (such as surface, device, or country), and derived attributes or counters that describe how the user interacted with a piece of content over time. A concrete example helps. Imagine a sequence made up of the last 500 engagements a user had with Pinterest Pins. Each event in that sequence might carry a timestamp, an action type, the surface where the action occurred, and a handful of embedding features or categorical attributes. As a data primitive, user sequences are powerful. They capture temporal b ## How We Built a Zero-Downtime Database Migration Service at Wix DevFeed: [How We Built a Zero-Downtime Database Migration Service at Wix]() Original publisher: [Read original article]() Author: Wix Engineering Published: 2026-05-14T07:19:17Z Content type: article Language: en Sources: [Wix Engineering]() Topics: [Databases](), [migration](), [Data Infrastructure](), [MySQL](), [Scalability]() Tags: [amazon-msk](), [data-infrastructure](), [database](), [database-migration](), [databases](), [db-topology](), [debezium-connector](), [good-tools-doesn-t-fit](), [how-db-mover-works](), [infrastructure](), [migration](), [mysql](), [scalability](), [summary-6-key-focal-points](), [the-challenge](), [the-python-service](), [the-result]() ### AI overview Wix describes DB Mover, an internal service for transparent, zero-downtime database migrations between shared and dedicated MySQL clusters. The article explains the operational risks of shared clusters, migration requirements, and the topology managed by Wix's Data Infrastructure team. ### Source excerpt The Challenge At Wix, multiple applications share the same DB cluster. The reasons vary: consolidating apps from the same domain, grouping several small apps that don't justify a dedicated cluster, or simply optimizing cost and operational overhead. However, this setup comes with a significant risk: every application on the cluster has the potential to impact all the others. One real example: we had a critical service related to user authentication sharing a DB cluster with several other... ## Powering self-driving vehicle analytics at Avride with ClickHouse Cloud DevFeed: [Powering self-driving vehicle analytics at Avride with ClickHouse Cloud]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-05-11T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [data](), [Apache Iceberg](), [telemetry](), [Grafana]() Tags: [analytics](), [apache-iceberg](), [autonomous](), [aws](), [clickhouse](), [cloud](), [data-infrastructure](), [driving](), [grafana](), [latency](), [lidar](), [metrics](), [robots](), [storage](), [streams](), [telemetry](), [tooling](), [visualization]() ### AI overview Avride uses ClickHouse Cloud as the data backbone for its autonomous vehicles and delivery robots, supporting ride-data indexing, metrics, analytics, and internal tooling. Its migration from Apache Iceberg reduced index lookup and ingestion latency. ### Source excerpt Avride replaced Apache Iceberg with ClickHouse Cloud, cutting index lookup latency from 20 seconds to under 100ms and ingestion from hours to seconds. ## From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines DevFeed: [From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines]() Original publisher: [Read original article]() Author: Mahendran Vasagam Published: 2026-05-05T14:00:01Z Content type: article Language: en Sources: [Engineering at Slack]() Topics: [Security](), [OpenSSH](), [Amazon Web Services](), [airflow](), [DataOps](), [Architecture & Design](), [Kubernetes]() Tags: [airflow](), [architecture](), [aws](), [big-data](), [data-analytics](), [data-engineering](), [data-infrastructure](), [kubernetes](), [modernization](), [security](), [ssh](), [uncategorized]() ### AI overview Slack describes migrating more than 700 SSH-based data pipeline jobs to a REST-based architecture across eight data regions, eliminating SSH access to production AWS EMR clusters without downtime. The article explains the security and operational problems that motivated the modernization, including attack-surface exposure, key-management overhead, resource contention, broken connections, zombie jobs, and unreliable job-status detection. ### Source excerpt Excerpt By 2024, Slack's data platform had accumulated 700+ SSH-based operators orchestrating critical data pipelines. We're talking daily search indexing that processed terabytes of data, analytics jobs powering business intelligence, the whole shebang. Every single one of these jobs required direct SSH access to production AWS Elastic MapReduce (EMR) clusters. We had a massive security... ## Gala supercharges analytics performance with ClickHouse on AWS DevFeed: [Gala supercharges analytics performance with ClickHouse on AWS]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-05-04T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Amazon Web Services (AWS)](), [data analytics](), [Data Infrastructure](), [data-platforms](), [telemetry](), [databricks](), [Blockchain]() Tags: [aws](), [blockchain](), [clickhouse](), [data-analytics](), [data-infrastructure](), [data-platforms](), [databricks](), [telemetry]() ### AI overview Gala migrated from Databricks to ClickHouse on AWS to address growing data volumes and improve analytics performance. The article reports a threefold increase in analytics capacity, a 30% cost reduction, and query times reduced from minutes to sub-second. ### Source excerpt Learn how Gala migrated to the ClickHouse to improve analytics performance and cut costs [Next page]()