# data-systems Published articles for data-systems. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## KathDB-FAO: Optimizing Multimodal Queries through Operator Grouping and Code Reuse DevFeed: [KathDB-FAO: Optimizing Multimodal Queries through Operator Grouping and Code Reuse]() Original publisher: [Read original article]() Author: Guorui Xiao, Douglas Brown, Artur Borycki, Magdalena Balazinska Published: 2026-11-03T05:00:00Z Content type: article Language: en Sources: [arXiv: Databases]() Topics: [AI Chat](), [LLM optimization]() Tags: [abstract](), [arxiv](), [call](), [challenge](), [code](), [complex](), [cost](), [cs-ai](), [cs-db](), [data](), [data-systems](), [evaluation](), [execution](), [fewer](), [fine](), [frontier](), [frontier-model](), [multimodal](), [operator](), [optimizing](), [queries](), [reuse]() ### AI overview KathDB-FAO reduces the cost of multimodal analytical queries by breaking them into fine-grained Function-as-Operators (FAOs), then grouping subsets into single functions tailored to each query so execution requires fewer model calls. Because function synthesis with a frontier model also costs resources and synthesized FAOs often recur, the system reuses pre-built functions and functions generated for earlier queries. On SemBench and an additional workload, it achieved quality comparable to the strongest competing system while reducing average end-to-end cost per query by 76%. ### Source excerpt arXiv:2609.28761v2 Announce Type: replace Abstract: Advances in multimodal large language models (multimodal LLMs) allow data systems to answer analytical queries over text and images alongside relational tables, but at a significant cost for large-scale data. Existing systems reduce this cost through rewrite rules written in advance, which cannot use the logic of an individual query or the semantics of the data it processes. To address this challenge, we design, implement, and evaluate KathDB-FAO, a query evaluation system over multimodal data. KathDB-FAO decomposes a complex query into an execution plan of fine-grained steps, which we call Function-as-Operators (FAOs). To reduce execution cost, it searches for subsets of these FAOs to implement together as one function, enabling query-specific optimizations that make fewer model calls. Synthesizing these functions with a frontier model has its own cost, yet we observe that synthesized FAOs often recur across queries. To further reduce plan generation cost, KathDB-FAO uses pre-built functions and reuses functions synthesized for earlier queries. On SemBench and an additional workload, KathDB-FAO achieves quality comparable to the strongest competing system while reducing end-to-end cost per query by 76% on average. ## Recommendation Systems for Exploratory Data Tasks DevFeed: [Recommendation Systems for Exploratory Data Tasks]() Original publisher: [Read original article]() Author: Anna Fariha, Whanhee Cho Published: 2026-11-02T05:00:00Z Content type: article Language: en Sources: [arXiv: Databases]() Topics: [recommendation systems](), [Data analysis](), [Artificial Intelligence]() Tags: [actions](), [agents](), [ai](), [ai-agents](), [bias](), [challenge](), [challenges](), [choice](), [coverage](), [cs-db](), [data](), [data-systems](), [efficiency](), [efficiently](), [exploration](), [goals](), [guide](), [recommendation](), [recommendation-systems](), [systems](), [tasks]() ### AI overview The paper envisions recommendation as a core capability of data systems to help users--and AI agents acting for them--navigate exploratory data tasks (EDTs), where each choice among many joins, transformations, and aggregations shapes later exploration. It frames recommendations along two axes: single-task versus multi-task settings, and single-action versus multi-action suggestions. Its proposed research agenda moves from recommending individual actions to constrained bundles and sequences, then to coordinated recommendations across interconnected tasks. The abstract identifies key challenges: a combinatorial, data-dependent action space whose options can be costly to materialize, and the need to account for context, efficiency, and coordination. It presents a vision and research directions, not a demonstrated recommendation system. ### Source excerpt arXiv:2609.39412v3 Announce Type: replace Abstract: Exploratory Data Tasks (EDTs) involve iteratively steering workflows as users refine their subjective goals based on newly emerging insights. EDTs are performed by millions of users with varying levels of expertise to understand unfamiliar data, discover trends, and identify evidence that informs critical decision-making. However, a key challenge in EDTs is the enormous space of possible actions that one can take at each step: users struggle to choose among thousands of joins, transformations, and aggregations, leading to exploration paralysis. Because EDT workflows are interconnected, each choice shapes subsequent exploration, and suboptimal choices can lead to inefficiency, missed insights, confirmation bias, and incomplete coverage. This calls for intelligent recommendations that efficiently guide users toward optimal EDT actions. We envision recommendation as a core capability of data systems, proactively guiding users toward promising actions that lower the barrier to EDTs. This capability becomes even more critical as AI agents emerge as a new class of users that perform EDTs on behalf of humans. EDT recommendation is challenging because the action space is combinatorial and actions are data-dependent, which require costly materialization. Moreover, recommendation often involves bundles or sequences of actions across interdependent tasks, requiring coordination across tasks. In this paper, we present our vision of EDT recommendation systems along two axes: single-task vs. multi-task settings and single-action vs. multi-action recommendations. We outline a research agenda that progresses from recommending individual EDT actions to constrained bundles and sequences of actions, and ultimately to coordinated recommendations across interconnected EDTs. We identify research directions for incorporating various contexts, addressing efficiency challenges, and coordinating across tasks. ## Referential Regimes: Transformation-Based Identity Profiles for Neutral Substrates DevFeed: [Referential Regimes: Transformation-Based Identity Profiles for Neutral Substrates]() Original publisher: [Read original article]() Author: Denise M. Case Published: 2026-10-30T04:00:00Z Content type: article Language: en Sources: [arXiv: Artificial Intelligence]() Topics: [Formal methods](), [data-processing]() Tags: [classification](), [cs-ai](), [cs-lo](), [data](), [data-systems](), [identity](), [invariant](), [lean](), [lean-4](), [models](), [records](), [substrates](), [transformation]() ### AI overview The paper formalizes how record transformations affect identity under explicitly declared identity bases. It classifies transformation families as preserving, breaking, neutral, or inapplicable, and certifies six classification cases in Lean 4 covering scope decomposition, rule refinement, and forking. The classifications depend on field granularity, allowed transformation steps, and continuity warrants. It also shows that two bases generate the same identity relation exactly when they agree on every admissible step in a shared comparison domain; a single separating step demonstrates a difference. These results make assumptions auditable, but their application to particular referents depends on the adequacy of the declared bases, models, and warrants. ### Source excerpt arXiv:2601.16152v3 Announce Type: replace-cross Abstract: Data systems operating under persistent legal, institutional, and analytic disagreement require shared reference without presupposing agreement about interpretation. This paper investigates identity preservation under transformation, building on the conditional neutrality result of Neutral Substrates (arXiv:2601.14271). We pair candidate identity bases with families of record transformations and classify each family as preserving, breaking, neutral, or inapplicable for a basis. Using invariant-defined identity bases and explicit effect models, we derive and certify six classification cells in Lean 4 for scope decomposition, rule refinement, and forking. The derivations show how classifications depend on field granularity, admissible transformation steps, and explicit continuity warrants. We also characterize the identity relations generated by invariant-preserving steps. Two bases generate equal relations exactly when they agree on the preservation of every admissible step in a shared comparison domain, and a single separating step witnesses a difference. The formalization makes identity claims auditable, provides testable conditions for systems that transform records, and exposes the assumptions behind each classification. The results are conditional on the declared bases, models, and warrants, whose adequacy for particular referents requires independent justification. ## Cloudera and Mistral bring open-weight models closer to enterprise data DevFeed: [Cloudera and Mistral bring open-weight models closer to enterprise data]() Original publisher: [Read original article]() Author: Victoria Gayton Published: 2026-10-29T21:15:23Z Content type: news Language: en Sources: [SiliconANGLE]() Topics: [AI Integration Strategies](), [AI Models](), [ai security]() Tags: [abhas-ricky](), [ai](), [ai-infrastructure](), [applications](), [artificial-intelligence](), [cloudera](), [cloudera-inc](), [collaboration](), [cube-event-coverage](), [customizable](), [data](), [data-systems](), [enterprise](), [enterprise-ai](), [enterprise-data](), [enterprise-data-infrastructure](), [enterprises](), [evolve26](), [evolve2626eventpage](), [flexibility](), [foundation](), [infra](), [marjorie-janiewicz](), [mistral](), [mistral-ai](), [mistral-ai-sas](), [mistral-large-4](), [models](), [news](), [open](), [open-weight-models](), [partnership](), [paul-gillin](), [rebecca-knight](), [thecube]() ### AI overview Cloudera and Mistral are partnering to help enterprises use customizable open-weight models with proprietary data. Cloudera's Abhas Ricky says high-fidelity outputs require grounding models or agents in domain-specific information, which may reside behind closed firewalls. Mistral's Marjorie Janiewicz says open weights let companies control deployment and retain business continuity if they stop using a model. The companies' joint engineering teams help customers select models and infrastructure for use cases; the text names Mistral Large 4 and cites financial services, public sector and manufacturing as areas of interest. It describes the partnership's aims but gives no implementation details or measured results. ### Source excerpt Open-weight models are giving enterprises more choice in how they put proprietary data to work. Turning that flexibility into useful applications requires models and enterprise data systems to work together. Cloudera Inc.'s partnership with Mistral AI SAS brings those pieces together. The collaboration pairs Cloudera's enterprise data foundation with Mistral's customizable models, according to Marjorie [...] The post Cloudera and Mistral bring open-weight models closer to enterprise data appeared first on SiliconANGLE. ## Do We Still Need Data Models? DevFeed: [Do We Still Need Data Models?]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-10-28T04:22:26Z Content type: opinion Language: en Sources: [Data Engineering Weekly]() Topics: [data-modeling](), [data-engineering](), [AI-assisted coding](), [semantic-layer](), [schema design](), [AI Integration Strategies](), [human review]() Tags: [about](), [account](), [actions](), [agent](), [agentic](), [agents](), [alert](), [alerts](), [all](), [also](), [analysis](), [apply](), [build](), [data](), [data-systems](), [help](), [models](), [need](), [systems](), [understand]() ### AI overview Data models remain necessary as agents build data systems: they capture business definitions and design decisions that functioning pipelines and passing tests cannot establish on their own. The article argues models should also help people inspect how an agent interpreted a request, distinguish confirmed rules from inferences, and surface unresolved questions before implementation. In its campaign example, the team must clarify that a response means a purchase within seven days, how recipients link to customers, and how returns count; that timing alone does not show the email caused the purchase. Dimensional modeling remains useful for analytical design, while semantic definitions, ontologies, and decision context can support agent work where needed. These approaches can coexist; broader ontologies require ownership and ongoing maintenance, and teams should check that implemented code, data, and tests match the agent's account. ### Source excerpt As agents build data systems, models must help us understand the decisions behind them. ## Arya Forge: Auditing Validation and Release Approval for Financial Instruction Data DevFeed: [Arya Forge: Auditing Validation and Release Approval for Financial Instruction Data]() Original publisher: [Read original article]() Author: Kushagra Bhatnagar, Deekshith Marla Published: 2026-10-27T04:00:00Z Content type: article Language: en Sources: [arXiv: Software Engineering]() Topics: [Data pipelines](), [Data Quality](), [Data validation](), [audit]() Tags: [auditing](), [case-study](), [checkpoint](), [conversion](), [cost](), [cs-ai](), [cs-cy](), [cs-se](), [data-pipelines](), [data-systems](), [dataset](), [financial](), [integrity](), [model](), [promotion](), [records](), [release](), [retention](), [validation]() ### AI overview Arya Forge separates content validation from permission to release a dataset by combining typed records, current-content checks, and approvals bound to a dataset manifest. In the reported case study, an integrity audit covered 100,000 synthetic records, while a separate arithmetic implementation matched all 15,000 numeric records across twelve formula families. Across 240 scenarios, added financial and release-record checks rejected every injected numeric fault and policy hold, retaining 220 of 240 clean controls; all twenty rejected controls were tested cash-conversion-cycle examples. In 19 persisted-release cases, eleven selected negative cases were blocked and four equivalent representations were rejected. Changes to external evidence after binding were outside ordinary Gold promotion checks. The findings show a tradeoff between strict producer contracts and retaining useful representations. They describe the tested implementation and workloads, not general financial correctness, authenticated approval, expert-validated quality, or downstream model improvement. ### Source excerpt arXiv:2610.34778v1 Announce Type: cross Abstract: Financial instruction-data pipelines must distinguish content validity from permission to release a particular dataset. Arya Forge combines typed records, current-content checks, and approval coverage bound to a dataset manifest. We evaluate its release boundaries in a financial-data systems case study. An integrity audit covers 100,000 synthetic instruction records; a separate arithmetic implementation agrees with all 15,000 numeric records across twelve formula families. In 240 scenarios with 2,400 treatment rows, adding financial and release-record checks to a short-instruction filtering recipe rejects all injected numeric faults and policy holds while retaining 220 of 240 clean controls. The twenty lost controls comprise every tested cash-conversion-cycle example. Nineteen persisted-release cases show that eleven selected negative cases are blocked, four equivalent representations are rejected, and changes to external evidence after binding are outside ordinary Gold promotion's checks. Local processing measurements and controlled checkpoint resumptions characterize cost and recovery through 100,000 input rows. The results expose a tradeoff between strict producer-contract enforcement and useful-representation retention, and identify the protected inputs required for manifest-bound approval. These findings apply to the tested implementation and workloads; they do not establish general financial correctness, authenticated approval, expert-validated quality, or downstream model improvement. ## Structure then Query: Enabling Precise Analytical Queries over Unstructured Documents DevFeed: [Structure then Query: Enabling Precise Analytical Queries over Unstructured Documents]() Original publisher: [Read original article]() Author: Teng Lin Published: 2026-10-23T04:00:00Z Content type: article Language: en Sources: [arXiv: Databases]() Topics: [AI-driven information retrieval](), [search-engines](), [llm-reasoning](), [AI for SQL query generation](), [olap]() Tags: [analysis](), [annotation](), [cs-db](), [cs-ir](), [data-systems](), [databases](), [datasets](), [large-language-models-llms](), [performance](), [queries](), [query-engine](), [retrieval](), [score](), [sql](), [structured]() ### AI overview AnnoIndex aims to make analytical queries over unstructured documents more precise by combining a structured index with staged query execution. SchemaLoop creates hierarchical annotation schemas and uses a lightweight language model to extract values into an index, shifting attribute extraction from each online query to a one-time build. Its SQL-extension-based engine filters documents through that index, then applies extraction steps in increasing cost order and uses LLMs for the remaining documents requiring deeper semantic understanding; extracted attributes are added to the index for future queries. On three real-world datasets, the paper reports an average F1 score of 0.87 and robust results on multi-hop joins and progressive reasoning. The abstract does not provide dataset details or a fuller account of costs and comparisons. ### Source excerpt arXiv:2608.13384v2 Announce Type: replace-cross Abstract: Unstructured documents constitute the majority of enterprise and web data. With the rapid development of large language models(LLMs), researchers have started to build data systems that analyze unstructured textual documents like operating on databases. However, because mainstream retrieval methods still relies on fuzzy matching based on vector similarity, accurately obtaining information and performing structured analysis and reasoning remains a major challenge. To address these limitations, AnnoIndex introduces two core fundamental components. The first is Annotation Index. The system uses a module called SchemaLoop to automatically create hierarchical annotation schemas from the raw corpus, and then uses lightweight language model to extract specific values. It turns scattered unstructured text into a materialized, structured index that enables low-cost filtering and querying. The annotation index avoids the black-box matching of vector similarity and amortizes attribute extraction costs from online queries to a one-time build. The second innovation is a Structured Query Engine. It compiles user questions into execution plans based on SQL extension. It first uses the Annotation Index for precise documents filtering, then gradually applies extraction operations in ascending order of cost, resorting to LLMs only for the remaining minimal fraction of the corpus that require deep semantic understanding. The extracted attributions are merged into the annotation index, reducing the cost of future queries. Experiments on three real-world datasets demonstrate that AnnoIndex consistently outperforms state-of-the-art baselines, achieving the highest average F1 score (0.87) while maintaining robust performance on complex multi-hop join and progressive reasoning queries. ## Local Inference of LLMs on Long Contexts: A Data Systems Perspective DevFeed: [Local Inference of LLMs on Long Contexts: A Data Systems Perspective]() Original publisher: [Read original article]() Author: Mattias D\\"urrmeier, Alberto Lerner Published: 2026-10-21T04:00:00Z Content type: article Language: en Sources: [arXiv: Databases]() Topics: [Local AI](), [LLM optimization](), [long-context](), [Large Language Model](), [AI Infrastructure](), [Artificial Intelligence]() Tags: [attention](), [cache](), [cs-ai](), [cs-db](), [data-management](), [data-systems](), [gpu](), [gpus](), [inference](), [kv-cache](), [llms](), [local-inference](), [long-context](), [quantization](), [sensitive-data]() ### AI overview The abstract reports that long-context local LLM inference is constrained by more than single-GPU memory capacity: self-attention dominates runtime, yet its current structure leaves key GPU resources underused, with the newest GPU often least well served. The study compares model behavior across GPU generations and frames several bottlenecks as data-management problems, including KV-cache paging, cache utilization, and GPU synchronization. It presents experimental observations and recommendations for future work, but the supplied text does not specify those recommendations or detailed measurements. ### Source excerpt arXiv:2610.28223v1 Announce Type: new Abstract: Large language models can solve complex problems across a variety of subjects, such as mathematics and coding. For such complex analytical tasks, models with a long context perform better. Serving models with long context is currently mostly done on powerful systems with multiple GPUs, which are too expensive for most users. An alternative is local inference, or serving a model on a user's own machine, which removes per-token costs and rate limits, besides keeping sensitive data private and giving users full control over the models they run. The common understanding, however, is that local LLMs need compression (e.g., quantization) to run on a single GPU. In this paper, we set out to study the limitations that prevent serving long-context models for single users, besides the capacity issue. Rather than characterizing a single model on one GPU, we analyze how different models behave across several GPU generations. This lets us uncover not just individual bottlenecks but broader trends in where inefficiencies arise. Our analysis confirms that, as expected, self-attention dominates the runtime for long-context inference. More unexpectedly, our evaluation reveals that the current structure of self-attention leaves key GPU resources significantly underutilized, and the newest GPU is often the least well served. Many of these findings are fundamentally data management problems, from paging the KV cache to improving cache utilization and synchronization in the GPU, which is why we bring this study to the Data Systems community. To engage this community, we present our findings as a series of experimental observations and formulate recommendations for future work. ## Collected, stored, and useless: The lifecycle of most customer data DevFeed: [Collected, stored, and useless: The lifecycle of most customer data]() Original publisher: [Read original article]() Author: Jesse Sumrak Published: 2026-09-23T00:00:00Z Content type: article Language: en Sources: [Twilio]() Topics: [data](), [data-processing](), [systems](), [Orchestration](), [Software]() Tags: [analytics](), [customer-data](), [data](), [data-systems](), [industry-insights](), [orchestration](), [software]() ### AI overview This article examines why customer data remains siloed, duplicated, outdated, or unused despite extensive collection. It argues that businesses should improve data reliability and shorten the path from capture to operational use, with orchestration helping connect systems and records. ### Source excerpt You probably have more data than you know what to do with. The real problem is that the data is sitting in a system that doesn't talk to the ones that need it. ## No more allocation delays: Decoupling snapshots from shard relocation in stateless Elasticsearch DevFeed: [No more allocation delays: Decoupling snapshots from shard relocation in stateless Elasticsearch]() Original publisher: [Read original article]() Author: David Turner,Yang Wang Published: 2026-09-02T00:00:00Z Content type: article Language: en Sources: [Elasticsearch Labs]() Topics: [elasticsearch](), [read-replica](), [control-plane]() Tags: [clusters](), [data-systems](), [elastic](), [elastic-cloud-serverless](), [indexing](), [inside-elastic](), [operations](), [scale](), [snapshot](), [store]() ### AI overview Stateless Elasticsearch snapshots now read directly from the object store, allowing primary shards to relocate while backups run and reducing delays during scale-out. The release eliminated server-side allocation warnings and was associated with cache misses dropping by more than 60% and median cache population throughput increasing by about 50%. ### Source excerpt Clusters scale out under load without waiting for a snapshot to finish, because snapshots now read straight from the object store and no longer pin shards in place. ## Why Open Data Infrastructure matters for data access and control | Blog | Fivetran DevFeed: [Why Open Data Infrastructure matters for data access and control | Blog | Fivetran]() Original publisher: [Read original article]() Author: Annie Sullivan Published: 2026-07-31T23:53:45Z Content type: article Language: en Sources: [Fivetran Blog]() Topics: [Open Data](), [Data Infrastructure](), [Self Hosted AI Tools](), [AI Integration Strategies]() Tags: [ai](), [data-systems](), [distributed](), [infrastructure](), [open-data]() ### AI overview The article argues that agentic AI workflows increase demand for timely access to governed data across operational and analytical systems. It presents Open Data Infrastructure--using open standards, interoperable storage, and decoupled storage and compute--as a way to improve data portability and reuse while maintaining consistent governance. It also notes that distributed systems still require strong governance, operational discipline, observability, and metadata practices. ### Source excerpt Agentic AI puts new pressure on data access, making architectural flexibility and control more important than ever. ## Devavrat Shah's research and Ikigai Labs use tabular data for real-time forecasting and decision-making DevFeed: [Devavrat Shah's research and Ikigai Labs use tabular data for real-time forecasting and decision-making]() Original publisher: [Read original article]() Author: David Chandler | Laboratory for Information and Decision Systems Published: 2026-07-14T20:25:00Z Content type: article Language: en Sources: [MIT AI News]() Topics: [Artificial Intelligence](), [decision-making](), [Structured-data](), [AI Models](), [data](), [Time Series](), [Computer science](), [Electrical engineering and computer science (EECS)](), [Temporal data](), [systems]() Tags: [ai](), [ai-and-decision-making](), [ai-in-business-planning](), [ai-models](), [algorithms](), [artificial-intelligence](), [business-and-management](), [business-modeling](), [celonis-context-model](), [computer-science-and-technology](), [data](), [data-systems](), [decision-making](), [devavrat-shah](), [electrical-engineering-and-computer-science-eecs](), [faculty](), [forecasting](), [idss](), [ikigailabs](), [laboratory-for-information-and-decision-systems-lids](), [mit-eecs](), [mit-idss](), [mit-intellectual-property](), [mit-lids](), [mit-schwarzman-college-of-computing](), [predictions](), [profile](), [real-time](), [research](), [school-of-engineering](), [startups](), [structured](), [tabular-data](), [time-series-data]() ### AI overview MIT Professor Devavrat Shah's research led to a foundation model for tabular and time-series enterprise data. Developed with Ikigai Labs, the system continuously tests predictions against real outcomes to support large-scale forecasting and decision-making. ### Source excerpt Through research and entrepreneurship, Professor Devavrat Shah is helping to design methods that can handle constant decision-making using limited computational resources. ## Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world DevFeed: [Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world]() Original publisher: [Read original article]() Author: laurenmackevich Published: 2026-06-09T19:34:04Z Content type: article Language: en Sources: [Airbnb Engineering & Data Science]() Topics: [data-architecture](), [data](), [data-modeling](), [Data Infrastructure](), [systems]() Tags: [2025](), [airbnb](), [analytics](), [architecture](), [data](), [data-architecture](), [data-engineers](), [data-infrastructure](), [data-modeling](), [data-systems](), [flexible](), [new-product](), [scaling](), [services](), [technical](), [technical-debt](), [uncategorized]() ### AI overview Airbnb's data and analytics engineers evolved its offline data warehouse and data-modeling framework to support Homes, Experiences, and Services. The article examines the trade-offs between separate product-specific models and a unified monolithic model. ### Source excerpt How Airbnb's data engineers and analytics engineers built a consistent and flexible data modeling framework to support the expansion into Homes, Experiences, and Services. ## A Case for Simulation-Driven Resilience in Agentic Data Systems DevFeed: [A Case for Simulation-Driven Resilience in Agentic Data Systems]() Original publisher: [Read original article]() Author: Murat (noreply@blogger.com) Published: 2026-06-08T03:40:45Z Content type: article Language: en Sources: [Metadata]() Topics: [distributed-systems](), [Resilience](), [systems](), [AI Bots](), [data]() Tags: [agentic](), [ai](), [ai-agents](), [capacity](), [cloud](), [data-systems](), [databases](), [distributed-systems](), [metastability](), [queues](), [resilience](), [retries](), [systems](), [work]() ### AI overview This article explains how metastability can arise when retries, queues, timeouts, and load shedding amplify overload instead of containing it. It argues that AI agents, as increasingly important clients of data systems, intensify these feedback loops and makes the case for simulation-driven approaches to building resilient distributed systems. ### Source excerpt As I mentioned in my previous post, I traveled to San Jose at the end of May for the ACM CAIS conference. On Day 0, I gave a very short talk at the Supporting our AI Overlords (SAO) workshop. This post is the promised summary of our paper, "A Case for Simulation-Driven Resilience in Agentic Data Systems", joint work with Aleksey Charapko (University of New Hampshire) and Akshat Vig (MongoDB). Metastability is critical for building the next generation of distributed systems Our story starts with metastability. Metastability is the failure mode where the mechanisms built to protect the system (retries, queues, timeouts, load shedding) turn into amplifiers. Even after the trigger that caused the overload goes away, the system stays behind, churning through busy work, perpetually trying to catch up with the remnants of failed and behind-schedule tasks. It's a bit like missing some foundational math in high school. You spend so long backfilling the old gaps that you never keep up with the new material piling on top, so you stay permanently behind. The catching up work is what keeps you behind, which requires catching up work later, which keeps you behind. (This is presumably why I'm not a rich machine learning scientist today at Deep Mind.) Avoiding and tolerating metastable failures is critical to building the next generation of reliable distributed systems. Aleksey's Metastable Failures in the Wild study (OSDI'22) cataloged these failures from real production incidents. They matter so much because they are the hard faults that remain. We have largely learned to deal with the straightforward ones (crashes, corruptions, dropped packets), which leaves emergent performance failures as the final boss. Metastable failures are responsible for a disproportionate share of critical cloud unavailability incidents, with no single broken part to point at and no obvious way to fix/reset them when they emerge. The cloud economics make the problem worse. Providers and operators have e ## ACM CAIS: Conference on AI and Agentic Systems DevFeed: [ACM CAIS: Conference on AI and Agentic Systems]() Original publisher: [Read original article]() Author: Murat (noreply@blogger.com) Published: 2026-06-02T17:29:07Z Content type: opinion Language: en Sources: [Metadata]() Topics: [Artificial Intelligence](), [Agent Skills](), [AI Development](), [software-architecture](), [Optimization](), [Deployment](), [data](), [cloud-infrastructure]() Tags: [agent-skills](), [agentic](), [ai](), [conference](), [data-systems](), [deployment](), [developer](), [development](), [optimization](), [systems](), [trip-report]() ### AI overview The author reflects on the ACM CAIS conference on AI and agentic systems, describing its research as exploratory and often anecdotal. They argue that the conference would benefit from a narrower focus on data systems and infrastructure for AI, and discuss engagement, short presentation formats, and lessons from an Agent Skills workshop. ### Source excerpt Last week, I traveled to San Jose to attend the ACM CAIS conference. On Day 0, I gave a short talk at the Supporting our AI Overlords (SAO) workshop. And yes, I promise to write a summary of our paper, "A Case for Simulation-Driven Resilience in Agent-First Data Systems" soon! To start with an overall impression of the conference: much of the work presented felt exploratory and anecdotal. Since the compound AI space is still so new, many work seemed to share on-the-ground best practices that worked for them rather than principled results. Some talks really leaned into the "agent, act like a senior engineer and don't make mistakes" vibe. This was especially apparent in the "Agent Skills Workshop". I am not saying this is a bad thing, I learned some valuable lessons from that workshop, which I'll share below. CAIS defines the conference's scope broadly as "research on compound AI architectures, optimization, and deployment". Unfortunately, this broadness seemed to work against the main track. Attendance at the primary conference talks was low, and it often felt like attendees were there solely to present their own work rather than engage with others, likely because the subject matter was spread too thin. In contrast, the workshops were highly focused, which led to much better engagement and active listening from the audience. Moving forward, I think CAIS would benefit greatly from narrowing its focus, maybe specifically focusing more on data systems and infrastructure in support of AI. On that note, what happened to our collective attention span? CAIS limited paper presentations to 7 minutes with just 2 minutes for questions. This year, SIGMOD also shifted to 9-minute talks. Our own paper, "LeaseGuard: Raft Leases Done Right!", got a mere 9 minutes in the spotlight after Jesse traveled all the way to Bangalore to present it. I've even heard that USENIX Security is down to 3-minute talks now. Should we maybe consider slowing down? After all, isn't attention all we need ## How agentic AI could reshape customer experience DevFeed: [How agentic AI could reshape customer experience]() Original publisher: [Read original article]() Author: Jackeline Gonzalez Published: 2026-04-14T00:00:00Z Content type: article Language: en Sources: [Thoughtworks Insights]() Topics: [Agentic AI](), [AI Agent](), [personalization](), [data](), [systems]() Tags: [agentic-ai](), [ai](), [ai-agent](), [article](), [customer-data](), [customer-experience](), [data-systems](), [personalization]() ### AI overview This article examines how agentic AI could improve customer experience by personalizing interactions and orchestrating processes across customer journeys. It emphasizes that reliable results depend on accessible, connected, structured and governed customer data, systems and processes. ### Source excerpt How agentic AI will change customer experience design forever ## Introducing Wiz AI Application Protection Platform (AI-APP) DevFeed: [Introducing Wiz AI Application Protection Platform (AI-APP)]() Original publisher: [Read original article]() Author: Dan Segev Published: 2026-03-23T12:00:01Z Content type: release Language: en Sources: [Wiz]() Topics: [AI Development](), [ai security](), [Security](), [Vulnerabilities](), [Authentication](), [Cloud](), [Model Context Protocol]() Tags: [agent](), [ai](), [ai-applications](), [ai-security](), [authentication](), [cloud](), [code](), [context](), [data-systems](), [graph](), [product](), [product-company-news](), [runtime-protection](), [security](), [sensitive-data](), [tools](), [vulnerability]() ### AI overview Wiz introduces AI-APP, a platform intended to secure AI applications across infrastructure, data, access, models, agents, and applications from code through runtime. It combines visibility, risk analysis, and runtime protection to identify connected attack paths across AI systems. ### Source excerpt Secure every layer of AI applications -- infrastructure, data, access, models, agents, and applications -- from code to runtime, across every environment you build in. ## System Design: Principles for Maintainability, Scalability, and Reliability DevFeed: [System Design: Principles for Maintainability, Scalability, and Reliability]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-03-16T00:00:00Z Content type: tutorial Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [Maintainability](), [reliability-engineering](), [Scalability](), [software-architecture](), [data](), [Software Testing](), [observability](), [CRUD](), [Latency]() Tags: [architecture](), [crud](), [data-systems](), [debug](), [latency](), [maintainability](), [observability](), [reliability](), [scalability](), [system-design]() ### AI overview This article presents production system design around reliability, scalability, and maintainability. It discusses data building blocks beyond CRUD, fault containment, isolation, testing, recovery, observability, latency versus response time, and vertical versus horizontal scaling. ### Source excerpt Reliability, scalability, and maintainability rest on one distinction: a fault is a component-level anomaly, a failure is the system no longer delivering its service. Design to contain faults so they never become failures, and stay precise that latency is waiting time while response time is the full client-visible number. ## Engineering The Semantic Layer: Principles for Data at Scale DevFeed: [Engineering The Semantic Layer: Principles for Data at Scale]() Original publisher: [Read original article]() Author: Sathiya Sarathi Gunasekaran Published: 2025-12-06T11:00:34Z Content type: article Language: en Sources: [Mercari Engineering]() Topics: [semantic-layer](), [metric-standardization](), [data](), [SQL](), [dashboards]() Tags: [2025](), [analytics](), [article](), [dashboards](), [data](), [data-systems](), [engineering](), [semantic-layer](), [source-of-truth](), [sql]() ### AI overview This article explains why inconsistent data definitions create redundant work across analysts, engineers, data scientists, and business users. It presents a semantic layer as infrastructure that standardizes business meaning and provides a single source of truth between raw data and end users. ### Source excerpt This post is for Day 6 of Mercari Advent Calendar 2025, brought to you by sathiya from the Mercari JB Data team. We are in an era of Data Intelligence, where we have efficient analytical datastores and advanced AI tooling, yet we are challenged in answering questions like "What was our revenue yesterday?" or "Why [...] ## Beyond Point to Point DevFeed: [Beyond Point to Point]() Original publisher: [Read original article]() Author: Pedram Navid Published: 2025-05-20T00:00:00Z Content type: article Language: en Sources: [Dagster]() Topics: [DataOps](), [CI/CD](), [Distributed Systems & Parallel Computing]() Tags: [data-engineering](), [data-orchestration](), [data-platform](), [data-quality](), [data-systems](), [developer](), [developer-experience](), [engineering](), [orchestration]() ### AI overview The article argues that modern data engineering teams need purpose-built orchestration rather than point-to-point integration tools. It highlights code-first workflows, observability, asset-centric architecture, version control, testing, lineage, and reusable components as ways to build maintainable data pipelines and handle legacy systems. ### Source excerpt Why Modern Data Teams Need Orchestration, Not Just Integration ## Fivetran Sync Failed? How to Fix Connector Timeouts DevFeed: [Fivetran Sync Failed? How to Fix Connector Timeouts]() Original publisher: [Read original article]() Author: Jeffrey Richman Published: 2025-04-04T16:47:08Z Content type: tutorial Language: en Sources: [Estuary]() Topics: [Streaming](), [streaming data analytics](), [Event-Streaming](), [rate-limiting](), [Snowflake streaming data architecture](), [Apache Kafka]() Tags: [backpressure](), [batch-processing](), [data-engineering](), [data-pipeline](), [data-reliability](), [data-systems](), [fivetran](), [how-to](), [streaming](), [sync](), [timeout]() ### AI overview The article explains common causes of Fivetran connector timeouts, including large batch jobs, source API throttling, and resource overload. It offers steps such as splitting syncs, adjusting timeout settings, and scheduling jobs during off-peak hours, then advocates Estuary's continuous streaming approach as a long-term alternative. ### Source excerpt Learn why Fivetran connectors frequently timeout, how to fix sync issues immediately, and why Estuary's streaming eliminates batch timeout risks. ## Data Inventory DevFeed: [Data Inventory]() Original publisher: [Read original article]() Author: Ritesh Modi Published: 2025-02-19T14:24:14Z Content type: article Language: en Sources: [Wise]() Topics: [Data Management](), [User interface design](), [Database]() Tags: [backend](), [built-in](), [catalog](), [data](), [data-governance](), [data-platform](), [data-systems](), [frontend](), [governance](), [infrastructure](), [integrate](), [inventory](), [iso-27001](), [least-privilege-access](), [metadata](), [network](), [network-segmentation](), [platform](), [scanners](), [scanning](), [security](), [security-controls]() ### AI overview Data Inventory catalogs Wise data assets to show their location and security or privacy controls, supporting an ISO 27001:2022 requirement. Its design must meet controls such as network segmentation and least privilege, integrate with Data Platform technology, and accommodate fragmented systems. The process discovers operational databases from provisioning metadata, registers assets under unique, human-readable identifiers at sensitivity-appropriate granularity, assigns owners using team metadata or producer/consumer rules, extracts relational schemas from MariaDB information schema and PostgreSQL system catalogs, and asks teams to classify assets through a shared UI. That UI is economical across data systems but disrupts developers' workflow; non-relational schema understanding and better-timed classification notifications remain unresolved. The team says it could not use DataHub, OpenMetadata, or BigID because their required security controls were too lax for its operating model. ### Source excerpt The Data Governance team, part of Data Platform, develops and operates platform products through which autonomous teams can govern their data. The team's vision is to democratise data in a secure and compliant manner. Data Inventory was the team's first product. What is Data Inventory? A data inventory is an extensive catalog of the Wise's data assets. It helps us understand where the data is located and what kind of security / privacy controls are in place. This is also a requirement for ISO 27001:2022. What are the constraints? Any system has to work well with our controls and way-of-working. This led to following constraints: It shall be built in a way that satisfies our security controls such as network segmentation, least privilege access, secure secret management, etc. It shall be easy to integrate with other in-house platforms to provide cohesive experience. It shall be built to connect with heterogeneous and fragmented infrastructure. Many data systems are built through off-the-shelf or managed service offering but others are built & operated in-house. The system shall be integrated by default with technology provided by Data Platform. On another hand, it shall be extensible enough to integrate with data systems not maintained by Data Platform. How did we build our inventory? At high level, we split this into 4 main outcomes: 1. Register assets at correct granularity. 2. Identify the correct owner and make them accountable for the life cycle. 3. Extract the schema of the data asset. 4. Classify the asset with correct sensitivity. This is how design looks at high level: To achieve the above design and outcome, we follow the following steps: Step 0: Discover data systems network metadata Scanners need to be aware of the network metadata before doing scanning. For example: - What is the host or connection string? - What is the technology of the data system? - What is the name of the data system? Let us take an example of how we solve it in our operational datab ## Engineering VP spotlight: Ivo Dimitrov DevFeed: [Engineering VP spotlight: Ivo Dimitrov]() Original publisher: [Read original article]() Author: Thomas Sobolik; Ivo Dimitrov Published: 2024-05-21T00:00:00Z Content type: article Language: en Sources: [Datadog]() Topics: [engineering-leadership](), [Software Engineering](), [data-engineering](), [Platform Engineering]() Tags: [analytics-workloads](), [backend](), [company-culture](), [core](), [data-systems](), [employee-spotlight](), [platforms](), [vp-of-engineering]() ### AI overview Ivo Dimitrov discusses his transition from software development to engineering management and his work leading Datadog's Distributed Data Systems organization. He describes the storage, metrics, events, query, and analytics platforms his teams build, as well as the leadership practices and technical challenges involved in scaling them. ### Source excerpt In this interview, Ivo Dimitrov, Distributed Data Systems VP of Engineering, describes the engineering career that led him to Datadog and his committment to helping build out our core backend platforms. ## How to Load Google Search Console Data into Amazon Redshift DevFeed: [How to Load Google Search Console Data into Amazon Redshift]() Original publisher: [Read original article]() Author: Jeffrey Richman Published: 2024-04-01T18:09:21Z Content type: tutorial Language: en Sources: [Estuary]() Topics: [Amazon Redshift](), [Google Search](), [Amazon S3](), [Google Analytics]() Tags: [amazon-redshift](), [amazon-s3](), [cli](), [command-line](), [csv](), [data-engineering](), [data-systems](), [estuary](), [google-search-console](), [gsc](), [redshift](), [s3]() ### AI overview This guide explains how to load Google Search Console data into Amazon Redshift using either Estuary's managed connector or a manual CSV-to-S3 workflow with Redshift's COPY command. It covers setup, authentication, staging, monitoring, common connection and schema issues, and tradeoffs between one-off and repeatable reporting. Combining search metrics with product, CRM, content, or revenue data enables broader analysis in the warehouse. ### Source excerpt Two ways to move GSC data into Amazon Redshift: automated sync with Estuary, or manual CSV export via S3 and the Redshift COPY command. [Next page]()