# data-architecture Published articles for data-architecture. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## A Unified Data Architecture For Sovereign Agentic AI With VMware Tanzu And VMware vSAN DevFeed: [A Unified Data Architecture For Sovereign Agentic AI With VMware Tanzu And VMware vSAN]() Original publisher: [Read original article]() Author: arnab chakraborty Published: 2026-09-03T23:27:50Z Content type: article Language: en Sources: [VMware Blogs]() Topics: [Agentic AI Architecture](), [AI Architecture](), [data-architecture](), [AI Agent](), [cloud-infrastructure](), [Latency](), [real-time](), [Security](), [data-processing]() Tags: [agentic-ai](), [agentic-ai-architecture](), [ai](), [ai-architecture](), [blog](), [cloud](), [data-architecture](), [efficiency](), [infrastructure](), [latency](), [modern-apps](), [real-time](), [security]() ### AI overview The article presents a unified, on-premises data architecture for sovereign agentic AI using VMware Tanzu, VMware Tanzu Greenplum, VMware Cloud Foundation, and VMware vSAN. It argues that placing AI compute close to enterprise data can improve performance and cost while reducing latency, data-transfer fees, and compliance risks. ### Source excerpt By combining VMware Tanzu Greenplum with VMware vSAN, organizations can bring their AI compute directly to their data storage layer for improved cost and latency. The post A Unified Data Architecture For Sovereign Agentic AI With VMware Tanzu And VMware vSAN appeared first on Tanzu. The post A Unified Data Architecture For Sovereign Agentic AI With VMware Tanzu And VMware vSAN appeared first on VMware Blogs. ## Machine vs. machine: The new reality of cybersecurity in ANZ DevFeed: [Machine vs. machine: The new reality of cybersecurity in ANZ]() Original publisher: [Read original article]() Author: Jeremy Pell Published: 2026-08-26T00:00:00Z Content type: article Language: en Sources: [Elastic Blog - Elasticsearch, Kibana, and ELK Stack]() Topics: [Cybersecurity](), [Frontier AI](), [Threat Research](), [threat detection](), [data-architecture](), [AI Platforms/Deployment](), [Security & Privacy]() Tags: [agentic-ai-cybersecurity-security-research](), [australia](), [compliance](), [cybersecurity](), [data-architecture](), [endpoint-security-siem-security](), [frontier-ai](), [policy](), [research](), [threat-detection]() ### AI overview Frontier AI is accelerating cyberattacks across Australia and New Zealand to machine speed, while defensive capabilities, policy, data visibility, and operational security are struggling to keep pace. Survey findings from more than 850 IT and cybersecurity professionals highlight gaps between regulatory intent and real-world protection, as well as the need for searchable, unified data architectures to support reliable AI-enabled defence. ### Source excerpt Frontier AI has accelerated cyber threats to machine speed, leaving many ANZ organisations vulnerable. Our latest research reveals how fragmented data and visibility gaps hinder defence and why a unified platform is essential to battle threats. ## Build a unified AI agent architecture with DynamoDB and Bedrock DevFeed: [Build a unified AI agent architecture with DynamoDB and Bedrock]() Original publisher: [Read original article]() Author: Dhananjay Karanjkar Published: 2026-08-21T18:19:23Z Content type: tutorial Language: en Sources: [AWS Architecture Blog]() Topics: [AI Bots]() Tags: [advanced-300](), [agent](), [agents](), [ai](), [ai-agent](), [amazon-bedrock](), [amazon-dynamodb](), [amazon-opensearch-service](), [ann](), [api](), [architecture](), [aws-lambda](), [data-architecture](), [database](), [embeddings](), [how-to](), [retrieval](), [search](), [streams](), [sync](), [vector]() ### AI overview A tutorial for building an Amazon Bedrock agent that uses one DynamoDB table for structured data and semantic vector search. DynamoDB Streams generates embeddings when content changes, keeping the vector index synchronized. ### Source excerpt With native vector search in Amazon DynamoDB, you can store vector embeddings alongside your operational data in a single table. This post shows how to build a unified AI agent architecture where an Amazon Bedrock agent uses one DynamoDB table for both structured lookups and semantic search, with a DynamoDB Streams pipeline that keeps embeddings in sync. ## Four pillars of agentic AI success DevFeed: [Four pillars of agentic AI success]() Original publisher: [Read original article]() Author: Simon Sear Published: 2026-07-29T15:47:00Z Content type: article Language: en Sources: [Scott Logic]() Topics: [Artificial Intelligence](), [Software Engineering](), [data](), [legacy](), [systems](), [context]() Tags: [accessibility](), [agentic-ai](), [ai](), [ai-accelerated-development](), [ai-agents](), [architecture](), [artificial-intelligence](), [context](), [data](), [data-architecture](), [data-governance](), [data-strategy](), [delivery](), [engineering](), [governance](), [legacy](), [legacy-modernisation](), [software-engineering](), [systems]() ### AI overview This article argues that organizations need strong foundations before scaling agentic AI. It identifies four pillars: modernizing legacy systems, building reliable data foundations and architecture, managing governance and risk, and democratizing access to data. ### Source excerpt Everyone is talking about agentic AI. That's not surprising. The promise is huge: AI agents that can plan, reason, use tools, work across systems and get real work done. In software engineering, that could mean faster delivery and better quality. In operations, it could mean complex processes moving with less manual effort, fewer handovers and better decisions. ## Why Most Single Source of Truth Initiatives Fail (And What Successful Teams Do Differently) DevFeed: [Why Most Single Source of Truth Initiatives Fail (And What Successful Teams Do Differently)]() Original publisher: [Read original article]() Author: Deepika Saini Published: 2026-07-20T10:07:52Z Content type: article Language: en Sources: [Housing.com]() Topics: [data](), [product analytics]() Tags: [analytics](), [dashboards](), [data](), [data-architecture](), [data-engineering](), [data-strategy](), [databricks](), [governance](), [leadership](), [product-analytics](), [sql](), [technical]() ### AI overview This article argues that Single Source of Truth initiatives often fail because teams use different definitions for shared business metrics. It presents governance, business ownership of KPI definitions, and alignment between teams as more important than centralizing tables, pipelines, or dashboards. ### Source excerpt "We have multiple dashboards showing different numbers. Which one is correct?" If you've worked in data long enough, you've probably heard this question more times than you'd like. Sales reports one revenue figure. Finance reports another. Product Analytics has a third. Executives spend more time debating whose dashboard is correct than discussing what action to take. The natural response is often: "Let's build a Single Source of Truth." Sounds simple. Build a few centralized tables. Move everyone onto the same dashboards. Problem solved. Except...it rarely is. After leading an enterprise-wide Single Source of Truth (SSOT) initiative, I learned an important lesson: The hardest part wasn't building pipelines or writing SQL. It was aligning people. Technology was the easy part. Changing how the organization thought about data was the real challenge. The Biggest Myth About Single Source of Truth Many organizations believe an SSOT is simply a technical project. The thinking usually goes like this: Collect Data ↓ Transform Data ↓ Build Gold Tables ↓ Everyone Uses Them Unfortunately, reality looks more like this: Different Teams ↓ Different Definitions ↓ Different Dashboards ↓ Different Decisions ↓ Lost Trust The problem isn't that data lives in different places. The problem is that different teams define the same business metrics differently. Figure 1: Moving from fragmented metric definitions to a trusted Single Source of Truth is as much about standardization and governance as it is about technology. A table cannot solve that. Only governance can. Technology Doesn't Create Trust Imagine a metric as simple as Revenue. Ask five departments what "Revenue" means, and you might receive five different answers. Finance may recognize revenue after invoicing. Sales may count closed deals. Marketing may include projected pipeline. Product Analytics may track subscription purchases. Customer Success may exclude refunds. None of them are necessarily wrong. They're answering differen ## You Don't Need to Build It to Design It DevFeed: [You Don't Need to Build It to Design It]() Original publisher: [Read original article]() Author: Humayun Tanwar Published: 2026-07-06T11:05:34Z Content type: tutorial Language: en Sources: [Mindorks - Medium]() Topics: [systems](), [data-architecture](), [Latency](), [Requirements]() Tags: [business](), [code](), [compliance](), [data-architecture](), [design](), [latency](), [machine-learning](), [patterns](), [requirements](), [software-engineering](), [systems](), [tech-stack](), [technology]() ### AI overview This article explains why solutions engineers can contribute meaningfully to system design without writing the implementation. It connects integration scoping, data ownership, real-time communication choices, and failure handling to core design concepts, then presents five questions about scale, constraints, dependencies, failure tolerance, and success criteria. ### Source excerpt System design is one of those topics that feels like it belongs exclusively to engineers who write the code. The diagrams, the architecture reviews, the database choices , it all looks like backend territory from the outside. You're Already Doing This Solutions engineers sit at an interesting intersection. You're deep enough in the technical conversation to understand how systems connect, and close enough to the customer to understand what they actually need from those systems. That combination gives you a view that's genuinely useful in design conversations, and one that's worth developing deliberately. When you scope an integration and map which system owns which data, that's data architecture. When you recommend a webhook approach over polling because the customer needs real-time updates, that's a latency and reliability trade-off. When you ask a customer how they handle failures before a build starts, that's systems thinking. The formal vocabulary of system design is mostly a structured way of describing what you're already doing in the field. Building that vocabulary, and understanding the patterns underneath it, makes it easier to contribute in technical conversations, ask the right questions earlier, and connect your field experience to the broader design decisions being made around you. The Five Questions That Structure Any Design Good design conversations don't start with a diagram. They start with these. Scale. What volume are you designing for, not today but at the ceiling that changes the architecture? A thousand requests a day is a different system from a thousand per second. That number changes your database choice, whether you need a queue, and how retries need to behave. Constraints. What's already fixed? Existing tech stack, compliance requirements, vendor contracts. Every real system gets built inside constraints. Name them before drawing anything. Dependencies. What does this system talk to, and what talks to it? Systems that look simple in a conv ## Data Engineering Weekly #274 DevFeed: [Data Engineering Weekly #274]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-06-15T05:29:03Z Content type: news Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [Data Quality](), [anthropic](), [data-architecture](), [AI, ML & Data Engineering](), [Claude](), [Messaging]() Tags: [anthropic](), [architecture](), [claude](), [data](), [data-architecture](), [data-engineering](), [data-quality](), [engineering](), [software]() ### AI overview Data Engineering Weekly #274 covers data platform fundamentals, Anthropic's use of Claude for self-service analytics, Airbnb's data architecture and ownership conventions, Uber's data abstraction layer, and semantic search for AI agents. ### Source excerpt The Weekly Data Engineering Newsletter ## Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world DevFeed: [Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world]() Original publisher: [Read original article]() Author: Patrick Lam Published: 2026-06-09T17:01:02Z Content type: article Language: en Sources: [The Airbnb Tech Blog - Medium]() Topics: [data-architecture](), [Data Infrastructure](), [data-modeling](), [data]() Tags: [analytics](), [architecture](), [data](), [data-architecture](), [data-engineering](), [data-infrastructure](), [data-modeling](), [engineering](), [offline](), [post](), [technology]() ### AI overview Airbnb's data and analytics engineering teams evolved a decade-old offline data warehouse to support Homes, Experiences, and Services. The article examines the trade-offs between separate product-specific data models and a unified monolithic model while describing the need for a consistent, flexible, and scalable data foundation. ### Source excerpt How Airbnb's data engineers and analytics engineers built a consistent and flexible data modeling framework to support the expansion into Homes, Experiences, and Services. By: Patrick Lam, Namrata Lamba, Jamie Stober With the May 2025 Summer Release, Airbnb redesigned its app, relaunched Experiences, and debuted Services, pushing us beyond our traditional Homes focus. For the data teams, this meant rapidly evolving a decade-old infrastructure to integrate two brand-new product pillars. Our data engineers and analytics engineers rose to the challenge by building a consistent and flexible framework to serve as a robust and scalable data foundation for the next decade of growth. But getting there wasn't straightforward. This fundamental shift surfaced a critical question for our data organization: How do you evolve your offline data architecture to support new product lines without introducing disorder in vital analytics services? We knew the approach we took would have long-lasting implications. A fragmented strategy risked creating data silos, inconsistent analytics, and a tangled web of technical debt that would likely slow down future innovation. In this post, we'll take you behind the scenes to share key decisions that we made, the framework that emerged, and the lessons that helped reshape our offline data warehouse for the future. Note that we focus specifically on our offline data warehouse (the analytics-oriented data infrastructure owned by our data engineers and analytics engineers) rather than the online data systems that serve the app directly, as the two domains have fundamentally different requirements, constraints, and design philosophies that warrant separate treatment. The core dilemma: separate vs. monolithic The first and most critical question was how to structure offline data for the new, three-product world, with Homes, a refreshed Experiences product, and the new Services offering. This involved a trade-off between two main approaches: Separate ## Agentic AI removes human safeguards from enterprise data feedback loops DevFeed: [Agentic AI removes human safeguards from enterprise data feedback loops]() Original publisher: [Read original article]() Author: flashdba Published: 2026-04-20T22:04:35Z Content type: opinion Language: en Sources: [flashdba]() Topics: [Artificial Intelligence](), [data-architecture](), [data-engineering](), [rate-limiting](), [audit trail]() Tags: [agentic](), [agentic-ai](), [ai](), [audit-trail](), [cloud](), [data-architecture](), [data-engineering](), [databases](), [databases-and-agentic-ai](), [inferencing](), [performance](), [rate-limiting]() ### AI overview This opinion article argues that enterprise data architectures traditionally relied on human involvement as an implicit safeguard. Delays, approvals and handoffs helped absorb errors, create audit points and limit the speed of changes. Agentic AI can close the feedback loop at machine speed, potentially removing those protections along with the human decision point. ### Source excerpt Classic enterprise data architecture had an implicit safeguard built into it. The human in the loop provided error absorption, audit accretion and natural rate-limiting - none of which were ever specified. Agentic AI removes the human. It removes all of those protections simultaneously. ## Two Betting Platforms, One Lesson: The Database Matters DevFeed: [Two Betting Platforms, One Lesson: The Database Matters]() Original publisher: [Read original article]() Author: Becca Weng Published: 2026-02-24T00:00:00Z Content type: article Language: en Sources: [Cockroach Labs]() Topics: [CockroachDB](), [Databases](), [Scalability](), [real-time](), [Resilience](), [Replication](), [Concurrency](), [consistency](), [data-architecture]() Tags: [cockroachdb](), [concurrency](), [consistency](), [data-architecture](), [database](), [distributed](), [real-time](), [replication](), [resilience](), [scalability]() ### AI overview The article describes how betting and gaming platforms face sharp traffic spikes, high transaction volumes, and reliability demands. It presents Newton and Kaizen Gaming as examples of operators using CockroachDB and distributed SQL to support global scale, real-time activity, replication, resilience, and consistency. ### Source excerpt In betting and gaming, growth is rarely gradual. Traffic spikes around major sporting events. Millions of real-time transactions hit your platform at once, and every one of them has to be correct. When the database falters, the consequences aren't abstract. Revenue stops. Players lose trust. Regulators start asking questions. ## How Silverflow simplified its data platform and cut end-to-end latency by 95% with ClickHouse Cloud DevFeed: [How Silverflow simplified its data platform and cut end-to-end latency by 95% with ClickHouse Cloud]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-02-04T13:43:36Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [data](), [dashboards](), [Grafana](), [Amazon S3]() Tags: [analytics](), [clickhouse](), [cloud](), [dashboards](), [data](), [data-architecture](), [events](), [grafana](), [json](), [latency](), [real-time](), [s3](), [sql](), [streaming]() ### AI overview Silverflow describes rebuilding its data platform on ClickHouse Cloud to support near-real-time analytics for payment events. Streaming events into ClickHouse reduced reported end-to-end latency from 20 minutes to 65 seconds and replaced numerous ETL pipelines with JSON ingestion and ClickPipes. ### Source excerpt "With ClickHouse, events are faster to query, and they're also available to be queried much faster, which has enabled us to do more real-time dashboarding and faster alerting." Roberta Gismondi, Software Engineer ## Rethinking the Global Reporting Platform DevFeed: [Rethinking the Global Reporting Platform]() Original publisher: [Read original article]() Author: Alex Seriy Published: 2026-01-28T00:00:00Z Content type: article Language: en Sources: [Cockroach Labs]() Topics: [CockroachDB](), [data-architecture](), [Database](), [dashboards](), [real-time](), [Software]() Tags: [cockroachdb](), [dashboards](), [data-architecture](), [database](), [real-time](), [reporting](), [software]() ### AI overview This article examines why traditional reporting architectures are struggling as reporting becomes more operational, distributed, governed, and increasingly real-time. It presents CockroachDB as a foundation for modern reporting platforms by embedding correctness, availability, and governance in the operational database. ### Source excerpt Data doesn't just serve the business anymore. Data is the business. Every transaction, every decision, every customer experience depends on having timely, trustworthy insight - not weekly, not daily, but continuously. ## How ClickHouse Cloud enabled LaunchDarkly to build and ship features faster DevFeed: [How ClickHouse Cloud enabled LaunchDarkly to build and ship features faster]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-01-22T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Data Infrastructure](), [data-architecture](), [product analytics](), [real-time](), [Low Latency](), [observability](), [Streaming](), [systems]() Tags: [analytics](), [clickhouse](), [data-architecture](), [data-infrastructure](), [development](), [latency](), [low-latency](), [observability](), [real-time](), [scale](), [storage](), [streaming]() ### AI overview LaunchDarkly uses ClickHouse Cloud to analyze raw feature evaluation events in real time for experimentation, product analytics, and observability. The migration replaced parts of its streaming and batch-oriented architecture, reducing data latency to seconds and enabling faster feature development. ### Source excerpt "With ClickHouse, we can serve fresh data, query raw events directly, lean on materialized views when needed, and build new capabilities in weeks instead of months or quarters." Joe Karayusuf, Software Engineer at LaunchDarkly ## "Like night and day": How Auditzy made queries 33x faster by switching from Postgres to ClickHouse DevFeed: ["Like night and day": How Auditzy made queries 33x faster by switching from Postgres to ClickHouse]() Original publisher: [Read original article]() Author: Mayank Joshi, Co-Founder and CTO, Auditzy Published: 2025-10-09T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [data-architecture](), [Latency](), [Back end](), [data](), [real-time](), [datasets](), [Amazon Web Services](), [Azure](), [browser](), [Operating system]() Tags: [architecture](), [aws](), [azure](), [backend](), [browser](), [clickhouse](), [compression](), [data](), [data-architecture](), [datasets](), [geolocation](), [golang](), [india](), [latency](), [metrics](), [performance](), [postgres](), [real-time](), [scalable-architecture](), [switching]() ### AI overview Auditzy migrated from a Postgres-based architecture to ClickHouse after query latency and ingestion problems emerged as data volumes grew. The migration delivered queries that were 33x faster and 10x better compression, supporting scalable, real-time website performance analytics. ### Source excerpt When Mumbai-based startup Auditzy hit Postgres performance limits, they switched to ClickHouse--and saw queries run 33x faster with 10x better compression. ## 18x faster, 15x cheaper: How Datavations rebuilt its pipeline with ClickHouse DevFeed: [18x faster, 15x cheaper: How Datavations rebuilt its pipeline with ClickHouse]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-10-06T12:00:47Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [data-architecture](), [data](), [Data Quality](), [datasets](), [Amazon S3](), [databricks](), [AWS Glue](), [Amazon EC2](), [Amazon Web Services](), [Algorithms](), [synthetic-data]() Tags: [architecture](), [aws](), [aws-glue](), [clickhouse](), [cloud](), [data](), [data-architecture](), [data-quality](), [databricks](), [latency](), [machine-learning](), [optimization](), [performance](), [real-time](), [s3]() ### AI overview Datavations rebuilt its data architecture around ClickHouse after its S3, Databricks, AWS Glue, Athena, and Tableau pipeline became costly and slow at larger scale. The migration aimed to reduce costs, improve performance, and deliver faster, more actionable data-quality insights for its home-improvement analytics platform. ### Source excerpt Learn how Datavations rebuilt its data architecture around ClickHouse, cutting costs, improving performance, and giving clients faster, more actionable insights ## Billions of transactions, two-thirds lower cost: Why ProcessOut switched from Elasticsearch to ClickHouse Cloud DevFeed: [Billions of transactions, two-thirds lower cost: Why ProcessOut switched from Elasticsearch to ClickHouse Cloud]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-09-30T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [elasticsearch](), [Cloud](), [data-architecture](), [real-time](), [Transactions](), [Benchmark]() Tags: [analytics](), [clickhouse](), [cloud](), [cost](), [data-architecture](), [elasticsearch](), [performance](), [real-time](), [transactions]() ### AI overview ProcessOut replaced its costly, difficult-to-change Elasticsearch-based analytics architecture with ClickHouse Cloud. The migration reduced costs by two-thirds and reduced latency from minutes to approximately two seconds. ### Source excerpt ProcessOut replaced a costly, slow, hard-to-change Elasticsearch setup with ClickHouse Cloud, lowering costs by two-thirds and reducing latency from minutes to ~2 seconds. ## Langfuse and ClickHouse: A new data stack for modern LLM applications DevFeed: [Langfuse and ClickHouse: A new data stack for modern LLM applications]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-06-23T14:42:08Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [observability](), [Large Language Model](), [Scalability](), [Self-hosted](), [data-architecture](), [Docker Compose]() Tags: [clickhouse](), [data-architecture](), [docker](), [infrastructure](), [llm](), [observability](), [open-source](), [scalability](), [self-hosted]() ### AI overview Langfuse rebuilt its LLM observability platform around ClickHouse after its Postgres-based architecture struggled with billions of rows, complex queries, and rapidly growing ingestion volumes. The article describes the migration, its scalability benefits, and the rollout to more than 1,000 self-hosted users without downtime. ### Source excerpt Langfuse rebuilt its observability platform for LLM applications with ClickHouse at the core, boosting performance, unlocking scalability, and rolling the change out to 1,000+ self-hosted users with zero downtime. ## How Kami scaled from 80M to nearly 2B events per week with ClickHouse DevFeed: [How Kami scaled from 80M to nearly 2B events per week with ClickHouse]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-05-16T16:16:20Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [data-architecture](), [Databases](), [Scalability](), [Back end](), [Rails](), [SQL]() Tags: [analytics](), [architecture](), [backend](), [clickhouse](), [covid-19](), [data](), [data-architecture](), [migration](), [new-zealand](), [performance](), [postgres](), [remote](), [scale](), [sql]() ### AI overview Kami's event-tracking system grew from about 80 million to nearly 2 billion weekly events during the COVID-19 pandemic, pushing its Postgres-based architecture beyond practical limits. The article describes Kami's migration to ClickHouse and the resulting improvements in scalability and query performance. ### Source excerpt "After testing multiple options, we decided ClickHouse was the best solution for our needs and really the only one that could handle our scale." - Jordan Thoms, co-founder and CTO ## Why Flock Safety turned to ClickHouse for real-time vehicle traffic analytics DevFeed: [Why Flock Safety turned to ClickHouse for real-time vehicle traffic analytics]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-04-29T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [real-time](), [data-engineering](), [Amazon Redshift](), [Architecture & Design](), [data]() Tags: [amazon-redshift](), [analytics](), [analytics-pipeline](), [clickhouse](), [dashboards](), [data-architecture](), [data-engineering](), [real-time]() ### AI overview This developer case study explains why Flock Safety moved its vehicle traffic analytics platform from an Amazon Redshift, DBT, Prefect, and QuickSight SPICE pipeline to ClickHouse Cloud. The former architecture refreshed data daily, required up to four hours to process some datasets, and made data unavailable during refreshes. ClickHouse enabled real-time analytics for camera traffic without the previous size constraints or row-level security limitations. ### Source excerpt "With ClickHouse, our customers now have real-time analytics for their camera traffic, and there are no more constraints on size or row-level security." ~ Leon Kozlowski, Data Engineering Manager ## Multi-tenancy and Database-per-User Design in Postgres DevFeed: [Multi-tenancy and Database-per-User Design in Postgres]() Original publisher: [Read original article]() Author: Dian M Fay Published: 2024-08-29T18:00:25Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [Per-user Database](), [Multi-tenancy](), [data-architecture](), [Database](), [Cloud](), [Serverless](), [Architecture & Design]() Tags: [architectures](), [cloud](), [data-architecture](), [database](), [development](), [multi-tenancy](), [postgres](), [product](), [schema](), [serverless]() ### AI overview This article introduces a series about building database-per-user applications with Postgres and Neon. It outlines the evolution of multi-tenant architectures--shared schema, schema-per-user, and database-per-user--and explains the tradeoff between stronger data isolation and greater operational complexity. It also describes how Neon's serverless design and hierarchy of projects, branches, and databases may affect data architecture and application design. ### Source excerpt Many Neon users aren't just storing their own information, but host data on behalf of many clients or customers, commonly called tenants. Over the decades, approaches to these multi-tenant data architectures have evolved in three main directions: shared schema, schema-per-user, a... ## Data Quality at Udemy -- Part 1 DevFeed: [Data Quality at Udemy -- Part 1]() Original publisher: [Read original article]() Author: Murat Migdisoglu Published: 2023-09-06T22:01:16Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Data Quality](), [data-architecture](), [data lake](), [Apache Spark](), [airflow](), [Streaming]() Tags: [airflow](), [architecture](), [data-architecture](), [data-catalog](), [data-governance](), [data-lake](), [data-lineage](), [data-quality](), [data-quality-management](), [pipelines](), [principal-engineer](), [quality](), [spark](), [streaming]() ### AI overview This article describes Udemy's efforts to improve data quality by establishing an end-to-end data lineage solution. It explains how distributed data ownership and self-service analytics make lineage important for impact analysis, change management, and identifying unused columns or orphan tables. ### Source excerpt Data Quality at Udemy -- Part 1Data Lineage Demystified- Why it Matters and How to Leverage its Magic for Informed Business Success! In late 2020, upon joining Udemy as a principal engineer for the data platform team, my focus shifted toward enhancing data quality within the organization. My journey began with conducting a comprehensive poll across the data organization, aimed at identifying the key pain points of data users. The results of the poll were eye-opening, revealing that 78% of users considered the absence of data provenance/lineage as a data quality issue. Furthermore, it was obvious that for a vast majority of the users, the inability to track data lineage was an important problem in impact analysis and detecting unused columns or orphan tables in the system. Inspired by these insights, I took the initiative to propose and launch two transformative projects. The first one, which is the subject of this article, is an ambitious initiative to establish a comprehensive end-to-end data lineage solution that will revolutionize our data ecosystem. The second project centers around data monitoring, which will be explored in another post. Udemy's sophisticated data architecture revolves around a data lake fed by diverse pipelines: system logs, streaming data from services, CDC listeners for replicated service databases, and more. The backbone of data transformations lies in Hive and Spark, while Airflow takes charge of orchestrating thousands of these pipelines. Unraveling Data Flow Complexity: Data Lineage in Growing Data Driven Organizations In the early stages of an organization's data-driven journey, data lineage may not be deemed crucial. With just a few pipelines managed by a centralized team, the dependency tree of the workflow orchestration typically suffices to comprehend the relationships between data entities. However, as the business scales up, relying on a single centralized team for all data flows becomes a bottleneck. Consequently, data organizatio ## The Complex Data Models Behind Shopify's Tax Insights Feature DevFeed: [The Complex Data Models Behind Shopify's Tax Insights Feature]() Original publisher: [Read original article]() Author: Siraj Ali Published: 2023-02-08T15:00:04Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [Shopify](), [data-architecture](), [data-modeling](), [Apache Spark](), [Google Cloud Platform (GCP)](), [Databases](), [Back end](), [SQL](), [Cloud]() Tags: [backend](), [cloud-storage](), [data](), [data-architecture](), [data-modeling](), [databases](), [google-cloud](), [shopify](), [spark](), [sql]() ### AI overview This article explains the data architecture behind Shopify Tax's Tax Insights feature. It covers requirements gathering, SQL-based data-model prototyping, dynamically changing data, Spark processing jobs, data warehousing in Google Cloud Storage, and publishing insights for display in the merchant application. ### Source excerpt The intensive data work behind Shopify's Tax Insights feature required building functionality to handle dynamically changing information ## How we processed 12 trillion rows during Black Friday DevFeed: [How we processed 12 trillion rows during Black Friday]() Original publisher: [Read original article]() Author: Javi Santana Published: 2020-12-21T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [clickhouse](), [data-architecture](), [real-time]() Tags: [analytics](), [architecture](), [clickhouse](), [data-architecture](), [engineering-excellence](), [infrastructure](), [real-time](), [scale]() ### AI overview This post explains the data architecture and infrastructure used to scale a real-time analytics service with ClickHouse during Black Friday, processing 12 trillion rows. ### Source excerpt In this post we explain the data architecture, infrastructure and how we scale our real-time analytics service with ClickHouse®. ## Presto at Zuora DevFeed: [Presto at Zuora]() Original publisher: [Read original article]() Author: Manfred Moser Published: 2020-06-16T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Architecture & Design](), [Microservice](), [Microservices](), [data-architecture](), [Query (disambiguation)](), [big-data]() Tags: [analytics](), [architecture](), [big-data](), [data-architecture](), [distributed](), [microservices-architecture](), [presentation](), [technical](), [update](), [webinar]() ### AI overview The article announces a Presto Summit virtual event about using Presto as a query layer in Zuora's distributed microservices architecture. It explains how Zuora used Presto to separate a monolithic data architecture, provide access to production data across services, and support complex queries over live data. ### Source excerpt The Presto Summit is morphing into a series of virtual events, and we already started with the State of Presto webinar recently. Next up is a talk about Presto with lots of practical insights at Zuora presented by Henning Schmiedehausen: Using Presto as Query Layer in a Distributed Microservices Architecture Update: We had a great event with lots of questions from the audience, taking us beyond the planned time frame. Check out the recording to learn more: