# data lake A data lake is a storage repository that stores large volumes of raw data in its original formats and supports data management, analytics, and processing at scale. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33% DevFeed: [How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%]() Original publisher: [Read original article]() Author: Nick Del Nano, Data Streaming Published: 2026-05-21T00:00:00Z Content type: article Language: en Sources: [Yelp]() Topics: [data](), [data lake](), [Apache Iceberg](), [Amazon S3](), [AWS IAM]() Tags: [analytics](), [apache-iceberg](), [aws](), [data](), [data-lake](), [iam](), [partition](), [s3]() ### AI overview Yelp describes visualizations that map partition keys against access-event timestamps to reveal daily batch jobs, backfills, and ad hoc queries. The resulting usage attribution supported Apache Iceberg migration and storage-efficiency work that reduced the cost of its petabyte-scale data lake by 33%. ### Source excerpt Introduction In large analytics environments, data teams often struggle to answer deceptively simple questions, like who their stakeholders are and how their data is being used. At Yelp, we address this by visualizing access patterns, plotting time-based partition key values against access event timestamps. These visualizations reveal distinct usage signatures - ad hoc queries, daily batch jobs, and periodic backfills - allowing data owners to understand their stakeholders and use cases. This deeper insight into data usage has enabled high-impact platform initiatives including migrating thousands of tables to Apache Iceberg format and identifying storage efficiencies which reduced the cost of... ## ClickHouse Announces Integration with Microsoft OneLake for Federated Analytics DevFeed: [ClickHouse Announces Integration with Microsoft OneLake for Federated Analytics]() Original publisher: [Read original article]() Author: Melvyn Peignon Published: 2025-11-18T00:00:00Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [interoperability](), [Microsoft](), [data lake](), [real-time](), [Apache Iceberg]() Tags: [apache-iceberg](), [clickhouse](), [data-lake](), [databases](), [fabric](), [integration](), [interoperability](), [latency](), [microsoft](), [real-time]() ### AI overview ClickHouse announced an integration with Microsoft OneLake, the unified data lake service within Microsoft Fabric. The integration exposes OneLake Iceberg tables to ClickHouse so users can query and analyze data across ClickHouse and Fabric, supporting real-time analytical workloads. ### Source excerpt ClickHouse today announced the availability of a powerful new integration with Microsoft OneLake, the unified data lake service within Microsoft Fabric. ## Project Teleport: Cost-Effective and Scalable Kafka Data Processing at Block DevFeed: [Project Teleport: Cost-Effective and Scalable Kafka Data Processing at Block]() Original publisher: [Read original article]() Author: Unni Krishnan Published: 2025-03-20T00:00:00Z Content type: article Language: en Sources: [Cash App Code Blog]() Topics: [data-processing](), [Kafka](), [migration](), [databricks](), [data lake](), [Apache Spark](), [Amazon S3](), [parquet](), [Amazon Redshift]() Tags: [acquisition](), [amazon](), [cloud](), [data](), [data-lake](), [data-processing](), [databricks](), [delta-lake](), [emr](), [kafka](), [parquet](), [pipelines](), [redshift](), [s3](), [scale](), [storage]() ### AI overview Project Teleport is Block's cross-region Kafka data-processing system for integrating Afterpay's Sydney-hosted data lake into Block's US-based ecosystem. Built with Delta Lake, Spark on Databricks, and object storage, it supports migration of legacy pipelines and reduced cloud egress costs by USD 540,000 per year. ### Source excerpt Teleport achieves efficient and reliable cross-region Kafka data processing at scale. Using this approach, Afterpay data team reduced cloud egress costs by USD 540,000 per year. ## Data Quality at Udemy -- Part 1 DevFeed: [Data Quality at Udemy -- Part 1]() Original publisher: [Read original article]() Author: Murat Migdisoglu Published: 2023-09-06T22:01:16Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Data Quality](), [data-architecture](), [data lake](), [Apache Spark](), [airflow](), [Streaming]() Tags: [airflow](), [architecture](), [data-architecture](), [data-catalog](), [data-governance](), [data-lake](), [data-lineage](), [data-quality](), [data-quality-management](), [pipelines](), [principal-engineer](), [quality](), [spark](), [streaming]() ### AI overview This article describes Udemy's efforts to improve data quality by establishing an end-to-end data lineage solution. It explains how distributed data ownership and self-service analytics make lineage important for impact analysis, change management, and identifying unused columns or orphan tables. ### Source excerpt Data Quality at Udemy -- Part 1Data Lineage Demystified- Why it Matters and How to Leverage its Magic for Informed Business Success! In late 2020, upon joining Udemy as a principal engineer for the data platform team, my focus shifted toward enhancing data quality within the organization. My journey began with conducting a comprehensive poll across the data organization, aimed at identifying the key pain points of data users. The results of the poll were eye-opening, revealing that 78% of users considered the absence of data provenance/lineage as a data quality issue. Furthermore, it was obvious that for a vast majority of the users, the inability to track data lineage was an important problem in impact analysis and detecting unused columns or orphan tables in the system. Inspired by these insights, I took the initiative to propose and launch two transformative projects. The first one, which is the subject of this article, is an ambitious initiative to establish a comprehensive end-to-end data lineage solution that will revolutionize our data ecosystem. The second project centers around data monitoring, which will be explored in another post. Udemy's sophisticated data architecture revolves around a data lake fed by diverse pipelines: system logs, streaming data from services, CDC listeners for replicated service databases, and more. The backbone of data transformations lies in Hive and Spark, while Airflow takes charge of orchestrating thousands of these pipelines. Unraveling Data Flow Complexity: Data Lineage in Growing Data Driven Organizations In the early stages of an organization's data-driven journey, data lineage may not be deemed crucial. With just a few pipelines managed by a centralized team, the dependency tree of the workflow orchestration typically suffices to comprehend the relationships between data entities. However, as the business scales up, relying on a single centralized team for all data flows becomes a bottleneck. Consequently, data organizatio ## The unnecessary hype strategy behind Microsoft Fabric DevFeed: [The unnecessary hype strategy behind Microsoft Fabric]() Original publisher: [Read original article]() Published: 2023-05-29T00:00:00Z Content type: opinion Language: en Sources: [Mutt0-ds Notes]() Topics: [Microsoft](), [data-engineering](), [Azure](), [Cloud](), [data lake](), [data-processing](), [etl](), [AI, ML & Data Engineering]() Tags: [azure](), [cloud](), [copilot](), [data](), [data-engineering](), [data-lake](), [data-processing](), [etl](), [microsoft](), [microsoft-azure](), [technical]() ### AI overview An opinion article examines Microsoft Fabric, a unified data platform announced at Microsoft Build. It describes Fabric's integration of data storage, processing, ETL, analytics, and business intelligence tools, while criticizing Microsoft's secrecy and hype-oriented launch strategy and noting that the platform was still in beta. ### Source excerpt If you are into Data Engineering in Microsoft Azure Cloud Environment, you probaly heard about Microsoft Fabric being announced last week... ## Trino Summit 2022: Sessions, speakers, and event details DevFeed: [Trino Summit 2022: Sessions, speakers, and event details]() Original publisher: [Read original article]() Author: Brian Olsen, Dain Sundstrom Published: 2022-09-22T00:00:00Z Content type: news Language: en Sources: [Trino Blog]() Topics: [data lake](), [SQL](), [Open Source](), [autoscaling]() Tags: [architectures](), [autoscaling](), [conference](), [data-lake](), [event](), [lyft](), [mpp](), [routing](), [summit]() ### AI overview This article announces Trino Summit 2022, a free hybrid conference taking place on November 10th, and previews selected sessions and speakers. Topics include Trino's open source project, query federation, large-scale ETL at Lyft, autoscaling, and fault-tolerant execution. ### Source excerpt Commander Bun Bun is back and this year we have an exciting lineup of speakers. Topics range from architectures like data mesh and data lakehouse, to running Trino at scale with fault-tolerant execution, and query federation. This conference is free and takes place on November 10th. The summit is a hybrid event for in-person and virtual attendance. Find out more details below! ## Integrating Confluent Schema Registry with Apache Spark applications DevFeed: [Integrating Confluent Schema Registry with Apache Spark applications]() Original publisher: [Read original article]() Author: Dominik Liebler Published: 2022-01-24T08:04:19Z Content type: tutorial Language: en Sources: [YAZIO Engineering - Medium]() Topics: [Kafka](), [Apache Spark](), [Streaming](), [data lake](), [parquet](), [Kotlin](), [Kubernetes](), [ceph](), [JSON Schema]() Tags: [apache-spark](), [backpressure](), [ceph](), [confluent](), [data-engineering](), [data-lake](), [data-pipeline](), [json](), [kafka](), [kotlin](), [kubernetes](), [parquet](), [payload](), [pipelines](), [schema](), [schemaregistry](), [serialization](), [spark](), [streaming]() ### AI overview This engineering article explains YAZIO's data pipeline from mobile and web applications through Kafka and Spark Structured Streaming into a Ceph-based data lake. It discusses why schemas matter and describes replacing JSON with Apache Avro and Confluent Schema Registry to reduce message size while keeping schema information externally stored and cached. ### Source excerpt At YAZIO, we believe in making decisions backed by data to help people live healthier lives through better nutrition. For each new and existing feature we want to evaluate how well it performs and how our users interact with it. In order to do so, we need a lot of data and we need to handle backpressure in our systems. To cope with that we use a Kafka cluster managed by Strimzi operators running in Kubernetes. The data itself is being ingested from our mobile and web apps via HTTP or TCP endpoints serialized into JSON and stored in Kafka by a small application written in Kotlin/JVM. Overview of our data pipeline architecture At the other end of the pipeline, different Spark Structured Streaming applications (also written in Kotlin) dump this information into our data lake residing in a Ceph bucket. They read data from Kafka, deserialize it, transform some of the fields and write Parquet files into the data lake using a new schema. Why schemas? Schemas play an important role in data pipelines because they give meaning and context to data. In a world without schemas we would still do random interpretations about the context and meaning of data every now and then when using it. As you might have guessed already this would lead to a lot of bugs and misunderstandings. Photo by EJ Strat https://unsplash.com/photos/VjWi56AWQ9k Similar to a legal contract that binds you to certain limits, a schema binds the data to certain limits and meaning which narrow down the need of interpretation. Choice of serialization formats At the time of writing, Confluent Schema Registry supports these three serialization formats: Apache Avro Protocol Buffers (protobuf) JSON Schema From those choices, only two really provide more than just validation of the data that is ingested and transmitted through our data pipelines. Avro and Protobuf also allow us to shrink the sizes of our topics because only the payload is contained in a message, while the repeating schema will not be stored. In the cas ## Building a Data Lake with Spark and Iceberg at Home to over-complicate shopping for a House DevFeed: [Building a Data Lake with Spark and Iceberg at Home to over-complicate shopping for a House]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2021-12-03T00:00:00Z Content type: tutorial Language: en Sources: [Christian Hollinger]() Topics: [data lake](), [Apache Iceberg](), [Apache Spark](), [data-processing](), [Python](), [geospatial]() Tags: [analytics](), [big-data](), [data](), [data-lake](), [data-processing](), [geopandas](), [geospatial](), [geospatial-data](), [hadoop](), [hive](), [iceberg](), [presto](), [python](), [scala](), [spark](), [sql](), [trino]() ### AI overview A tutorial about building a local data lake with Spark, Iceberg, and Python to query demographic and geographic data while searching for a new neighborhood. It discusses data lakes, geospatial inconsistencies, and local systems design. ### Source excerpt How I build what is essentially a self-service Data Lake at home to narrow down the search area for a new house, instead of using Zillow like a normal person, using Spark, Iceberg, and Python. ## Pinion -- The Load Framework Part-2 DevFeed: [Pinion -- The Load Framework Part-2]() Original publisher: [Read original article]() Author: Saurabh Jain Published: 2021-10-29T16:50:24Z Content type: article Language: en Sources: [Groupon Engineering -- Medium]() Topics: [data lake](), [Framework](), [Data Quality](), [Transactions](), [Streaming](), [Open Source]() Tags: [acid](), [audit](), [aws](), [cloud](), [data-validation](), [delta-lake](), [deltalake](), [hdfs](), [logging](), [optimization](), [reporting](), [s3](), [schema](), [science](), [spark](), [streaming](), [transactions]() ### AI overview This second post in the Pinion -- The Load Framework series explains how Pinion extends Apache Delta Lake APIs for slowly changing dimension operations. It covers data validation, compaction, auditing, streamlined logging, and chained APIs, and introduces Delta Lake capabilities such as ACID transactions, schema enforcement, batch and streaming interfaces, and time travel. ### Source excerpt Pinion -- The Load Framework Part-2 This post is the 2nd part of the "Pinion -- The Load Framework" series. In case you have not read the 1st post, you can read it here. In this post, we are going to cover the following topics. How does Pinion use Delta Lake for SCD operations? Small file problem with Delta Lake and its resolution. Before we dive into the topics of this post, let's look at the definition of DeltaLake to set the context right. Apache Delta Lake - Apache Delta Lake is an open-source framework that enables the addition of ACID transactions support to a new data lake or an existing data lake created on top of S3, GCS, and HDFS. In addition to this, it provides other features such as scalable metadata handling, unified interface for both batch and streaming application, schema enforcement, time travel, and a rich interface of APIs to enable complex use cases like change-data-capture (CDC) and slowly-changing-dimension (SCD) operations. To keep the post concise and to the point, I won't go into much detail here about Delta Lake, since there is already great documentation available about it, that you can read it here. How does Pinion use Delta Lake for SCD operations? - Apache Delta Lake provides a rich set of APIs to handle slowly-changing dimensions, however, those APIs were not enough alone to build the features that we want to have in The Pinion Framework. So, we decided to enrich the APIs provided by Delta Lake by adding the following features to it: Data Validation Compaction Audit Streamlined logging infrastructure to make the data engineer's life easier during debugging of a failed job Chained APIs Let's dive a little further into the features that we had listed above. Data Validation -- By default, schema enforcement is enabled in Pinion for all the APIs where we have a need of inserting rows from source data(LRFs) into the target table. In case of a schema mismatch, Pinion raises an error and stops processing of further stages. It ensures the data a ## How we build the Image Gallery on trivago DevFeed: [How we build the Image Gallery on trivago]() Original publisher: [Read original article]() Author: Praneeth Peiris I want Published: 2021-07-07T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [Google Cloud Platform (GCP)](), [etl](), [data lake](), [Kafka](), [Front end](), [Amazon Web Services]() Tags: [amazon-web-services-aws](), [architecture](), [backend](), [cloud](), [data-lake](), [engineering](), [etl](), [frontend](), [gcp](), [kafka](), [platforms]() ### AI overview trivago describes migrating its hotel image-gallery ETL pipeline from Amazon Web Services to Google Cloud Platform. The redesigned architecture uses Dataflow jobs to snapshot images and tags, fetch changed accommodations, build and validate sorted galleries, and stream approved changes to frontend teams through Kafka. ### Source excerpt When was the last time you booked accommodation without checking its photos? Most probably never! Because having imagery information makes our decision-making process much easier and faster. How... ## Prophecy: Teamwork's Data Lake DevFeed: [Prophecy: Teamwork's Data Lake]() Original publisher: [Read original article]() Author: Joe Minichino Published: 2020-07-17T11:41:10Z Content type: opinion Language: en Sources: [Teamwork]() Topics: [data lake](), [data-engineering](), [data](), [decision-making](), [big-data](), [Databases](), [Google Analytics](), [stripe]() Tags: [analytics](), [aws](), [big-data](), [data](), [data-engineering](), [data-lake](), [databases](), [decision-making](), [google-analytics](), [stripe]() ### AI overview Teamwork describes reevaluating its analytics approach because data was distributed across product databases and third-party services, making it difficult to connect leads, product usage, and revenue. The article argues that a data lake can make analytics and data-informed decision-making more practical, while clarifying that data lakes and big data are not limited to enterprise-scale environments. ### Source excerpt You need a Data Lake. The Context Teamwork has been around for more than 10 years. Starting out as a project management and work collaboration platform and later expanding into other areas, such as help-desk, chat, document management and CRM software. As the company has grown and evolved, data has grown, changed, expanded, diversified, fragmented, then changed again. Analytics in this landscape are not for the faint of heart. The Problem The issue at the base of the decision to start re-evaluating our analytics approach, at Teamwork, was because we have way too many data sources to make sense of all the data we collect. Much of the data ends up being "dark", underutilized, unleveraged. We have multiple databases shards, for each product, along with platform databases containing global customer information; and then we have 3rd parties such as Stripe, Marketo, Chart Mogul and Google Analytics to name the most important ones. We realized that making a connection between sales leads in Marketo, their usage of our products (GA and our DBs) and their revenue (our DBs, Stripe, Chart Mogul) was impossible. Well, it was possible, but incredibly painful and poorly automated, making it slow and error prone. And by addressing this problem we are getting a large number of benefits back, practically for free. You need Analytics Let's cut to the chase: you need analytics. No matter how big or small your business or enterprise is, you need analytics to take more informed business decisions. I do not doubt there are individuals with great gut-driven decision-making skills, but by and large, it's better if you look at data to make decisions. That's why you need a Data Lake. It could be a really small lake, it could be a pond or a puddle. But you need it for better decision making. Data Lakes, Big Data and other buzzwords Let's clear the air on a couple of misconceptions. The terms "Big Data" and "Data Lake" absolutely scream of corporate, of enterprise, of governance, compliance, r ## Data Lakes: Some thoughts on Hadoop, Hive, HBase, and Spark DevFeed: [Data Lakes: Some thoughts on Hadoop, Hive, HBase, and Spark]() Original publisher: [Read original article]() Author: Christian Hollinger Published: 2017-11-04T00:00:00Z Content type: opinion Language: en Sources: [Christian Hollinger]() Topics: [data lake](), [Hadoop](), [Apache Spark](), [data](), [etl](), [BigQuery]() Tags: [big-data](), [bigquery](), [cloud](), [data](), [data-lake](), [etl](), [hadoop](), [hbase](), [hive](), [phoenix](), [scala](), [spark]() ### AI overview An overview of data lakes and the organizational and technical questions involved in building and using them. It discusses data storage, metadata, connected business systems, data models, ETL requirements, regulations, and tools including Hadoop, BigQuery, Hive, HBase, and Spark. ### Source excerpt This article will talk about how organizations can make use of the wonderful thing that is commonly referred to as "Data Lake" - what constitutes a Data Lake, how probably should (and shouldn't) use it to gather insights and why evaluating technologies is just as important as understanding your data... ## A Journey Towards a Custom Data Warehouse Solution Part 2: We Need Storage DevFeed: [A Journey Towards a Custom Data Warehouse Solution Part 2: We Need Storage]() Original publisher: [Read original article]() Author: Robert Bordo (robert@upday.com) Published: 2017-08-22T04:39:55Z Content type: article Language: en Sources: [Upday]() Topics: [data](), [data lake](), [Amazon S3](), [Amazon Redshift](), [Amazon Web Services](), [Scalability]() Tags: [aws](), [business-intelligence](), [data](), [data-lake](), [s3](), [scalability](), [storage](), [warehouse]() ### AI overview This article examines storage choices for a custom data warehouse. It describes application log data, the limitations of time-series databases for additional master and historical data, business intelligence access needs, and the use of Amazon S3 as a data lake while considering other storage options including AWS Redshift. ### Source excerpt In the beginning we created a cluster. And the cluster was without form, and void; and nulls were upon the face of the storage. As we learned in part 1 of our series, a data warehouse consists of several components. The key component is the storage. All the others group around it. But how can one draw a decision on which storage solution to adopt? What is out there anyway? Preface In a perfect world there would be only one kind of storage that fits all the needs of current DWH development and analysis. But since we are not living in that kind of place, we have several options. And the number of options increase the deeper one dives into the topic. There seem to be solutions for every use case you can think of. That might be a good starting point. What is our most common use case? What are we going to store? And how would we like to access our data in the end? Our major source is a massive amount of log data coming from our app. Everything the user does (e.g swiping through articles, selecting categories, leaving the app) is tracked, enriched with metadata (e.g. the user's location, app version, article identifier) and stored by a third-party service in big, semi-structured log files. Having only this source, a time series database like Graphite or InfluxDB could do the job. But also having slow changing master data, like user profiles, article metadata and maybe even to keep a history of data, this solution would not satisfy our current and future needs. Another thing that comes to my mind is how the data will be accessed by our final consumer (namely: Business Intelligence). Usually they use tools like Jasper Reports or Tableau for generating reports. For analyses we have to pre-aggregate the data to make queries more performant and translate raw information into a digestible format. What else is on the market? Storage good at bad at Example S3/Flat Files scalability, easy to use, data lake querying S3 Time Series DB handling time series data non time series data G