# data lake Published articles for data lake. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33% DevFeed: [How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%]() Original publisher: [Read original article]() Author: Nick Del Nano, Data Streaming Published: 2026-05-21T00:00:00Z Content type: article Language: en Sources: [Yelp]() Topics: [data](), [data lake](), [Apache Iceberg](), [Amazon S3](), [AWS IAM]() Tags: [analytics](), [apache-iceberg](), [aws](), [data](), [data-lake](), [iam](), [partition](), [s3]() ### AI overview Yelp describes visualizations that map partition keys against access-event timestamps to reveal daily batch jobs, backfills, and ad hoc queries. The resulting usage attribution supported Apache Iceberg migration and storage-efficiency work that reduced the cost of its petabyte-scale data lake by 33%. ### Source excerpt Introduction In large analytics environments, data teams often struggle to answer deceptively simple questions, like who their stakeholders are and how their data is being used. At Yelp, we address this by visualizing access patterns, plotting time-based partition key values against access event timestamps. These visualizations reveal distinct usage signatures - ad hoc queries, daily batch jobs, and periodic backfills - allowing data owners to understand their stakeholders and use cases. This deeper insight into data usage has enabled high-impact platform initiatives including migrating thousands of tables to Apache Iceberg format and identifying storage efficiencies which reduced the cost of... ## Using Materialized Views and Derived Datasets to Optimize Data Queries DevFeed: [Using Materialized Views and Derived Datasets to Optimize Data Queries]() Original publisher: [Read original article]() Published: 2025-12-07T09:05:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [data-architecture](), [Data Management](), [Database](), [datasets](), [Latency](), [elasticsearch](), [parquet]() Tags: [data](), [data-lake](), [data-management](), [database](), [datasets](), [elasticsearch](), [latency](), [parquet](), [performance]() ### AI overview The article explains how pull-based queries retrieve matching records at query time and why this can create performance, data-format, data-shape, and data-location challenges. It presents materialized views and derived datasets as a way to precompute query results and store them in an optimized format, shape, and location. ### Source excerpt Table of Contents Materialized Views Embracing Data Duplication Streams for machines, tables for humans Historically, data management systems have been built around the notion of pull queries: users query data which, for instance, is stored in tables in an RDBMS, Parquet files in a data lake, or a full-text index in Elasticsearch. When a user issues a query, the engine will produce the result set at that point in time by churning through the data set and finding all matching records (oftentimes sped up by utilizing indexes). ## ClickHouse Announces Integration with Microsoft OneLake for Federated Analytics DevFeed: [ClickHouse Announces Integration with Microsoft OneLake for Federated Analytics]() Original publisher: [Read original article]() Author: Melvyn Peignon Published: 2025-11-18T00:00:00Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [interoperability](), [Microsoft](), [data lake](), [real-time](), [Apache Iceberg]() Tags: [apache-iceberg](), [clickhouse](), [data-lake](), [databases](), [fabric](), [integration](), [interoperability](), [latency](), [microsoft](), [real-time]() ### AI overview ClickHouse announced an integration with Microsoft OneLake, the unified data lake service within Microsoft Fabric. The integration exposes OneLake Iceberg tables to ClickHouse so users can query and analyze data across ClickHouse and Fabric, supporting real-time analytical workloads. ### Source excerpt ClickHouse today announced the availability of a powerful new integration with Microsoft OneLake, the unified data lake service within Microsoft Fabric. ## Building a Scalable AWS Backup System for S3 Data Warehouses DevFeed: [Building a Scalable AWS Backup System for S3 Data Warehouses]() Original publisher: [Read original article]() Author: Oleh Motrunych Published: 2025-09-22T00:00:00Z Content type: article Language: en Sources: [Scribd Tech]() Topics: [Amazon S3](), [Amazon Web Services](), [AWS Lambda](), [parquet](), [cloud-infrastructure]() Tags: [aws](), [aws-lambda](), [backup](), [backups](), [cleanup](), [compute](), [cost](), [data](), [data-lake](), [databases](), [datasets](), [deltalake](), [efficiency](), [featured](), [incremental](), [lambda](), [lifecycle](), [parquet](), [pipeline](), [reduce](), [retention](), [s3](), [terraform]() ### AI overview This article describes a monthly backup system for S3 data warehouses that uses incremental copying, S3 Inventory validation, parallel processing, and Glacier retention. It routes smaller workloads to AWS Lambda and larger workloads to ECS Fargate, with lifecycle-based cleanup for eligible files. ### Source excerpt We designed and implemented a scalable, cost-optimized backup system for S3 data warehouses that runs automatically on a monthly schedule. The system handles petabytes of data across multiple databases and uses a hybrid approach: AWS Lambda for small workloads and ECS Fargate for larger ones. At its core, the pipeline performs incremental backups -- copying only new or changed parquet files while always preserving delta logs -- dramatically reducing costs and runtime compared to full backups. Data is validated through S3 Inventory manifests, processed in parallel, and stored in Glacier for long-term retention. To avoid data loss and reduce storage costs, we also implemented a safe deletion workflow. Files older than 90 days, successfully backed up, and no longer present in the source are tagged for lifecycle-based cleanup instead of being deleted immediately. This approach ensures reliability, efficiency, and safety: backups scale seamlessly from small to massive datasets, compute resources are right-sized, and storage is continuously optimized. ## ClickHouse Release 25.8 DevFeed: [ClickHouse Release 25.8]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-09-05T00:00:00Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [parquet](), [Flight](), [Apache Iceberg]() Tags: [apache-arrow](), [bug](), [bug-fixes](), [clickhouse](), [data](), [data-lake](), [features](), [files](), [format](), [integration](), [new-features](), [parallelism](), [parquet](), [partitioning](), [performance](), [release](), [storage](), [streaming]() ### AI overview ClickHouse 25.8 introduces 45 new features, 47 performance optimizations, and 119 bug fixes. Highlights include a faster experimental native Parquet reader, Data Lake improvements, Hive-style partitioned writes, initial PromQL support, and Arrow Flight integration. ### Source excerpt ClickHouse 25.8 is available. In this post, you will learn about new features, including a new Parquet reader and Arrow Flight integration. ## Why HTAP Databases Are Giving Way to Disaggregated Architectures DevFeed: [Why HTAP Databases Are Giving Way to Disaggregated Architectures]() Original publisher: [Read original article]() Author: Zhou Sun Published: 2025-05-04T10:00:00Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [Databases](), [olap](), [Cloud](), [Latency](), [consistency](), [NoSQL](), [MongoDB](), [Hadoop](), [hdfs](), [CockroachDB](), [Amazon Redshift](), [vitess]() Tags: [cloud](), [cloud-data](), [cockroachdb](), [consistency](), [data-lake](), [databases](), [engineering](), [hadoop](), [hdfs](), [latency](), [mongodb](), [olap](), [redshift](), [sql](), [vitess]() ### AI overview The article traces the separation of transactional and analytical database workloads, explaining how differing storage and scaling requirements led to specialized OLTP and OLAP systems. It argues that HTAP as a single database architecture is declining while its underlying ideas persist in today's disaggregated data stack. ### Source excerpt This blog is inspired by Jordan Tigani's "Big Data is Dead." Jordan and I actually spent some time building an HTAP database at SingleStore. From the one database that did everything in the '80s, to the great divide, to HTAP, to today's disaggregated stack--here's why HTAP as a database is dead, but its spirit lives on. ## Project Teleport: Cost-Effective and Scalable Kafka Data Processing at Block DevFeed: [Project Teleport: Cost-Effective and Scalable Kafka Data Processing at Block]() Original publisher: [Read original article]() Author: Unni Krishnan Published: 2025-03-20T00:00:00Z Content type: article Language: en Sources: [Cash App Code Blog]() Topics: [data-processing](), [Kafka](), [migration](), [databricks](), [data lake](), [Apache Spark](), [Amazon S3](), [parquet](), [Amazon Redshift]() Tags: [acquisition](), [amazon](), [cloud](), [data](), [data-lake](), [data-processing](), [databricks](), [delta-lake](), [emr](), [kafka](), [parquet](), [pipelines](), [redshift](), [s3](), [scale](), [storage]() ### AI overview Project Teleport is Block's cross-region Kafka data-processing system for integrating Afterpay's Sydney-hosted data lake into Block's US-based ecosystem. Built with Delta Lake, Spark on Databricks, and object storage, it supports migration of legacy pipelines and reduced cloud egress costs by USD 540,000 per year. ### Source excerpt Teleport achieves efficient and reliable cross-region Kafka data processing at scale. Using this approach, Afterpay data team reduced cloud egress costs by USD 540,000 per year. ## How ClickHouse Integrates with Apache Iceberg and Modern Data Lakehouses DevFeed: [How ClickHouse Integrates with Apache Iceberg and Modern Data Lakehouses]() Original publisher: [Read original article]() Author: Melvyn Peignon Published: 2025-02-19T00:00:00Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Apache Iceberg](), [Amazon S3](), [data]() Tags: [amazon-s3](), [analytics](), [apache-iceberg](), [clickhouse](), [data](), [data-lake](), [integrations]() ### AI overview This article explains how ClickHouse integrates with Apache Iceberg, data lakes, lakehouses, and other external systems. It covers querying data in place, ingesting data for analytics, and ClickHouse's planned roadmap for 2025, including performance and catalog integration work. ### Source excerpt Read how ClickHouse integrates with Apache Iceberg and other open table formats, enabling seamless querying, data ingestion, and federation across modern data architectures, with a roadmap for enhanced performance and deeper catalog integrations. ## Out with the old file system DevFeed: [Out with the old file system]() Original publisher: [Read original article]() Author: Manfred Moser, David Phillips, Mateusz Gajewski Published: 2025-02-10T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Filesystems](), [Apache Iceberg](), [Hadoop](), [Amazon S3](), [Azure](), [Google Cloud Platform (GCP)]() Tags: [amazon-s3](), [apache-iceberg](), [apache-parquet](), [azure](), [compression](), [data-lake](), [deprecated](), [google-cloud](), [hadoop](), [maintenance](), [release]() ### AI overview Trino 470 deprecated its legacy Hadoop-based file system support, which will be removed in a future release. The article explains Trino's move toward custom file system implementations for cloud storage and describes the migration path for users, including catalog-level file system configuration and warnings for deprecated properties. ### Source excerpt What a long journey it has been! From the start Trino supported querying Hive data and used libraries from the Hive and Hadoop ecosystem. With the release of Trino 470 we mark another milestone to more features and better performance for data lake and lakehouse querying with Trino. We deprecated the legacy file system support, and will permanently remove them in an upcoming release. ## A report from the Trino Conference Tokyo 2023 DevFeed: [A report from the Trino Conference Tokyo 2023]() Original publisher: [Read original article]() Author: Yuya Ebihara Published: 2023-10-11T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Apache Iceberg](), [schema-evolution](), [Apache Spark](), [Data Infrastructure](), [migration](), [Serverless](), [Kubernetes](), [Amazon Web Services](), [Amazon S3]() Tags: [analytics](), [article](), [aws](), [conference](), [data-infrastructure](), [data-lake](), [japan](), [k8s](), [migration](), [performance](), [report](), [s3](), [schema-evolution](), [serverless](), [spark]() ### AI overview A report on the Trino Conference Tokyo 2023, covering sessions about Trino, Starburst products, Iceberg-based log infrastructure, Spark and Trino data infrastructure on Kubernetes, migration from Impala, and serverless Athena analytics. ### Source excerpt The Trino community in Japan held an online event on October 5th, 2023. This article is a summary of the conference aiming to share the presentations and provide an overview. ## Data Quality at Udemy -- Part 1 DevFeed: [Data Quality at Udemy -- Part 1]() Original publisher: [Read original article]() Author: Murat Migdisoglu Published: 2023-09-06T22:01:16Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Data Quality](), [data-architecture](), [data lake](), [Apache Spark](), [airflow](), [Streaming]() Tags: [airflow](), [architecture](), [data-architecture](), [data-catalog](), [data-governance](), [data-lake](), [data-lineage](), [data-quality](), [data-quality-management](), [pipelines](), [principal-engineer](), [quality](), [spark](), [streaming]() ### AI overview This article describes Udemy's efforts to improve data quality by establishing an end-to-end data lineage solution. It explains how distributed data ownership and self-service analytics make lineage important for impact analysis, change management, and identifying unused columns or orphan tables. ### Source excerpt Data Quality at Udemy -- Part 1Data Lineage Demystified- Why it Matters and How to Leverage its Magic for Informed Business Success! In late 2020, upon joining Udemy as a principal engineer for the data platform team, my focus shifted toward enhancing data quality within the organization. My journey began with conducting a comprehensive poll across the data organization, aimed at identifying the key pain points of data users. The results of the poll were eye-opening, revealing that 78% of users considered the absence of data provenance/lineage as a data quality issue. Furthermore, it was obvious that for a vast majority of the users, the inability to track data lineage was an important problem in impact analysis and detecting unused columns or orphan tables in the system. Inspired by these insights, I took the initiative to propose and launch two transformative projects. The first one, which is the subject of this article, is an ambitious initiative to establish a comprehensive end-to-end data lineage solution that will revolutionize our data ecosystem. The second project centers around data monitoring, which will be explored in another post. Udemy's sophisticated data architecture revolves around a data lake fed by diverse pipelines: system logs, streaming data from services, CDC listeners for replicated service databases, and more. The backbone of data transformations lies in Hive and Spark, while Airflow takes charge of orchestrating thousands of these pipelines. Unraveling Data Flow Complexity: Data Lineage in Growing Data Driven Organizations In the early stages of an organization's data-driven journey, data lineage may not be deemed crucial. With just a few pipelines managed by a centralized team, the dependency tree of the workflow orchestration typically suffices to comprehend the relationships between data entities. However, as the business scales up, relying on a single centralized team for all data flows becomes a bottleneck. Consequently, data organizatio ## Run analytics on Amazon S3 data with Tinybird S3 connector DevFeed: [Run analytics on Amazon S3 data with Tinybird S3 connector]() Original publisher: [Read original article]() Author: Tinybird Published: 2023-07-20T00:00:00Z Content type: article Language: en Sources: [Tinybird]() Topics: [Amazon S3](), [data](), [Query (disambiguation)]() Tags: [amazon-s3](), [analytics](), [data](), [data-lake](), [etl](), [pipelines](), [product-updates](), [s3]() ### AI overview The article presents the Tinybird S3 Connector as a way to run analytics on Amazon S3 data lake files without complex ETL pipelines. ### Source excerpt S3 analytics the easy way with the Tinybird Connector. Query your data lake files without complex ETL pipelines. ## The unnecessary hype strategy behind Microsoft Fabric DevFeed: [The unnecessary hype strategy behind Microsoft Fabric]() Original publisher: [Read original article]() Published: 2023-05-29T00:00:00Z Content type: opinion Language: en Sources: [Mutt0-ds Notes]() Topics: [Microsoft](), [data-engineering](), [Azure](), [Cloud](), [data lake](), [data-processing](), [etl](), [AI, ML & Data Engineering]() Tags: [azure](), [cloud](), [copilot](), [data](), [data-engineering](), [data-lake](), [data-processing](), [etl](), [microsoft](), [microsoft-azure](), [technical]() ### AI overview An opinion article examines Microsoft Fabric, a unified data platform announced at Microsoft Build. It describes Fabric's integration of data storage, processing, ETL, analytics, and business intelligence tools, while criticizing Microsoft's secrecy and hype-oriented launch strategy and noting that the platform was still in beta. ### Source excerpt If you are into Data Engineering in Microsoft Azure Cloud Environment, you probaly heard about Microsoft Fabric being announced last week... ## Just the right time date predicates with Iceberg DevFeed: [Just the right time date predicates with Iceberg]() Original publisher: [Read original article]() Author: Marius Grama Published: 2023-04-11T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Apache Iceberg](), [Query (disambiguation)](), [Optimization](), [DateTime](), [SQL]() Tags: [cast](), [data-lake](), [expression](), [filter](), [optimization](), [partitioning](), [performance](), [range](), [sql]() ### AI overview This article explains how date predicates can be optimized when querying Iceberg tables in a data lake. It covers partition pruning, hidden partitioning, constant folding, predicate pushdown, range predicates, and casting to help Trino avoid scanning irrelevant data and improve query performance. ### Source excerpt In the data lake world, data partitioning is a technique that is critical to the performance of read operations. In order to avoid scanning large amounts of data accidentally, and also to limit the number of partitions that are being processed by a query, a query engine must push down constant expressions when filtering partitions. ## Lakehouse summer camp at Trino Fest 2023 DevFeed: [Lakehouse summer camp at Trino Fest 2023]() Original publisher: [Read original article]() Author: Brian Olsen Published: 2023-04-05T00:00:00Z Content type: news Language: en Sources: [Trino Blog]() Topics: [data-architecture](), [Query (disambiguation)](), [data-governance](), [SQL](), [Data Space](), [Apache Iceberg]() Tags: [analytics](), [collaboration](), [community](), [conference](), [data-governance](), [data-lake](), [education](), [event](), [latest-features](), [sql](), [tools-and-integrations](), [trends]() ### AI overview Trino Fest 2023 is a virtual two-day event focused on Trino as a data lakehouse query engine and on improving data lakehouse management. The event emphasizes education, community collaboration, and practical insights into Trino, its ecosystem, and data and analytics platforms. ### Source excerpt Get ready to kick off your summer with Commander Bun Bun at Trino Fest 2023! This year's event is going virtual and will take place over two days, the 14th and 15th of June. The focus of the event will be on Trino as a data lakehouse query engine, with discussions on how new features and the ecosystem around Trino can support better data lakehouse management. Trino Fest 2023 is the new annual summer event dedicated to all things Trino. Building on the success of last year's Cinco de Trino, we're excited to bring the community together once again to explore the latest trends and innovations in Trino and data lakehouse management. With a focus on education, community collaboration, and inspiration, Trino Fest 2023 will be a valuable experience for anyone interested in improving their data and analytics platform. We hope to see you there as attendee, speaker, or sponsor! Read below to find out how to sign up. ## Securely Managing Your Audit Logs with Teleport and Snowflake DevFeed: [Securely Managing Your Audit Logs with Teleport and Snowflake]() Original publisher: [Read original article]() Author: kenneth.dumez@goteleport.com (Kenneth DuMez) Published: 2022-10-11T00:00:00Z Content type: tutorial Language: en Sources: [Teleport]() Topics: [audit](), [Logging](), [Security](), [SIEM, Security, Observability](), [event driven](), [Amazon S3](), [data]() Tags: [audit](), [aws](), [cloud](), [cybersecurity](), [data](), [data-lake](), [data-storage](), [database](), [deployment](), [event](), [event-driven](), [export](), [filter](), [format](), [how-to](), [infrastructure](), [latency](), [logging](), [logs](), [snowflake]() ### AI overview A tutorial explains how to forward Teleport audit logs to Snowflake using Fluentd as a collector, Amazon S3 as an intermediary, and Snowpipe for ingestion. It covers the security, organization, searchability, and low-latency benefits of this workflow for SIEM-related log management. ### Source excerpt How to securely manage all of your event-driven audit logs with Snowflake and Teleport. ## Trino Summit 2022: Sessions, speakers, and event details DevFeed: [Trino Summit 2022: Sessions, speakers, and event details]() Original publisher: [Read original article]() Author: Brian Olsen, Dain Sundstrom Published: 2022-09-22T00:00:00Z Content type: news Language: en Sources: [Trino Blog]() Topics: [data lake](), [SQL](), [Open Source](), [autoscaling]() Tags: [architectures](), [autoscaling](), [conference](), [data-lake](), [event](), [lyft](), [mpp](), [routing](), [summit]() ### AI overview This article announces Trino Summit 2022, a free hybrid conference taking place on November 10th, and previews selected sessions and speakers. Topics include Trino's open source project, query federation, large-scale ETL at Lyft, autoscaling, and fault-tolerant execution. ### Source excerpt Commander Bun Bun is back and this year we have an exciting lineup of speakers. Topics range from architectures like data mesh and data lakehouse, to running Trino at scale with fault-tolerant execution, and query federation. This conference is free and takes place on November 10th. The summit is a hybrid event for in-person and virtual attendance. Find out more details below! ## Cinco de Trino recap: Learn how to build an efficient data lake DevFeed: [Cinco de Trino recap: Learn how to build an efficient data lake]() Original publisher: [Read original article]() Author: Brian Olsen, Brian Zhan Published: 2022-05-17T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [data](), [data-engineering](), [big-data](), [Query (disambiguation)](), [datasets](), [Apache Iceberg](), [Stack Overflow](), [X (Twitter)]() Tags: [analytics](), [architecture](), [big-data](), [build](), [clusters](), [community](), [conference](), [cost-savings](), [data](), [data-engineering](), [data-lake](), [etl](), [learn](), [pipelines](), [recap](), [recovery](), [scale](), [tools]() ### AI overview A recap of the Cinco de Trino conference covering Trino's role in data lakehouse architectures, interactive and federated querying, adaptive query planning, and fault-tolerant execution. It highlights Project Tardigrade for autoscaling, spot-instance use, cost savings, and failure recovery, plus a Starburst Galaxy lab for ingesting, cleaning, and analyzing Twitter and Stack Overflow data. ### Source excerpt When Trino (formerly PrestoSQL) arrived on the scene almost 10 years ago, it immediately became known as the much faster alternative to the data warehouse of big data, Apache Hive. The use cases that you, as the community, have built had far exceeded anything we had imagined in complexity. Together we've made Trino not only the fastest way to interactively query large data sets, but also a convenient way to run federated queries across data sources to make moving all the data optional. At Cinco de Trino, we came full circle back to the next iteration of analytics architecture with the data lake. This conference offers advice from industry thought leaders about how to use best lakehouse tools with Trino to manage that data complexity. Hear from industry thought leaders like Martin Traverso (Trino), Dain Sundstrom (Trino), James Campbell (Great Expectations), Jeremy Cohen (DBT Labs), Ryan Blue (Iceberg), Denny Lee (Delta Lake), Vinoth Chandar (Hudi). You can watch the talks on-demand on the Cinco de Trino playlist. In this post, I'd like to cover the key items from each talk you won't want to miss. ## Integrating Confluent Schema Registry with Apache Spark applications DevFeed: [Integrating Confluent Schema Registry with Apache Spark applications]() Original publisher: [Read original article]() Author: Dominik Liebler Published: 2022-01-24T08:04:19Z Content type: tutorial Language: en Sources: [YAZIO Engineering - Medium]() Topics: [Kafka](), [Apache Spark](), [Streaming](), [data lake](), [parquet](), [Kotlin](), [Kubernetes](), [ceph](), [JSON Schema]() Tags: [apache-spark](), [backpressure](), [ceph](), [confluent](), [data-engineering](), [data-lake](), [data-pipeline](), [json](), [kafka](), [kotlin](), [kubernetes](), [parquet](), [payload](), [pipelines](), [schema](), [schemaregistry](), [serialization](), [spark](), [streaming]() ### AI overview This engineering article explains YAZIO's data pipeline from mobile and web applications through Kafka and Spark Structured Streaming into a Ceph-based data lake. It discusses why schemas matter and describes replacing JSON with Apache Avro and Confluent Schema Registry to reduce message size while keeping schema information externally stored and cached. ### Source excerpt At YAZIO, we believe in making decisions backed by data to help people live healthier lives through better nutrition. For each new and existing feature we want to evaluate how well it performs and how our users interact with it. In order to do so, we need a lot of data and we need to handle backpressure in our systems. To cope with that we use a Kafka cluster managed by Strimzi operators running in Kubernetes. The data itself is being ingested from our mobile and web apps via HTTP or TCP endpoints serialized into JSON and stored in Kafka by a small application written in Kotlin/JVM. Overview of our data pipeline architecture At the other end of the pipeline, different Spark Structured Streaming applications (also written in Kotlin) dump this information into our data lake residing in a Ceph bucket. They read data from Kafka, deserialize it, transform some of the fields and write Parquet files into the data lake using a new schema. Why schemas? Schemas play an important role in data pipelines because they give meaning and context to data. In a world without schemas we would still do random interpretations about the context and meaning of data every now and then when using it. As you might have guessed already this would lead to a lot of bugs and misunderstandings. Photo by EJ Strat https://unsplash.com/photos/VjWi56AWQ9k Similar to a legal contract that binds you to certain limits, a schema binds the data to certain limits and meaning which narrow down the need of interpretation. Choice of serialization formats At the time of writing, Confluent Schema Registry supports these three serialization formats: Apache Avro Protocol Buffers (protobuf) JSON Schema From those choices, only two really provide more than just validation of the data that is ingested and transmitted through our data pipelines. Avro and Protobuf also allow us to shrink the sizes of our topics because only the payload is contained in a message, while the repeating schema will not be stored. In the cas ## Trino on ice II: In-place table evolution and cloud compatibility with Iceberg DevFeed: [Trino on ice II: In-place table evolution and cloud compatibility with Iceberg]() Original publisher: [Read original article]() Author: Brian Olsen Published: 2021-07-12T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Apache Iceberg](), [schema-evolution](), [Cloud](), [migration](), [Filesystems]() Tags: [cloud-storage](), [compatibility](), [data-lake](), [evolution](), [migration](), [schema-evolution]() ### AI overview This Trino on Ice article explains how Apache Iceberg supports in-place partition and schema evolution while addressing compatibility with cloud and object storage. It contrasts Iceberg's live partition-spec updates with Hive's table migration requirements. ### Source excerpt Welcome to the Trino on ice series, covering the details around how the Iceberg table format works with the Trino query engine. The examples build on each previous post, so it's recommended to read the posts sequentially and reference them as needed later. Here are links to the posts in this series: Trino on ice I: A gentle introduction to Iceberg Trino on ice II: In-place table evolution and cloud compatibility with Iceberg Trino on ice III: Iceberg concurrency model, snapshots, and the Iceberg spec Trino on ice IV: Deep dive into Iceberg internals The first post covered how Iceberg is a table format and not a file format It demonstrated the benefits of hidden partitioning in Iceberg in contrast to exposed partitioning in Hive. There really is no such thing as "exposed partitioning." I just thought that sounded better than not-hidden partitioning. If any of that wasn't clear, I recommend either that you stop reading now, or go back to the first post before starting this one. This post discusses evolution. No, the post isn't covering Darwinian nor Pokémon evolution, but in-place table evolution! ## How we build the Image Gallery on trivago DevFeed: [How we build the Image Gallery on trivago]() Original publisher: [Read original article]() Author: Praneeth Peiris I want Published: 2021-07-07T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [Google Cloud Platform (GCP)](), [etl](), [data lake](), [Kafka](), [Front end](), [Amazon Web Services]() Tags: [amazon-web-services-aws](), [architecture](), [backend](), [cloud](), [data-lake](), [engineering](), [etl](), [frontend](), [gcp](), [kafka](), [platforms]() ### AI overview trivago describes migrating its hotel image-gallery ETL pipeline from Amazon Web Services to Google Cloud Platform. The redesigned architecture uses Dataflow jobs to snapshot images and tags, fetch changed accommodations, build and validate sorted galleries, and stream approved changes to frontend teams through Kafka. ### Source excerpt When was the last time you booked accommodation without checking its photos? Most probably never! Because having imagery information makes our decision-making process much easier and faster. How... ## Prophecy: Teamwork's Data Lake DevFeed: [Prophecy: Teamwork's Data Lake]() Original publisher: [Read original article]() Author: Joe Minichino Published: 2020-07-17T11:41:10Z Content type: opinion Language: en Sources: [Teamwork]() Topics: [data lake](), [data-engineering](), [data](), [decision-making](), [big-data](), [Databases](), [Google Analytics](), [stripe]() Tags: [analytics](), [aws](), [big-data](), [data](), [data-engineering](), [data-lake](), [databases](), [decision-making](), [google-analytics](), [stripe]() ### AI overview Teamwork describes reevaluating its analytics approach because data was distributed across product databases and third-party services, making it difficult to connect leads, product usage, and revenue. The article argues that a data lake can make analytics and data-informed decision-making more practical, while clarifying that data lakes and big data are not limited to enterprise-scale environments. ### Source excerpt You need a Data Lake. The Context Teamwork has been around for more than 10 years. Starting out as a project management and work collaboration platform and later expanding into other areas, such as help-desk, chat, document management and CRM software. As the company has grown and evolved, data has grown, changed, expanded, diversified, fragmented, then changed again. Analytics in this landscape are not for the faint of heart. The Problem The issue at the base of the decision to start re-evaluating our analytics approach, at Teamwork, was because we have way too many data sources to make sense of all the data we collect. Much of the data ends up being "dark", underutilized, unleveraged. We have multiple databases shards, for each product, along with platform databases containing global customer information; and then we have 3rd parties such as Stripe, Marketo, Chart Mogul and Google Analytics to name the most important ones. We realized that making a connection between sales leads in Marketo, their usage of our products (GA and our DBs) and their revenue (our DBs, Stripe, Chart Mogul) was impossible. Well, it was possible, but incredibly painful and poorly automated, making it slow and error prone. And by addressing this problem we are getting a large number of benefits back, practically for free. You need Analytics Let's cut to the chase: you need analytics. No matter how big or small your business or enterprise is, you need analytics to take more informed business decisions. I do not doubt there are individuals with great gut-driven decision-making skills, but by and large, it's better if you look at data to make decisions. That's why you need a Data Lake. It could be a really small lake, it could be a pond or a puddle. But you need it for better decision making. Data Lakes, Big Data and other buzzwords Let's clear the air on a couple of misconceptions. The terms "Big Data" and "Data Lake" absolutely scream of corporate, of enterprise, of governance, compliance, r ## Implementing ETL on GCP DevFeed: [Implementing ETL on GCP]() Original publisher: [Read original article]() Author: Deepmala Published: 2020-07-15T20:53:17Z Content type: tutorial Language: en Sources: [Brave New Geek]() Topics: [DataOps](), [Cloud](), [BigQuery](), [data loss prevention](), [Low code](), [No-code](), [olap]() Tags: [analytics](), [analytics-pipeline](), [bi-tools](), [bigquery](), [cdap](), [cloud-data-loss-prevention](), [cloud-dataflow](), [cloud-dataprep](), [cloud-dataproc](), [cloud-pub-sub](), [cloud-storage](), [cloud-tasks](), [data-analytics](), [data-fusion](), [data-lake](), [data-loss-prevention](), [elt](), [etl](), [gcp](), [no-code](), [serverless](), [sql]() ### AI overview A practical guide to building ETL pipelines on Google Cloud Platform using Google-managed services. It explains a two-phase architecture with Cloud Storage as a data lake, Cloud Data Loss Prevention for sensitive-data detection or redaction, and BigQuery as the curated data warehouse, with attention to low-code and no-code approaches. ### Source excerpt ETL (Extract-Transform-Load) processes are an essential component of any data analytics program. This typically involves loading data from disparate sources, transforming or enriching it, and storing the curated data in a data warehouse for consumption by different users or systems. An example of this would be taking customer data from operational databases, joining it with data from Salesforce and Google Analytics, and writing it to an OLAP database or BI engine. ## Object Storage Use Cases Part 3: Storing binary files in Object Storage instead of in your git repository DevFeed: [Object Storage Use Cases Part 3: Storing binary files in Object Storage instead of in your git repository]() Original publisher: [Read original article]() Author: Unknown (noreply@blogger.com) Published: 2019-04-29T09:06:00Z Content type: tutorial Language: en Sources: [CSC - IT Center For Science - Cloud Team]() Topics: [Git](), [ceph](), [pip](), [Haskell]() Tags: [aws](), [ceph](), [data](), [data-lake](), [git](), [install](), [object-storage](), [python](), [rados](), [radosgw](), [s3](), [storage]() ### AI overview This post explains why binary files should not be stored directly in Git repositories and explores using CSC's Pouta Cloud Object Storage instead. It documents attempts with git-annex and git-bigstore, including a patch to configure a non-default S3-compatible endpoint. ### Source excerpt 1. It's bad to store binary blobs in git. This is known [1] 2. A great place to store binary files is in object storage 3. Can we use CSC's Pouta Cloud Object Storage service for this? Why yes, yes you can! This is a post in a series about different use cases for object storage. Others include for example hosting a reveal.js presentation and NFS server backups More details about CSC's Pouta Cloud Object Storage Service, it is based on CEPH Rados Object Storage Gateway. What The general solution* to storing binaries in git is to not store them in git and instead create a link to some other place Attempt 1 Install an application that you like. In https://www.perforce.com/blog/storing-large-binary-files-in-git-repositories I searched for "S3", found git-annex which had recent commits so it looked like it wasn't dead. Installation instructions: https://git-annex.branchable.com/install/ The version in EPEL/Ubuntu Xenial does not have S3 support and is from 2014. There's a recent one in Fedora's repos though. Unfortunately, compiling it requires cabal. I don't know Haskell. https://spack.readthedocs.io/en/latest/package_list.html does not have any Haskell, cabal or git-annex available. Next. Attempt 2 Next application in the blog post was https://github.com/lionheart/git-bigstore To install it was much more comfortable for me: pip install "git-bigstore<=2.0" First issue was that looking at https://github.com/lionheart/git-bigstore/blob/master/bigstore/backends/s3.py there was no way to tell it to use another endpoint than the default of AWS S3. A small patch takes care of that, add , endpoint_url='https://object.pouta.csc.fi:443' to the client() boto3 call in the backends/s3.py file from git-bigstore. For a more detailed example see this Pull Request. After spending some time going down some rabbit holes it seems that the Profile option that is supported in git-bigstore does not support endpoints. It can only be used to specify credentials and a few other parameters. Mino [Next page]()