# Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption

DevFeed: [Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption](<https://devfeed.tech/articles/scaling-grab-s-data-lake-our-journey-to-apache-iceberg-adoption-1252.md>)

Original publisher: [Read original article](<https://engineering.grab.com/our-journey-to-apache-iceberg-adoption>)

Author: Rahul Penti

Published: 2026-07-10T00:00:00Z

Content type: article

Language: en

Sources: [Grab Tech](<https://devfeed.tech/sources/grab-tech.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [data](<https://devfeed.tech/topics/data.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [compute](<https://devfeed.tech/tags/compute.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [latency](<https://devfeed.tech/tags/latency.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [performance](<https://devfeed.tech/tags/performance.md>), [s3](<https://devfeed.tech/tags/s3.md>), [scale](<https://devfeed.tech/tags/scale.md>), [spark](<https://devfeed.tech/tags/spark.md>)

## AI overview

Grab describes its transition from Hive Parquet and a directory-based Hive Metastore architecture toward Apache Iceberg and a table-centric lakehouse. The article outlines scalability problems involving metadata latency, small files, S3 request overhead, and operational management, and mentions a unified Spark catalog built to hide table-format differences.

## Source excerpt

Introduction: The evolution of Grab's Data Lake At Grab's scale, managing petabytes of data across billions of S3 objects demands more than a storage layer. It demands a robust architectural primitive that supports the high-concurrency needs of a modern "Lakehouse." Our goal is full storage-compute separation, leveraging S3 as an elastic foundation for both near-real-time metrics and large-scale batch transformations. For years, the vast majority of our tables were Hive Parquet, managed through the Hive Metastore with a directory-based layout. This model served us well, but as data volume grew, the directory-and-metastore approach became the limiting factor. We are now transitioning to a table-centric architecture built on modern table formats, treating data as a first-class primitive to ensure consistency and performance across our internal data transformation platforms: Slide, which powers batch transformations, and Hugo, which handles online-to-data-lake ingestion. Along the way, we also built the UnifiedSparkCatalog, a unified Spark catalog that hides table-format differences from users entirely, which we are open-sourcing alongside this post. The catalyst for change: Challenges with Hive Parquet For years, Hive Parquet was the backbone of our Data Lake, representing the vast majority of our tables. However, as data volume scaled, the architectural limitations of directory-based storage became apparent. We identified four primary bottlenecks: Catalog latency: The Hive Metastore (HMS) became a centralized failure point. High concurrency during metadata access led to O(n) listing overhead, where query planning time scaled linearly with partition count, crippling throughput. The small file problem: The directory layout left us with severe file fragmentation. Certain Machine Learning (ML) datasets had an average file size under 1 MB, with thousands of files in each partition. At this scale, the overhead of S3 object listing and metadata request latency drove up Appl