# How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes

DevFeed: [How We Refresh Razorpay's Data Warehouse 10x Faster with Graphs and Indexes](<https://devfeed.tech/articles/how-we-refresh-razorpay-s-data-warehouse-10x-faster-with-graphs-and-indexes-24040.md>)

Original publisher: [Read original article](<https://engineering.razorpay.com/how-we-refresh-razorpays-data-warehouse-10x-faster-with-graphs-and-indexes-538abc244703?source=rss----6407ad2e59af---4>)

Author: Amit Prabhu

Published: 2026-07-14T14:06:16Z

Content type: article

Language: en

Sources: [Razorpay Engineering - Medium](<https://devfeed.tech/sources/razorpay-engineering-medium.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [microservices architecture](<https://devfeed.tech/topics/microservices-architecture.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [batch](<https://devfeed.tech/tags/batch.md>), [data](<https://devfeed.tech/tags/data.md>), [graphs](<https://devfeed.tech/tags/graphs.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [razorpay](<https://devfeed.tech/tags/razorpay.md>), [spark](<https://devfeed.tech/tags/spark.md>), [trino](<https://devfeed.tech/tags/trino.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>)

## AI overview

Razorpay describes its data warehouse refresh pipeline, which builds wide denormalized Facts by joining data from multiple microservices. The article covers the original Airflow- and Spark-based full-refresh process, the underlying lake formats and query layer, and the scaling challenges that led the team to reconsider refresh strategy, data layout, and high-cardinality dimensions.

## Source excerpt

Contributors: Utkarsh Koppikar Rohan Background Razorpay provides the payment infrastructure for millions of merchants globally. Behind every payment, settlement, and refund is a microservices architecture where each service owns its own database. While this keeps services independent and scalable, it creates a challenge for stakeholders who need to see across those boundaries. The Data Platform team manages the infrastructure that bridges this gap. Transactional data flows into the lake via CDC pipelines, ingested onto S3 in Delta Lake, Apache Iceberg, or plain Parquet formats. On top of the lake, we build domain-specific warehouse tables -- wide, pre-joined tables that co-locate all the data a consumer needs, queryable via Trino. These power two use cases: Analytics (internal dashboards on Tableau and Superset) and Reporting (merchants and regulated entities who download structured data exports; Razorpay generates nearly a million such reports per month). The warehouse tables that power both use cases are called Facts. A Fact is a flat denormalised table on S3, produced by joining 10 to 30 microservice tables and materialising the result once. A settlement Fact, for example, merges payments, refunds, adjustments, and card details into a single wide row so that a dashboard or report reads from a single table instead of joining across services in real time. It is closer to a domain-specific materialised view than a classical data warehouse fact table. We maintain over 50 such Facts, and approximately 40% of all merchant reports are served directly from them. As data volumes and the number of entities per fact grew, the batch generation pipeline began to show its limits, prompting us to rethink the refresh strategy, the data layout, and how to handle high-cardinality dimensions. The rest of this post covers that journey. The Full Refresh Pipeline: Our Baseline and the Pain The original full-refresh pipeline was straightforward. Schedule: Airflow schedules Spark jobs o