# How we built ngrok's data platform

DevFeed: [How we built ngrok's data platform](<https://devfeed.tech/articles/how-we-built-ngrok-s-data-platform-83245.md>)

Original publisher: [Read original article](<https://ngrok.com/blog/how-we-built-ngroks-data-platform>)

Author: Christian Hollinger

Published: 2024-09-26T00:00:00Z

Content type: article

Language: en

Sources: [\[ngrok news\]](<https://devfeed.tech/sources/ngrok-news.md>)

Topics: [data lake](<https://devfeed.tech/topics/data-lake.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [backend-engineering](<https://devfeed.tech/tags/backend-engineering.md>), [bazel](<https://devfeed.tech/tags/bazel.md>), [company](<https://devfeed.tech/tags/company.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [engineering](<https://devfeed.tech/tags/engineering.md>)

## AI overview

Ngrok describes how a small engineering team built and evolved its data platform, including batch ingestion, streaming Protobuf events through Kafka and Flink into Iceberg, and analytics with dbt and Superset. The article also details schema handling, integrating data tools with the Go monorepo and CI, and using platform data for abuse investigations.

## Source excerpt

At ngrok, we manage a ~100TiB, 500+ table data lake, managed by a very small team. Here's a look at how we built it and what unique challenges we solved.