# Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

DevFeed: [Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned](<https://devfeed.tech/articles/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-135.md>)

Original publisher: [Read original article](<https://netflixtechblog.com/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-f4b792f3f0d8?source=rss----2615bd06b42e---4>)

Author: Netflix Technology Blog

Published: 2026-07-13T22:44:11Z

Content type: article

Language: en

Sources: [Netflix](<https://devfeed.tech/sources/netflix.md>), [Netflix TechBlog - Medium](<https://devfeed.tech/sources/netflix-techblog-medium.md>)

Topics: [telemetry](<https://devfeed.tech/topics/telemetry.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [incident](<https://devfeed.tech/topics/incident.md>), [cpu](<https://devfeed.tech/topics/cpu.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [backend-development](<https://devfeed.tech/tags/backend-development.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [ebpf](<https://devfeed.tech/tags/ebpf.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [observability](<https://devfeed.tech/tags/observability.md>), [scale](<https://devfeed.tech/tags/scale.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [tracing](<https://devfeed.tech/tags/tracing.md>)

## AI overview

Netflix describes the engineering lessons from building a real-time service-dependency map at scale. The system ingests streaming network-flow, IPC-metric, and tracing data to support topology queries while addressing production issues such as Kafka lag, memory pressure, traffic skew, and garbage-collection overhead.

## Source excerpt

By Parth Jain, Rakesh Sukumar, Yingwu Zhao, Renzo Sanchez-Silva & Nathan Fisher A deep dive into the engineering challenges of building a real-time service dependency map at Netflix scale: from streaming architectures and distributed aggregation pipelines to time-travel queries and the methodology that made it work. Introduction In our first post, we introduced the problem: engineers at Netflix needed a unified, real-time view of service dependencies to troubleshoot faster, understand blast radius, and navigate our distributed architecture. We described our multi-source approach, combining eBPF network flows, IPC metrics, and distributed tracing into physically separate graph layers that can be queried independently or merged into a comprehensive view. That post explained what we built and why. This post is about how, the engineering reality of building this system at Netflix scale. Here's the truth: the first version worked perfectly... in our local environment. Production was a different story. Kafka consumers fell behind. Instances ran out of memory. Some nodes received 100x the traffic of others. Garbage collection pauses consumed more CPU than actual business logic. What you'll learn in this post isn't a success story, it's a learning journey. We'll walk through the architecture decisions that enabled scale, the production challenges that tested those decisions, the optimization methodology that guided us through, and the lessons that apply to any distributed system. Along the way, we'll share the innovations that made it possible to process millions of flow records per second, reconstruct topology at any point in time, and provide sub-second query responses, all while maintaining near real-time freshness. Architecture Deep-Dive: Building for Streaming and ScaleStreaming-First: Why Real-Time Matters Traditional service topology systems use batch processing, aggregating data hourly or daily, then storing complete snapshots. This approach works at a modest scale bu