# When you deserve better (systems)

DevFeed: [When you deserve better (systems)](<https://devfeed.tech/articles/when-you-deserve-better-systems-19844.md>)

Original publisher: [Read original article](<https://tech.gc.com/when-you-deserve-better-systems/>)

Author: GameChanger

Published: 2019-10-25T09:00:00Z

Content type: article

Language: en

Sources: [GameChanger](<https://devfeed.tech/sources/gamechanger.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [data](<https://devfeed.tech/topics/data.md>), [systems](<https://devfeed.tech/topics/systems.md>), [event driven](<https://devfeed.tech/topics/event-driven.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [blog](<https://devfeed.tech/tags/blog.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [data](<https://devfeed.tech/tags/data.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [logs](<https://devfeed.tech/tags/logs.md>), [overview](<https://devfeed.tech/tags/overview.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [systems](<https://devfeed.tech/tags/systems.md>), [technical](<https://devfeed.tech/tags/technical.md>)

## AI overview

A technical blog post revisits a data pipeline and data warehouse architecture built around Kafka and Zookeeper. It describes how the system supported warehouse growth, business intelligence, and analysis, while infrastructure, Kafka capabilities, and data consumers changed over time.

## Source excerpt

Preface Before we begin, I suggest settling in. Unlike my previous post, this one won't be short. It'll also be more technical, though I will do my best to link to resources in case you get lost along the way. So my suggestion is get a nice cuppa, turn off notifications, and brace yourself: We're going on an adventure. In the beginning Back in 2016, when we were all younger and more innocent, Alex Etling wrote a series of blog posts about his learnings in setting up our original data pipeline, which was built around Kafka, which is built upon Zookeeper. I'll let his blog posts speak for themselves in case you're interested in that history, though they're not necessary prerequisites for this post if you'd rather save them for later. Fun With Kafka: Adding a New Box Type Part 1 and Part 2 Experimenting With Kafka Scaling With Kafka Instead to give you a lay of the land, here's a high level overview of the data pipeline and data warehouse systems, to function as a map for our adventure: Bringing back this architecture diagram, as it's so wonderful. [1] Spiffy, huh? Thus the Kafka and the Zookeeper clusters, and all the host of them, were finished This pipeline setup has allowed us to do things like grow our data warehouse, provide business intelligence to data users, and do deep analysis as a data team to answer questions about how we can better serve our customers. At the same time, a lot has changed around this system: in our infrastructure setup, in Kafka's capabilities, and in who uses the pipeline and for what. Back then, the desire was to answer a few basic questions about the customers using a variety of data; now the consumers of our data go in hard to look at the nitty gritty details themselves and really understand the complexity in the data to get their answers. What makes a team successful? Are our features helpful to teams across sports and ages? The people need to know. You cannot know everything a system will be used for when you start: it is only at the