# From pipeline to beyond

DevFeed: [From pipeline to beyond](<https://devfeed.tech/articles/from-pipeline-to-beyond-19833.md>)

Original publisher: [Read original article](<https://tech.gc.com/from-pipeline-to-beyond/>)

Author: GameChanger

Published: 2021-05-05T09:00:18Z

Content type: article

Language: en

Sources: [GameChanger](<https://devfeed.tech/sources/gamechanger.md>)

Topics: [Kafka](<https://devfeed.tech/topics/kafka.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data](<https://devfeed.tech/topics/data.md>), [streaming-data-processing](<https://devfeed.tech/topics/streaming-data-processing.md>), [Back end](<https://devfeed.tech/topics/backend.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [backend](<https://devfeed.tech/tags/backend.md>), [consul](<https://devfeed.tech/tags/consul.md>), [data](<https://devfeed.tech/tags/data.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [redshift](<https://devfeed.tech/tags/redshift.md>), [s3](<https://devfeed.tech/tags/s3.md>), [terraform](<https://devfeed.tech/tags/terraform.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>)

## AI overview

The article examines options for moving data out of Kafka into a data warehouse and archive. It discusses off-the-shelf tools such as Kafka Connect, Secor, and Gobblin, the limitations encountered, and the development of a custom solution. The requirements include preserving Avro data and schemas, writing to S3, and partitioning data by processing time or event time.

## Source excerpt

An overview of off-the-shelf solutions for moving data out of Kafka, problems we had making those systems work, and how we wrote our own solution and stood it up for those in a similar situation. You cannot know everything a system will be used for when you start: it is only at the end of its life you can have such certainty. [1] Many moons ago I wrote about our design for upgrading our data pipeline, which lightly touched on how we'd move data out of our pipeline (Kafka) to downstream systems, namely our data warehouse and data archive. At that time we hadn't really been able to dive into focusing on getting the data out of Kafka, because getting data in to Kafka is often much more custom and complex, and we thought we'd be able to use an off the shelf solution like Kafka Connect to move data out, don't even worry about it. We were, uh -- we were wrong. Let me take you on our journey, in case you're on this journey too. The problem space Solution 1: Kafka Connect Solution 2: Secor or Gobblin Solution 3: we'll do this ourselves Tangent: naming things How you can do this yourselves, code edition How you can do this yourselves, infrastructure and metrics edition Takeaways The problem space Programmers are not to be measured by their ingenuity and their logic but by the completeness of their case analysis. [4] At a high level, the problem we needed a solution for was as follows: Data enters the data pipeline from numerous backend systems. This crossover point is producers into the pipeline, which we'd already implemented. Data from the data pipeline needs to move into the data warehouse and the data archive. This crossover point would be a consumer on the pipeline. Ideally we'd like the same consumer for both needs that we can simply configure differently. We want to preserve our data's Avro format along side its schemas. This would allow every system that interacts with the data to use the same language. We want to write our data to S3. data warehouse: This will be our