# Why Data Pipelines Should Avoid Monolithic Queue Processors

DevFeed: [Why Data Pipelines Should Avoid Monolithic Queue Processors](<https://devfeed.tech/articles/building-data-pipelines-learning-number-01-die-monolith-die-28152.md>)

Original publisher: [Read original article](<http://fuzzyblog.io/blog/data_pipeline/2020/07/20/building-data-pipelines-die-monolith-die.html>)

Author: Fuzzygroup

Published: 2020-07-20T00:00:00Z

Content type: article

Language: en

Sources: [Scott Johnson](<https://devfeed.tech/sources/scott-johnson.md>)

Topics: [data observability](<https://devfeed.tech/topics/data-observability.md>), [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [debugging](<https://devfeed.tech/topics/debugging.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [cpu](<https://devfeed.tech/topics/cpu.md>), [Tensorflow](<https://devfeed.tech/topics/tensorflow.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [process](<https://devfeed.tech/tags/process.md>), [sqs](<https://devfeed.tech/tags/sqs.md>), [tensorflow](<https://devfeed.tech/tags/tensorflow.md>)

## AI overview

The article describes lessons from building a high-performance, near-real-time data pipeline on AWS with SQS and machine learning components. It argues that a monolithic queue processor can increase cloud costs when GPU-backed processing is applied to work that does not require a GPU, make debugging harder, and limit independent scaling of different processing routines.

## Source excerpt

I am in the process of wrapping up a year long engagement where I: Built a high performance data pipeline Capable of processing all of Twitter in real time / near real time Applied multiple tools to the data at different stages of the pipeline Applied one or more Machine Learning models at different stages of the pipeline Operated on AWS using SQS as the queueing structure This blog post talks about one of the key things I learned in terms of the data pipeline, specifically: **Do Not Build Data Pipelines Around a Monolithic Queue Processor ** Note: By queue processor I mean the bit of software which pulls the data of the queue, operates on it and then puts it back. This lesson may be obvious to some but we took a meandering approach to this problem where we started with the idea of distributed pipeline components, moved to a monolithic approach and then ended up back at a distributed approach. As with a lot of research endeavors, the obvious conclusion wasn't quite so obvious in the throes of the research. Lesson 01: Monolithic Queue Processing Raises Your Costs At the heart of our queue processing were a number of Machine Learning components (python / tensorflow) that really needed a GPU for efficient data processing. The problem here is that when you have a monolithic queue processor, all your processing happens on a box with the GPU whether or not all that processing needs the GPU. When you are using cloud computing, you pay for the GPU whether not not it is being used for a given operation. And since GPU boxes generally cost at least 5x to 6x more than CPU only boxes, well, our monolithic queue processor proved to be an economic disaster. Lesson 02: Monolithic Queue Processing Is Harder to Debug After realizing Lesson 01, I took our monolithic queue processor apart and broke it down into 8 (ultimately 9) individual queue processors. One thing that I quickly found is that debugging the 8 individual queue processors was dramatically easier than debugging the singl