# Let me automate that for you

DevFeed: [Let me automate that for you](<https://devfeed.tech/articles/let-me-automate-that-for-you-19839.md>)

Original publisher: [Read original article](<https://tech.gc.com/let-me-automate-that-for-you/>)

Author: GameChanger

Published: 2019-09-20T18:29:38Z

Content type: article

Language: en

Sources: [GameChanger](<https://devfeed.tech/sources/gamechanger.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Refactoring](<https://devfeed.tech/topics/refactoring.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [Database](<https://devfeed.tech/topics/database.md>), [SQL](<https://devfeed.tech/topics/sql.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [automation](<https://devfeed.tech/tags/automation.md>), [backend](<https://devfeed.tech/tags/backend.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [refactor](<https://devfeed.tech/tags/refactor.md>), [refactoring](<https://devfeed.tech/tags/refactoring.md>), [sql](<https://devfeed.tech/tags/sql.md>), [tests](<https://devfeed.tech/tags/tests.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>)

## AI overview

The article describes GameChanger's data pipeline and warehouse, where engineers configure producers, pipeline connections, and warehouse tables. A refactoring project simplified producers, removed boilerplate, and added tests, but creating or updating warehouse tables remained a difficult manual task because the warehouse uses different SQL syntax and lacks library-generated code.

## Source excerpt

As GameChanger's data engineer, I oversee the data pipeline and data warehouse. Sounds simple, right? And at a high level, it is! Fig 1.1: high level architecture diagram. Some complexity removed due to it being kinda boring for this post. Producers produce into the pipeline, and our main consumer is the ETL job which moves data to our warehouse, enabling anybody to come get answers to their questions and see what's happening across all our systems. Boom: easy. Well, not quite. Who owns what Since data can come from any number of backend systems and teams, engineers are responsible for writing the setup that shepherds their data through the system: a producer that lives near their data, the pipe it travels through in the pipeline, and the warehouse table. This often means new data that I'm unfamiliar with arrives in our warehouse without me even knowing it's been set up, which is actually kind of neat: the system should be so easy to work with that you don't need the data engineer. After a recent refactoring project, producers were made as simple as possible with removed boilerplate and plenty of tests to automatically catch the most common bugs engineers encounter. Typically, engineers have no problems with making their producers. Fig 2.1: engineers before and after producer refactor project. Studies have shown that engineers prefer to be happy. The pipe their data travels through is set up by filling in a form and pressing a button. Again, engineers typically have no problems with this. It's the warehouse table that becomes a pain point. Follow the readme There are two times non data engineers need to interact with warehouse tables: they've created a new producer which needs a table for their data to land in. they've updated an existing producer which needs its table updated as well. The second point is trickier and easier to get wrong, but the first point proved just as difficult for many engineers and far more common, especially if the engineers in question had