# Let me automate that for you II, Electric Bugaloo

DevFeed: [Let me automate that for you II, Electric Bugaloo](<https://devfeed.tech/articles/let-me-automate-that-for-you-ii-electric-bugaloo-19838.md>)

Original publisher: [Read original article](<https://tech.gc.com/let-me-automate-that-for-you-ii-electric-bugaloo/>)

Author: GameChanger

Published: 2021-05-07T18:29:38Z

Content type: article

Language: en

Sources: [GameChanger](<https://devfeed.tech/sources/gamechanger.md>)

Topics: [SQL](<https://devfeed.tech/topics/sql.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [Pull Request](<https://devfeed.tech/topics/pull-request.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>), [Structured-data](<https://devfeed.tech/topics/structured-data.md>), [Script](<https://devfeed.tech/topics/script.md>), [Slack](<https://devfeed.tech/topics/slack.md>), [Documentation](<https://devfeed.tech/topics/documentation.md>)

Tags: [automation](<https://devfeed.tech/tags/automation.md>), [blog](<https://devfeed.tech/tags/blog.md>), [blog-post](<https://devfeed.tech/tags/blog-post.md>), [building](<https://devfeed.tech/tags/building.md>), [data](<https://devfeed.tech/tags/data.md>), [documentation](<https://devfeed.tech/tags/documentation.md>), [github](<https://devfeed.tech/tags/github.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [reactive](<https://devfeed.tech/tags/reactive.md>), [redshift](<https://devfeed.tech/tags/redshift.md>), [sql](<https://devfeed.tech/tags/sql.md>), [systems](<https://devfeed.tech/tags/systems.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>)

## AI overview

The article describes evolving an embedded SQL generator and related scripts into a standalone SQL producer for warehouse schema and table management. The system generates SQL migrations, creates or updates tables, optimizes table performance, documents proposed changes, opens pull requests, and notifies engineers in Slack for review. The article also discusses limitations of the reactive original implementation and outlines the design of the improved system.

## Source excerpt

Improving our original, embedded SQL generator and some related scripts by converting them to a better, long term, stand alone SQL producer that's faster, more reliable, and more obvious. About seventeen years ago, in 2019, I published my blog post "Let me automate that for you" about a design for automating creating warehouse tables based on schemas for new event data. The idea was when our ETL system couldn't load waiting data into a warehouse table (as there was no table to be found), it would look up the schema for that data, convert the schema to a SQL statement, then issue a PR to the repo where SQL migrations for such needs are kept. Eventually creating tables made a friend, updating tables when there was a mismatch between the schema of the data we were loading and the schema of the table in the warehouse, and a third buddy joined the part, optimizing a table to improve its performance. The system had some absolutely great qualities: it automated acting on errors it saw, it generated great documentation in the PR and the SQL statement (with comments for discussions and places to review more closely), and it posted to Slack to let engineers know that there was something for them to do a final review on. However... it wasn't perfect. Reading is going toward something that is about to be, and no one yet knows what it will be. [1] Let me take you through the evolution of our embedded SQL generator to stand-alone SQL producer. Limitations of previous implementation Opportunities to build it better Building blocks of a stand-alone SQL producer Joining the human needs with the computer's logic Detailed breakdown of the services available Troubleshooting live Final thoughts Appendix A: Redshift optimization queries Appendix B: select Github logic Limitations of previous implementation While the SQL generator eased so much work for so many different people in the company, it had some... strange caveats, shall we say. Some were more noticable than others but all were, in