# Using Terraform, Consul, and Ansible to Define and Deploy a Data Pipeline

DevFeed: [Using Terraform, Consul, and Ansible to Define and Deploy a Data Pipeline](<https://devfeed.tech/articles/i-ll-say-this-exactly-once-19836.md>)

Original publisher: [Read original article](<https://tech.gc.com/ill-say-this-exactly-once/>)

Author: GameChanger

Published: 2019-12-18T12:00:00Z

Content type: tutorial

Language: en

Sources: [GameChanger](<https://devfeed.tech/sources/gamechanger.md>)

Topics: [Infrastructure as code](<https://devfeed.tech/topics/infrastructure-as-code.md>), [Ansible](<https://devfeed.tech/topics/ansible.md>), [Terraform](<https://devfeed.tech/topics/terraform.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [JSON](<https://devfeed.tech/topics/json.md>), [YAML](<https://devfeed.tech/topics/yaml.md>), [Git](<https://devfeed.tech/topics/git.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [ansible](<https://devfeed.tech/tags/ansible.md>), [collection](<https://devfeed.tech/tags/collection.md>), [consul](<https://devfeed.tech/tags/consul.md>), [data](<https://devfeed.tech/tags/data.md>), [git](<https://devfeed.tech/tags/git.md>), [iac](<https://devfeed.tech/tags/iac.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [infrastructure-as-code](<https://devfeed.tech/tags/infrastructure-as-code.md>), [json](<https://devfeed.tech/tags/json.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [terraform](<https://devfeed.tech/tags/terraform.md>), [yaml](<https://devfeed.tech/tags/yaml.md>)

## AI overview

The article explains how infrastructure as code can document and reproduce a data pipeline across environments. It describes combining Terraform, Consul, and Ansible so infrastructure, machine configuration, and related settings can be managed together.

## Source excerpt

You've finally got everything working in your staging environment: the new systems talk to each other, everything is running smoothly, your dashboards are beautiful and pristine. Now you need to get it into production with zero down time and no interruptions. And then you realize, you've no idea what you actually did in your staging environment. Let's talk about infrastructure I don't consider myself an "infrastructure lady;" it's just not my jam, and that's fine. However I have had to learn a lot about our infrastructure for setting up our new data pipeline system -- and I don't just show up to learn the minimum. I come to m a s t e r. IaC, or infrastructure as code, is the idea of setting up your infrastructure using definition files, code, and standard programming practices. In a way, it brings infrastructure to those of us who might otherwise be overwhelmed by what's going on and what we're suppose to do. Why manually enter configuration values when you can store them in a YAML or JSON file in a git repo? Suddenly you can see its history, you can search it, and it documents itself to a certain extent. This last point is of particular interest to me, as so much of the data pipeline is just how the systems work together, what the configurations are to facilitate this, and and documenting how all this works. If -- if -- we could have our data pipeline and all its friends live in some straightforward, self documenting IaC setup, it would make it easy not only to remember what we did but also to onboard someone new to the system and to deploy it to different environments with high confidence that it'll work. At GameChanger, that meant making three systems work together: Terraform, Consul, and Ansible. Terraform sets up what we want the landscape of our machines to look like. Ansible sets up what we want the landscape on our machines to look like. And Consul is the new kid in the collection, just here to have a good time and be helpful. Leibnizian optimism Alright, so we