# How We Run Terraform At Scale

DevFeed: [How We Run Terraform At Scale](<https://devfeed.tech/articles/how-we-run-terraform-at-scale-20127.md>)

Original publisher: [Read original article](<https://benchling.engineering/how-we-run-terraform-at-scale-da7bb75dc394?source=rss----3d4aa8fb07ea---4>)

Author: Christian Monaghan

Published: 2025-03-04T16:32:39Z

Content type: article

Language: en

Sources: [Benchling](<https://devfeed.tech/sources/benchling.md>)

Topics: [Terraform](<https://devfeed.tech/topics/terraform.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [DynamoDB](<https://devfeed.tech/topics/dynamodb.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [cloud-infrastructure](<https://devfeed.tech/tags/cloud-infrastructure.md>), [dynamodb](<https://devfeed.tech/tags/dynamodb.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [infrastructure-as-code](<https://devfeed.tech/tags/infrastructure-as-code.md>), [management](<https://devfeed.tech/tags/management.md>), [operations](<https://devfeed.tech/tags/operations.md>), [s3](<https://devfeed.tech/tags/s3.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [terraform](<https://devfeed.tech/tags/terraform.md>)

## AI overview

Benchling describes how it manages more than 165,000 cloud resources across 625 Terraform workspaces and 38 AWS accounts. The article explains that its earlier laptop-based Terraform workflow, using S3 state files and DynamoDB locks, became inefficient as the number of workspaces grew.

## Source excerpt

Managing over 165k cloud resources across hundreds of workspaces could seem daunting. But for us, it's just another day at Benchling. Here's how we do it. We currently have: 165k cloud resources under management 625 Terraform workspaces 38 AWS accounts 170 engineers (40 of whom are infra specialists) We perform: 225 infrastructure releases daily (terraform apply operations) 723 plans daily (terraform plan operations) We've been successfully operating Benchling's infrastructure release system for the past two years (spoiler, it's Terraform Cloud), over which time we've doubled our infrastructure footprint with minimal additional release overhead. Before Terraform Cloud: The Chaos Our infra release process wasn't always this smooth. Let me rewind and take you back to how it was before. As is common guidance for small Terraform projects, our team would previously apply all infrastructure changes via laptop. Also in line with common guidance, our team used S3 to store state files, with DynamoDB state locks, which prevented any apply-time collisions. This is a great strategy for a small team working on up to a dozen workspaces. However, this slowly starts to break down as the team's workspace footprint grows. It's like the proverbial frog in the pot of water, slowly heated to a boil. By the time we made the switch, Benchling was managing 350 workspaces. We were approaching the boiling point. Pain Points: Developer Toil and Inefficiency Managing 350 workspaces with this approach had several downsides: Necessitated elevated AWS access permissions for the infrastructure team. It was time-consuming as the engineer had to navigate to each directory, run terraform apply, review and approve the run, then verify it succeeded. Very commonly a single change could affect over 120 workspaces, which would mean repeating this process 120 times. (We had developed a custom python script which helped parallelize this somewhat.) Accumulated infra drift. Often an engineer would go to apply