# We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform

DevFeed: [We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform](<https://devfeed.tech/articles/we-cut-cloud-waste-before-touching-cluster-sizes-lessons-from-running-a-data-platform-26516.md>)

Original publisher: [Read original article](<https://medium.com/engineering-housing/we-cut-cloud-waste-before-touching-cluster-sizes-lessons-from-running-a-data-platform-9ea96a1f9fbe?source=rss----3a69e32e2594---4>)

Author: Deepika Saini

Published: 2026-09-07T06:33:31Z

Content type: article

Language: en

Sources: [Housing.com](<https://devfeed.tech/sources/housing-com.md>)

Topics: [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [data-platforms](<https://devfeed.tech/topics/data-platforms.md>), [Platform Engineering](<https://devfeed.tech/topics/platform-engineering.md>), [migration](<https://devfeed.tech/topics/migration.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>)

Tags: [bigquery](<https://devfeed.tech/tags/bigquery.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-optimization](<https://devfeed.tech/tags/cost-optimization.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [finops](<https://devfeed.tech/tags/finops.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [migration](<https://devfeed.tech/tags/migration.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [platform-engineering](<https://devfeed.tech/tags/platform-engineering.md>)

## AI overview

This article explains how a data platform team reduced cloud costs by removing obsolete BigQuery data, adjusting Delta Lake retention, right-sizing DMS infrastructure, identifying unmonitored Databricks jobs, and standardizing pipeline onboarding and cost alerts. It reports that DMS costs were cut by over 50% and that retention was reduced from 90 days to 7 days for appropriate workloads after operational validation.

## Source excerpt

How orphaned BigQuery storage, Delta retention, DMS right-sizing, and Databricks System Tables became our biggest cloud cost wins. The biggest cloud cost optimization we made wasn't shrinking clusters.It was deleting data we'd forgotten we were paying for.Like most teams, our first instinct was to tune infrastructure first. Instead, we discovered a treasure trove of hidden costs: orphaned BigQuery datasets, 90-day Delta retention, 24-hour jobs no one monitored, and DMS infrastructure that no longer matched business needs.We stopped treating cloud bills as a finance problem and started treating them as a platform engineering problem.30-second takeaway Why deleting forgotten data saved more than shrinking clusters. How we cut DMS costs by over 50%. How Databricks System Tables exposed hidden 24-hour jobs. How config.metadata standardized pipeline onboarding. How weekly Slack alerts turned cost optimization into a habit. Section 1: Storage Was Our Biggest Leak -- We Were Paying to Store Data Nobody Used This is the most overlooked cost on many data platforms. Storage duplication across platforms We had already migrated several workloads from BigQuery to Databricks. Large datasets were still sitting in BigQuery long after they had stopped serving production workloads - quietly generating storage costs month after month. Nothing failed. No alerts fired. Every month, we paid for storage that no longer served production workloads.A migration isn't complete until the old storage is decommissioned.The hidden cost of long retention The next surprise came from Delta Lake retention settings. Our workspace was configured to retain deleted table data and transaction history for 90 days to support time travel. Time travel is incredibly useful. But did every table need three months of historical recovery? Not really. We reduced retention to 7 days for appropriate workloads after validating operational needs. What changed immediately: Less storage tied up in deleted data. Faster clea