# CI Doesn't Need On-Demand: Moving Our Build Pipelines to Spot Instances

DevFeed: [CI Doesn't Need On-Demand: Moving Our Build Pipelines to Spot Instances](<https://devfeed.tech/articles/ci-doesn-t-need-on-demand-moving-our-build-pipelines-to-spot-instances-24037.md>)

Original publisher: [Read original article](<https://engineering.razorpay.com/ci-doesnt-need-on-demand-moving-our-build-pipelines-to-spot-instances-6fff1cd92ba8?source=rss----6407ad2e59af---4>)

Author: Yuvraj Singh Singhel

Published: 2026-08-05T15:07:35Z

Content type: article

Language: en

Sources: [Razorpay](<https://devfeed.tech/sources/razorpay-engineering-medium.md>)

Topics: [CI/CD](<https://devfeed.tech/topics/cicd.md>), [self-hosted runner](<https://devfeed.tech/topics/self-hosted-runner.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>)

Tags: [artifacts](<https://devfeed.tech/tags/artifacts.md>), [aws](<https://devfeed.tech/tags/aws.md>), [build](<https://devfeed.tech/tags/build.md>), [builds](<https://devfeed.tech/tags/builds.md>), [ci](<https://devfeed.tech/tags/ci.md>), [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [costs](<https://devfeed.tech/tags/costs.md>), [github](<https://devfeed.tech/tags/github.md>), [github-actions](<https://devfeed.tech/tags/github-actions.md>), [job](<https://devfeed.tech/tags/job.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [node](<https://devfeed.tech/tags/node.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [pod](<https://devfeed.tech/tags/pod.md>), [pods](<https://devfeed.tech/tags/pods.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [retries](<https://devfeed.tech/tags/retries.md>), [runner](<https://devfeed.tech/tags/runner.md>), [self-healing](<https://devfeed.tech/tags/self-healing.md>)

## AI overview

Razorpay reports running 80% of its GitHub Actions CI workloads on AWS Spot instances, with a 60% reduction in CI compute spend and a 99.2% job success rate. Its Kubernetes recovery pipeline uses GitHub webhooks to detect failed jobs, classifies likely spot terminations by checking for a runner registration log line, retries jobs through the GitHub API, and periodically removes stale runner pods. The log check was a quick alternative to full AWS interruption-event correlation, but it broke after a runner update; metrics and safeguards now help catch divergence and prevent broad cleanup deletions. The account also describes histogram metrics to control Prometheus cardinality, idempotency keys and distributed locking to prevent duplicate retries, and exponential backoff with jitter and GitHub rate-limit handling to contain retry storms. The text notes that current classification can retry permanent failures and identifies finer failure classification, cost attribution, and multi-cluster support as future work.

## Source excerpt

Contributors: Guptaanuj CI/CD pipelines have always had a money-vs-stability problem. Run on-demand AWS instances and your build infrastructure is rock solid, expensive, and predictable. Run on spot instances and your costs drop 70-90%, but AWS can pull the rug with 2 minutes of warning. For most teams, this is a false choice. Either pay full price for reliability, or save money and accept that builds will fail in ways nobody can debug. At Razorpay, we stopped accepting that trade-off. We built a self-healing infrastructure layer for GitHub Actions on Kubernetes that runs 80% of our CI workloads on spot instances while maintaining a 99.2% job success rate. When AWS terminates a spot node mid-build, our system detects it, retries the job, cleans up the orphaned pods, and the developer never knows. This is the story of what we built, why polling wasn't an option, and the war stories that taught us how to do retries without burning everything down. The Problem With Spot Instances On CI Spot instances are AWS capacity that nobody else wants right now. The pricing is brutal compared to on-demand: a c5.2xlarge that costs around $0.34/hour on-demand drops to roughly $0.08/hour on spot. For workloads like CI/CD, where jobs are short-lived and parallelizable, the math is obvious. The catch is in the contract. AWS reserves the right to take spot capacity back at any moment, with a 2-minute warning. That works for some workloads. For others, it's catastrophic. GitHub Actions runners on Kubernetes is the hard case. A typical CI job goes like this: GitHub assigns the job to a runner. The runner is a pod on a Kubernetes cluster running on an AWS Spot instance. The job downloads dependencies, runs tests, builds artifacts. The runner reports back to GitHub. Now insert a spot termination at minute 4 of a 7-minute build. What happens? The runner pod dies mid-job. GitHub never gets a completion signal; the job hangs until timeout, then marks as "failed". A new runner pod gets schedule