Skip to content

A dark server rack lit green, with most of its lights off

Decommissioning Dev Clusters

We had more development and QA environments than teams to use them: 13 Kubernetes clusters, each split into shards, and each costing thousands a month. I led the work to retire the ones nobody needed, together with the platform and SRE teams. Removals had been waiting on a platform team that couldn't keep pace, so my team took the work on.

The outcome: 8 of the 13 clusters retired. Spend on those clusters fell by more than 99% within three months, and the GitOps controller had about 1,000 fewer applications to manage, which improved its performance for every other team.

Creating a cluster is a well-trodden path. Removing one isn't. These clusters had been built at different times by different methods, so the process had to be worked out and written down as we went. This page is that process, condensed.

Five stages of decommissioningChoose with evidence, announce, wind down, tear down in order, then prove it is gone and report the savings.ChooseScorecard + ownersAnnounceDates and scopeWind downNodes to zeroTear downIn order, by ownerProveSweep and reportCost falls at wind down; the rest makes sure it stays downEvidencePeopleInfrastructure

1. Choose with evidence

Proposing to remove a development environment gets pushback, and fair enough: nobody wants their environment to vanish. Evidence turns that into a conversation about facts. For each candidate cluster we built a scorecard:

Signal Where it came from What it tells you
Cost Daily cost filtered by the cluster tag, with a projection for next month What keeping it costs
Usage Login snapshots from every shard, restored side by side Unique logins per day, and how many teams were still using each cluster
Health Error counts and pod churn from daily log queries over four weeks, compared with a baseline Whether it's healthy, or quietly broken and ignored
Data size Snapshot sizes How much real work lives there
Version drift Product versions running on each cluster, compared with the latest release How far behind it was, and whether anyone was still deploying to it
Owners Polls and direct questions What the data can't say

The usage analysis never touched the clusters. Snapshots were restored into local containers and queried there, so it needed no cluster access and put no load on anything still in use.

The first batch was chosen from polls, activity checks and a shared spreadsheet. As the obvious candidates ran out, I turned the scorecard into a small CLI that produced all four reports for any list of clusters, so any review could be rerun on demand.

The scorecard also told us when to stop. Once the remaining clusters were genuinely in use, decommissioning paused, and the tooling stayed ready for the next review.

Somewhere to go

Retiring shared clusters only works if teams have somewhere else to work. In parallel, we were moving teams to local Kubernetes development with Tilt: services ran and were tested on the developer's own machine, and we tracked adoption across repositories to see which teams had moved. Every team that moved needed a shared cluster a little less. For a hands-on example of local Kubernetes with Tilt, see Implementing Observability with Prometheus, VictoriaMetrics, and Tilt.

2. Announce

Before anything changed, the owning teams got a notice with a date and a plain list of what would go: buckets, volumes and instances, each with what to back up. Nobody should find out by losing access.

3. Wind down

The first technical step was to power the nodes down, not delete anything. Cost drops straight away, and anyone who still needs the cluster finds out while it can still be brought back. The silence that follows is the final confirmation.

4. Tear down in order

Teardown runs top down, and each step has one owning team. Every change of lane in the diagram is a handoff, and the next team doesn't start until the step before is done. The runbook was written with the SRE and platform engineers, and signed off by the platform team.

Teardown runbookEight steps in three lanes, one per owning team: release engineering, compute platform and storage platform. Each step shows how you know it is done, and each waits for the one before it.Before step 1: owners told, nodes powered down, data date agreedRelease engineeringCompute platformStorage platform1. Finalizers presentReport covers every app2. Platform apps deletedGitOps PR merged3. System apps deletedOnly teardown apps left4. Buckets, databasesDeleted on the agreed date5. Shared services downNothing depends on them6. Cluster destroyedAtlantis apply succeeds7. External data deletedStreams, warehouse gone8. GitOps root removedApp-of-apps deletedEach step waits for the one before it; a change of lane is a handoff

Finalizers come first. Applications are removed through GitOps (Argo CD), and finalizers make that deletion cascade to the stateful sets and volume claims underneath. Without them, deleting an application leaves its storage behind. A short script listed every application on the cluster and reported which ones were missing finalizers, so they could be added before anything was deleted.

Infrastructure goes through review. The cluster itself was destroyed with Terraform through pull requests, using Atlantis: the plan is posted on the PR, reviewed and approved, and only then applied. The first change for each cluster was to its Atlantis project config, so the removal started as a reviewed PR like any other. Removing a cluster is as visible as creating one.

Gotchas

Teardown never went exactly to plan. Each of these went into the runbook the first time it bit.

Teardown gotchasFour problems hit during teardown, each with what you see, why it happens and what we did: versioned buckets that would not delete, a volume that blocked the Terraform destroy, applications stuck deleting, and applications with no exclusion option.Versioned buckets wouldn't deleteYou seeBucket looks empty, delete still failsWhyOld versions and delete markers remainWe didExpiry rule, then wait for it to runA volume blocked the destroyYou seeTerraform delete stalls on a volumeWhyA system app had no finalizerWe didReviewed the volumes, deleted by handApps stuck in “deleting”You seeArgo never finishes removing the appWhyIts resources drifted from GitWe didSync the app with pruneNo way to exclude an appYou seeApp can't be excluded from the clusterWhyIts chart has no exclusion settingWe didSet enabled: false in the GitOps configDataTerraformGitOps

Buckets were the long pole. Deleting the objects in a versioned bucket only adds delete markers, so the old versions stay and the bucket can't be removed. A lifecycle rule to expire them works, but it runs in the background and in its own time, so we waited, checked and came back. It was the most manual part of the process. Start bucket expiry as early as the agreed date allows, so it isn't the last thing holding a cluster open.

5. Prove it's gone, and report

A cluster isn't decommissioned when the destroy finishes. It's decommissioned when nothing it owned is still being billed. Each cluster was tracked to a clean finish:

  • a sweep for orphaned resources (unattached volumes, idle load balancers, leftover clusters), each with an estimated monthly cost;
  • cost by cluster tag, watched until it reached zero;
  • a cleanup record per cluster, so every leftover had an owner and a date.

The reported figure was the drop in spend on the retired clusters, month by month.

Looking back

  • Build the scorecard first. The early picks relied on polls and spreadsheets. The tooling came later, and would have made the first round faster and easier to defend.
  • Require finalizers when apps are created, not as a check at teardown. A missing finalizer is cheap to add on day one and expensive to discover on the last day.
  • Write the runbook as you go. Every cluster taught us something. Writing it down each time is what made the next one quicker and more complete.