Cost and safety¶
Ephemeral infrastructure is only cheap if it really is ephemeral. The original framework relied on the first table below: short lifetimes and small defaults. The rest of this page covers guardrails I'd recommend adding.
Defaults that keep cost low¶
| Lever | Default |
|---|---|
| Lifetime | Infrastructure exists only for the duration of a run, except failed runs paused for triage |
| Instance size | The smallest size that can run the product, per architecture |
| Disk | Small root volumes, sized to the product's needs |
| NAT gateway | Off unless a topology needs private egress |
| Cluster size | One control-plane node, with workers only as needed by the triggers |
| Region | One region per run, unless the trigger is region-specific |
Tag everything¶
Every resource carries tags for the purpose, run ID, topology and owning team. Tags make costs attributable per run and let a cleanup job find anything a crashed run left behind.
Guard the fail path¶
Pausing on failure keeps infrastructure alive on purpose, so it needs limits:
- give paused environments an expiry tag with a deadline for triage;
- run a scheduled sweeper that reports, then destroys, expired environments;
- post a reminder on the tracking ticket while an environment is still running.
Keep blast radius small¶
- Run in a dedicated test account, separate from anything customer-facing.
- Give the automation role permissions only for the resource types its topologies create.
- Allow inbound access only from the runner's IP, and only through the bastion on cluster topologies.
- Generate per-run keys and destroy them with the run.
Reproducibility¶
Every run stores its generated configuration, Terraform outputs and test reports in object storage. If you also keep plans, store them with sensitive values redacted, because plan files hold input variables in plaintext. If a run's results are ever questioned, the exact inputs are available, and the run can be repeated.