Testing the framework

A framework that proves other software works needs its own evidence too. The aim isn't a high test count. It's a small set of tests that each answer a real question, with the support matrix run itself as the end-to-end test.

The levels

Level	Question it answers	Runs	Cost
Static checks	Is the code well formed, and does the plan break any rules?	Every commit	Seconds, no cloud
Configuration tests	Do these triggers produce exactly the Terraform variables expected?	Every commit	Seconds, no cloud
Contract tests	Does each module output what the next module expects?	Every commit	Seconds, no cloud
Module integration	Does a changed module actually create working resources?	When a module changes	Minutes, small cloud cost
Support matrix run	Does the product work on this topology with these triggers?	Every trigger	The real run

Static checks

terraform fmt, terraform validate and a linter catch typos and invalid configuration before anything reaches a cloud API. Policy rules run against the plan as well, and they're where cost and safety become enforceable:

• instance sizes must come from an allowed list;
• every resource must carry the run ID and an expiry tag;
• no security group may allow inbound traffic from 0.0.0.0/0.

Linters built for Terraform catch provider-specific mistakes, such as invalid instance types or deprecated arguments, that validate alone won't flag.

Configuration tests

The configuration layer turns triggers into Terraform variables. It's pure data in and data out, so it can be tested exactly, with no mocks:

def test_two_triggers_make_two_hosts():
    triggers = [Trigger("ubuntu", "24.04", "arm64"), Trigger("rhel", "9.4", "x86_64")]
    config = build_host_config(triggers, resolver=FixedImages({...}))

    assert set(config["hosts"]) == {"ubuntu-24.04-arm64", "rhel-9.4-x86_64"}
    assert config["hosts"]["ubuntu-24.04-arm64"]["size"].endswith("arm")

These tests check real properties: how many hosts, which keys, which sizes. They don't check that a function was called.

Contract tests

Modules depend on each other's outputs. The compute module needs a subnet from networking, and the cluster needs the bastion's address. A contract test runs plan against a composed root and asserts that each output feeds the input that expects it. It catches the classic failure where one module renames an output and another module silently gets null.

Module integration

When a shared module changes, apply just that module in a sandbox account (Python helpers exist that wrap terraform init/apply/output/destroy for use inside pytest), check that the resource really exists and behaves (the subnet routes traffic, the bastion accepts SSH from the runner), then destroy it. This happens before the module is used in real runs.

The support matrix run as end-to-end test

Every real run is already a full end-to-end test of the framework: it provisions, installs, validates, publishes and tears down. Two habits make that evidence count:

• A canary trigger: one known-good combination that runs on a schedule. If it fails, the framework is broken, not the product.
• Every escaped bug becomes a check: when a support issue reaches a customer, the missing check is added to the suite. Over time, knowledge that lived in one person's head becomes part of the matrix.

Tests worth keeping

Keep	Retire
Tests that assert real outputs: host counts, keys, sizes, wiring	Tests that only prove a mock was called
One canary run on a schedule	Many near-duplicate end-to-end runs
Policy rules that block cost and security mistakes	Rules nobody remembers the reason for
A check for every bug that reached a customer	Tests kept only to raise a coverage number

Tests that no longer earn their place should be marked for removal, not kept running out of habit.