Framework layers¶

This page follows a run through the framework, stage by stage. Design principles covers how those stages depend on each other.

The framework is split into five layers. Each one has a single job and talks to its neighbours through plain data: dictionaries, JSON files and typed result objects. That keeps any layer replaceable without touching the others.

1. Trigger intake¶

A trigger is the unit of work: one new third-party version on one axis of the matrix, plus enough context to place it.

run:
  topology: vm-host
  product_version: "4.2.0"
triggers:
  - id: ubuntu-24.04-arm64
    os: ubuntu
    os_version: "24.04"
    arch: arm64
  - id: rhel-9.4-x86_64
    os: rhel
    os_version: "9.4"
    arch: x86_64

In CI, the same triggers arrive in a compact form, one per OS, as name:version:arch, with an optional fourth field to pin an exact image:

TRIGGER_OS="ubuntu:24.04:arm64, rhel:9.4:x86_64, debian:12:x86_64:img-0a1b2c"

A small builder parses each entry into a typed OperatingSystem(name, version, arch, image) model. When the image field is empty, image resolution fills it in. When it's set, that exact image is tested, which is how a specific build gets re-checked. Cluster triggers carry more fields (Kubernetes version, node image type, install method, bastion OS), but they're parsed into typed models the same way.

Triggers come from two places:

Automatic detection: a release watcher checks for new versions on the axes that change most often, such as a new OS image version or a new version of the product's agent or CLI, and emits a trigger as soon as one appears. No one has to remember to ask.
Manual requests: specific combinations, such as a particular Kubernetes version with a given runtime and network plugin, or a customer's exact environment, are requested by an engineer or a customer-facing team.

Both produce the same structure, so everything downstream is identical. Automating detection for another axis never requires changing the rest of the framework. See Trigger sources for the detector types and how duplicates are filtered out.

Keep triggers minimal

A trigger should say what to test (OS name, version, architecture), not how. Resolving an OS name to a concrete machine image is the framework's job, not the requester's. See image resolution.

2. Configuration generation¶

This layer translates triggers into two outputs:

Terraform variables: a JSON file Terraform loads automatically (*.auto.tfvars.json), keyed by trigger ID. Terraform's native JSON support makes this a clean contract: Python writes data, and HCL never needs templating.
Test parameters: the same trigger IDs, written where the test suites can read them, so each trigger becomes its own set of parametrised test cases.

{
  "hosts": {
    "ubuntu-24.04-arm64": { "image_id": "img-0a1b2c", "size": "small-arm" },
    "rhel-9.4-x86_64":    { "image_id": "img-3d4e5f", "size": "small-x86" }
  }
}

Two triggers produce two hosts and two sets of test cases. Twenty triggers produce twenty, with no code changes.

3. Orchestration¶

The framework drives Terraform rather than a person: init, plan, apply, read outputs, and later destroy. It also checks that each step succeeded before moving on, and captures the Terraform outputs as run artefacts.

Treating Terraform as a tool the framework calls, rather than the entry point, is what lets one run own its whole lifecycle, including the decision about when to destroy.

The cloud-specific part is a registered executor. Each provider registers itself under a name, and the provisioner asks for one by name:

@TerraformExecutors.register("aws")
class AwsTerraformExecutor(TerraformExecutor):
    def build(self, workdir: Path) -> TerraformResult: ...    # init, validate, plan, apply, write outputs.json
    def destroy(self, workdir: Path) -> TerraformResult: ...


executor = TerraformExecutors.create("aws", run_config)
result = executor.build(run.workdir)

This is what "a provider is a plug-in" means in practice. Supporting another cloud is a new executor, new root modules and new image resolution. The provisioner, validation and reporting stay as they are. Run artefacts follow the same idea: a FileManager with a storage-specific implementation uploads each run's directory and finds the latest results for replay.

Each run gets its own working directory, and everything Terraform needs is generated into it as data:

File	Written by	Holds
`triggers.json`	Trigger intake	What this run is testing
`*.auto.tfvars.json`	Configuration generation	Variables for the topology's root module
`backend.tf.json`	Configuration generation	Where this topology's state lives (a per-run key is the better default)
plan file	Orchestration	The exact change Terraform will make
`outputs.json`	Orchestration	Host IDs, addresses and endpoints for the validation layer

Generating the backend configuration as well means the state location is a value the framework chooses, not something hard-coded in the module. The whole directory is uploaded with the run's artefacts, which is what makes a failed run replayable on a laptop.

4. Validation¶

Validation follows the same shape as provisioning. A validation factory for the cloud creates the right validator for the topology, and each validator works from the run's typed outputs rather than raw JSON:

validator = ValidationFactories.for_cloud("aws").create(run.topology, outputs)

Topology	Outputs model, read from `outputs.json`	What the validator starts from
VM host	`HostOutputs(instance_ids)`	One instance per trigger
Self-managed K8s	`ClusterOutputs(control_plane_id, worker_ids, bastion_id)`	Nodes reached through the bastion
Managed K8s	`ManagedClusterOutputs(cluster_name, node_group, bastion_id)`	The provider's cluster and node group (infrastructure built; validation not finished)

Validators share a small hierarchy. Common set-up, such as region, timeouts and where results go, lives in a base class, and each topology adds only what's different:

class Validation(ABC):                       # config, timeouts, results location
    @abstractmethod
    def checks(self) -> list[Check]: ...


class HostValidation(Validation):
    def checks(self):
        return [
            InstancesRunning(self.outputs.instance_ids),           # the cloud reports every host as running
            AgentServiceActive(),                              # the installed agent is up on each host
            HostDataReachedBackend(datasets=["inventory"]),  # the backend holds this host's data
            AssessmentResultsReturned(),                           # shared end-to-end suite
        ]


class SelfManagedClusterValidation(Validation):
    def checks(self):
        return [
            AllNodesReady(),                                       # every node joined and reports Ready
            AgentPodsHealthy(install=self.trigger.install_method),  # Helm or DaemonSet install came up
            HostDataReachedBackend(datasets=["inventory"]),
            AssessmentResultsReturned(),                           # the same suite as the VM host topology
        ]

Validators sit on three levels:

Level	Question it answers	Example
Infrastructure	Did the environment come up as intended?	Nodes ready, cluster reachable through the bastion
Install	Did the product install and start?	Service running, pods healthy, correct version reported
Behaviour	Does the product actually work here?	Data reaches the backend, assessments return expected results

Two smaller patterns sit underneath the checks:

Commands and executors: a check builds a command, and an executor runs it and returns a CommandResult (command, stdout, stderr, exit code). There are three executors: a local one for the CLI, a remote one that runs over SSH straight to a host, and a proxy one that opens a tunnel through the bastion to reach private nodes. The check doesn't know which one it got. It's the same command and executor split as in my BDD framework design guide.
Polling: anything that becomes true eventually (a node reporting Ready, the agent service starting) is a command run by a shared poller with an interval and a deadline. A poll succeeds when the command exits cleanly, writes nothing to stderr, and the until check passes on its output. Otherwise it times out with an error naming the command, so a stuck cluster fails with a clear reason instead of hanging.

poll(executor=bastion_proxy(outputs, node="control-plane"),
     command="kubectl get nodes --no-headers",
     until=all_nodes_ready, every=15, timeout=600)

5. Reporting¶

Reporting is a pluggable interface. Each publisher receives the same run result and decides where it goes:

Object storage: the complete record of a run: generated configuration, Terraform outputs, reports and logs.
Issue tracker: the tracking ticket is updated, and bugs are opened for failures.
Test reports: suites emit standard reports that dashboards can pick up for trends across runs.

Adding a new destination means adding a publisher, not changing the framework. See Publishing for the event model, routing rules, and documentation and chat sinks.

Running outside CI¶

Because the framework writes every artefact to object storage itself, a run doesn't depend on the CI system's own artefact handling. An engineer can rerun a single trigger locally, inspect the exact configuration and reports from a failed CI run, and debug against the same inputs.