Skip to content

Run lifecycle

This page describes the designed flow for every run.

Every run is tied to a tracking ticket from the moment a trigger arrives. The ticket is the system of record: it says what was requested, what was built, what was tested and what was decided. The framework writes to it at every stage, so nobody has to reconstruct a run after the fact.

Run lifecycleA release opens a tracking ticket, provisions a topology and runs validation. All pass: teardown and a documentation update. Any fail: bugs are opened and the run pauses with infrastructure kept for triage.Release or requestDetected version or combinationOpen tracking ticketSystem of recordProvision topologyTest tenant and infrastructureRun validation suitesResults attached to ticketAll passAny failTear downNote the listing changeMatrix updatedDocs publish the listingOpen bugs and pauseInfrastructure kept for triageTriage, then tear downResumed by a person

Stages

  1. Trigger received. A ticket is opened with the requested combination: topology, triggers and product version.
  2. Topology provisioned. A test tenant for the product's backend is created alongside the infrastructure, and the infrastructure details are added to the ticket.
  3. Validation run. Suites execute, and their reports are attached to the ticket and stored as run artefacts.
  4. Branch on the result. The run follows one of two paths.

Many triggers, one run

A run isn't limited to one trigger. Send ten new OS images and the framework writes all ten into one variables file, and one Terraform apply creates ten small hosts. Each host installs and validates the product in parallel, and each trigger gets its own result and its own matrix cell.

Many triggers, one runSeveral OS image triggers arrive together. One run writes them into one variables file and one Terraform apply, which creates one small host per trigger. Every host installs and validates in parallel, and each trigger gets its own result.OS image ATriggerOS image BTriggerOS image CTriggerOS image NTriggerOne runAll triggers in one variables fileOne Terraform applyfor_each: one host per triggerHost ASmall instanceInstall + validateIn parallelResult AOwn matrix cellHost BSmall instanceInstall + validateIn parallelResult BOwn matrix cellHost CSmall instanceInstall + validateIn parallelResult COwn matrix cellHost NSmall instanceInstall + validateIn parallelResult NOwn matrix cellTen images means ten hosts in one run, not ten runs in a queuePythonTerraformInputs and steps

The pass path

When every check passes:

  • teardown starts straight away, because nothing needs inspecting;
  • the ticket gets the results plus a note on which listing should change;
  • whoever owns the support page makes the edit and closes the ticket.

The framework never edits customer-facing documentation itself. It produces the evidence and says which listing should change, and a person publishes. That separation keeps the support page under human control while removing all the work that came before it.

The fail path

When any check fails:

  • every failing check gets its own linked bug, so each one can be triaged and owned separately;
  • the environment is left standing, so an engineer can log in to the exact host or cluster that failed, read the agent's logs, and compare them with what reached the data backend;
  • teardown waits for a person to say triage is done.

A deliberate trade-off

Keeping failed environments alive makes triage far faster: the broken host or cluster is right there. It also costs money and risks environments being forgotten. In the original design, teardown after triage was manual. Cost and safety recommends guardrails such as an expiry sweeper.

Support states

A combination moves through a small set of states. Making them searchable lets anyone answer "where are we with X?" without asking:

State Meaning
In development The topology or trigger type isn't automated yet
In test A run is in progress or paused for triage
Supported All checks passed, and the listing is published or pending
Not supported Validated and known not to work, or explicitly out of scope

"Not supported" is as valuable as "supported". A tested negative answer saves customers and support engineers from finding out the hard way.

The coverage report

Alongside the listing itself, a coverage report shows every combination the matrix claims to care about, per topology, in one of four statuses:

Coverage report (illustrative)An example grid of OS and architecture rows against three topology columns. Each cell is Passed, Ready (automation exists, not yet run), Gap (no automation yet) or N/A (combination does not exist).VM hostSelf-managed K8sManaged K8sDistro A 9 · x86_64PassedPassedGapDistro A 9 · arm64PassedReadyGapDistro B 24 · x86_64PassedPassedGapDistro B 24 · arm64PassedGapGapDistro C 3 · x86_64ReadyGapN/ADistro C 3 · arm64N/AN/AN/APassedReady: automated, not yet runGap: not automatedN/A
Status Meaning What it tells you
Passed A run validated this combination It can be listed
Ready Automation can build and test it, but it hasn't been run yet Queue a run
Gap The framework can't build or test it yet Where to invest automation next
N/A The combination doesn't exist, such as an OS with no image for that architecture Nothing to do

The two views connect simply: Passed cells are Supported, and Ready and Gap cells are In development until a run moves them to In test. The distinction that matters is Ready versus Gap. "Not tested" on its own hides whether the problem is time or missing capability. Counting each status per topology showed where coverage was strong, and where the next piece of automation would unlock the most rows.

What the ticket holds

By the end of a run, the tracking ticket contains:

  • the triggers and product version under test;
  • the topology and its infrastructure details (resolved images, sizes, versions);
  • links to the stored test reports and logs;
  • the outcome, and links to any bugs opened;
  • on a pass, a note of which listing should change.