Operate / Runbook
Operate Gondola
Readiness is generation-scoped. A controller revision becomes healthy only after it owns the lease, validates every GitHub session, completes a poll, and reconciles capacity for every fleet.
Health is layered
| Signal | What it proves | What it does not prove |
|---|---|---|
/healthz | The process and local HTTP server are alive | Leadership, GitHub connectivity, or fleet operation |
/readyz | The current controller generation completed startup proof | That every later job launch will succeed |
/version | The running build identity | That the infrastructure plan points at the intended digest |
FleetReady | The current per-fleet listener and reconciliation state | That workflow code itself will pass |
| Disposable job | Assignment, launch, registration, execution, and termination | Every workload-specific dependency or permission |
The HTTP endpoints are local controller interfaces used by ECS health checks. Gondola requires no public inbound endpoint. Treat an ECS task in RUNNING state as liveness, not readiness.
Signals to monitor
Structured CloudWatch logs and cost tags are always present. Custom metrics and alarms are opt-in because they add recurring CloudWatch charges.
| Signal | What it answers |
|---|---|
FleetReady | Is each fleet’s current listener and reconciliation path operational? |
AssignedJobs / CompletedJobs | Are queued assignments becoming completed work? |
ActiveRunners / RunnerLaunches | How much capacity exists and how often the fleet creates it. |
RunnerLaunchLatency / JobStartLatency | Launch-call time and instance-launch-to-job-start time, respectively. Neither measures the full GitHub queue interval. |
RunnerLaunchErrors | Are EC2, subnet, quota, image, or permission failures preventing capacity? |
RunnerTerminationErrors | Are disposable runners being retired cleanly? |
RunnerReconcileErrors | Is periodic desired-versus-actual cleanup failing? |
SpotFallbacks | How often is the fleet moving from Spot to On-Demand? |
Metrics use deployment and fleet dimensions; capacity-specific events also identify Spot or On-Demand. Publishing is asynchronous and bounded so a CloudWatch slowdown cannot block runner reconciliation. Whenmetrics_enabled and alarms_enabled are true, the module creates a readiness alarm plus launch, termination, and reconcile error alarms for each fleet. Notification actions remain customer-selected.
An alarm without a working notification destination is not an on-call process. Choose a recipient, confirm delivery with that person, and record who responds. Add the entitlement deadline and renewal procedure to the same operating handoff.
Job correlation and bootstrap stages
Controller v0.6.0 adds job-to-instance lifecycle records and optional retention of observed bootstrap stages. Use the CloudWatch query guide to follow an instance, investigate startup, and compare queue timing. Enable bootstrap collection explicitly in the matching module.
Ephemeral diagnostic limits
Controller logs are in CloudWatch. Runner bootstrap output is available on the instance and EC2 console, but the full runner diagnostic files are not automatically retained outside the disposable instance. Plan customer-owned collection and redaction for workflows that require it; do not assume those files survive termination. Avoid sending private keys, tokens, JIT configuration, Terraform state, or raw workflow output in a support request.
Operating cadence
| Cadence | Review |
|---|---|
| Continuously | Controller readiness, per-fleet errors, queue delay, and non-terminating instances |
| Weekly | Spot fallback, EC2/EBS spend, log ingestion, stale GitHub runners, and quota headroom |
| Each release | Digests, SBOM/checksums, Terraform or OpenTofu plan, one acceptance job per fleet, rollback path |
| Each credential rotation | New secret version, generation nonce, fleet readiness, old-key revocation, audit events |
| Quarterly | Controller loss, coordination-table recreation, key rotation, rollback, and full runner cleanup |
Upgrade procedure
Start from the exact controller-and-module pair published on the installation page. Gondola does not publish a mutable latest runtime path: an upgrade changes reviewed, immutable references and produces a visible infrastructure plan.
- Select the new controller by exact OCI digest and keep the previous digest available.
- Review
terraform planfor the intended task-definition and launch-template generation. - Schedule a short new-job scheduling window, then apply. Already-running EC2 jobs continue.
- Wait for the new generation to acquire leadership and prove every fleet.
- Run a disposable job on each fleet and verify runner termination.
A new generation does not borrow the old generation’s readiness. New GitHub jobs may queue briefly while leadership hands off; already running EC2 jobs do not depend on the controller revision and continue. Terraform and OpenTofu wait for ECS steady state by default.
If the new generation cannot initialize, the ECS deployment circuit breaker can restore the prior task definition. To reconcile the state intentionally, restore the previous digest and apply it with the same engine.
Incident triage
- Check ECS desired, running, and pending task counts plus deployment state.
- Inspect readiness, recent per-fleet transitions, and structured logs.
- Confirm the DynamoDB coordination item has one unexpired owner. Do not edit a live lease.
- Inspect GitHub queue/App permissions and matching tagged EC2 instances.
- Check Spot capacity, subnet addresses, service quotas, Secrets Manager, DynamoDB, and GitHub API errors before changing fleet policy.
Preserve the first error and avoid broad policy changes during triage. The deployment name, generation, fleet, GitHub job ID, EC2 instance ID, and UTC timestamps are the safest cross-system correlation keys.
A runner also has an independent hard-expiry timer. Manually terminate an instance only after confirming its job is no longer active, and retain the instance and job identifiers for review.
Rotate the GitHub App key
Create a new key and Secrets Manager version. When the ARN remains unchanged, set a new deployment_generation_nonce so ECS starts tasks that read the new value. Verify every fleet on that generation before revoking the old key.
- Create a new App key without revoking the old key.
- Store it as a new secret version and record the rotation event.
- Change
deployment_generation_noncewhen the ARN stays the same, then apply with Terraform or OpenTofu. - Verify readiness and a disposable job on every fleet.
- Revoke the old App key and verify no task still references it.
Recovery responsibilities
| State | Recovery source | Action |
|---|---|---|
| Infrastructure state | Protected, versioned customer backend | Restore before any apply; never reconstruct by guesswork |
| Controller image | Signed release digest or customer ECR copy | Pin the last known-good digest |
| GitHub App key | Customer rotation procedure | Issue a new key when integrity is uncertain |
| Coordination table | No backup required | Recreate empty; never restore an old lease |
| Runner instances | No recovery source | Treat as disposable; terminate or allow hard expiry |
| GitHub scale set | Known infrastructure/runtime configuration | Reconcile by owned name or deliberately recreate after drain |
Decommission
- Drain queued work and scale the ECS service to zero.
- Delete each Gondola-owned GitHub scale set with the explicit deletion command and matching deployment ID.
- After jobs drain, retain any needed cache data and explicitly empty enabled cache buckets. Populated buckets block destruction. See cache retention and cleanup.
- Run
terraform destroyortofu destroywith the exact state for this deployment. - Verify no tagged EC2/EBS resources, cache bucket, secret, table, log group, or scale-set registration remains.