Operations
Prove the control plane before trusting it.
Readiness is generation-scoped. A controller revision becomes healthy only after it owns the lease, validates every GitHub session, completes a poll, and reconciles capacity for every fleet.
Health is layered
| Signal | What it proves | What it does not prove |
|---|---|---|
/healthz | The process and local HTTP server are alive | Leadership, GitHub connectivity, or fleet operation |
/readyz | The current controller generation completed startup proof | That every later job launch will succeed |
/version | The running build identity | That Terraform points at the intended image digest |
FleetReady | The current per-fleet listener and reconciliation state | That workflow code itself will pass |
| Disposable job | Assignment, launch, registration, execution, and termination | Every workload-specific dependency or permission |
The HTTP endpoints are local controller interfaces used by ECS health checks. Gondola requires no public inbound endpoint. Treat an ECS task in RUNNING state as liveness, not readiness.
Signals to monitor
Structured CloudWatch logs and cost tags are always present. Custom metrics and alarms are opt-in because they add recurring CloudWatch charges.
| Signal | What it answers |
|---|---|
FleetReady | Is each fleet’s current listener and reconciliation path operational? |
AssignedJobs / CompletedJobs | Are queued assignments becoming completed work? |
ActiveRunners / RunnerLaunches | How much capacity exists and how often the fleet creates it. |
RunnerLaunchLatency / JobStartLatency | Whether cloud capacity or runner registration is delaying jobs. |
RunnerLaunchErrors | Are EC2, subnet, quota, image, or permission failures preventing capacity? |
RunnerTerminationErrors | Are disposable runners being retired cleanly? |
RunnerReconcileErrors | Is periodic desired-versus-actual cleanup failing? |
SpotFallbacks | How often is the fleet moving from Spot to On-Demand? |
Metrics use deployment and fleet dimensions; capacity-specific events also identify Spot or On-Demand. Publishing is asynchronous and bounded so a CloudWatch slowdown cannot block runner reconciliation. Whenmetrics_enabled and alarms_enabled are true, Terraform creates a readiness alarm plus launch, termination, and reconcile error alarms for each fleet. Notification actions remain customer-selected.
Operating cadence
| Cadence | Review |
|---|---|
| Continuously | Controller readiness, per-fleet errors, queue delay, and non-terminating instances |
| Weekly | Spot fallback, EC2/EBS spend, log ingestion, stale GitHub runners, and quota headroom |
| Each release | Digests, SBOM/checksums, Terraform plan, one acceptance job per fleet, rollback path |
| Each credential rotation | New secret version, generation nonce, fleet readiness, old-key revocation, audit events |
| Quarterly | Controller loss, coordination-table recreation, key rotation, rollback, and full runner cleanup |
Upgrade procedure
- Select the new controller by exact OCI digest and keep the previous digest available.
- Review
terraform planfor the intended task-definition and launch-template generation. - Schedule a short new-job scheduling window, then apply. Already-running EC2 jobs continue.
- Wait for the new generation to acquire leadership and prove every fleet.
- Run a disposable job on each fleet and verify runner termination.
A new generation does not borrow the old generation’s readiness. New GitHub jobs may queue briefly while leadership hands off; already running EC2 jobs do not depend on the controller revision and continue. Terraform waits for ECS steady state by default.
If the new generation cannot initialize, the ECS deployment circuit breaker can restore the prior task definition. To reconcile Terraform intentionally, restore the previous digest and apply it.
Incident triage
- Check ECS desired, running, and pending task counts plus deployment state.
- Inspect readiness, recent per-fleet transitions, and structured logs.
- Confirm the DynamoDB coordination item has one unexpired owner. Do not edit a live lease.
- Inspect GitHub queue/App permissions and matching tagged EC2 instances.
- Check Spot capacity, subnet addresses, service quotas, Secrets Manager, DynamoDB, and GitHub API errors before changing fleet policy.
Preserve the first error and avoid broad policy changes during triage. The deployment name, generation, fleet, GitHub job ID, EC2 instance ID, and UTC timestamps are the safest cross-system correlation keys.
A runner also has an independent hard-expiry timer. Manually terminate an instance only after confirming its job is no longer active, and retain the instance and job identifiers for review.
Rotate the GitHub App key
Create a new key and Secrets Manager version. When the ARN remains unchanged, set a new deployment_generation_nonce so ECS starts tasks that read the new value. Verify every fleet on that generation before revoking the old key.
- Create a new App key without revoking the old key.
- Store it as a new secret version and record the rotation event.
- Change
deployment_generation_noncewhen the ARN stays the same, then apply Terraform. - Verify readiness and a disposable job on every fleet.
- Revoke the old App key and verify no task still references it.
Recovery responsibilities
| State | Recovery source | Action |
|---|---|---|
| Terraform state | Protected, versioned customer backend | Restore before any apply; never reconstruct by guesswork |
| Controller image | Signed release digest or customer ECR copy | Pin the last known-good digest |
| GitHub App key | Customer rotation procedure | Issue a new key when integrity is uncertain |
| Coordination table | No backup required | Recreate empty; never restore an old lease |
| Runner instances | No recovery source | Treat as disposable; terminate or allow hard expiry |
| GitHub scale set | Known Terraform/runtime configuration | Reconcile by owned name or deliberately recreate after drain |
Decommission
- Drain queued work and scale the ECS service to zero.
- Delete each Gondola-owned GitHub scale set with the explicit deletion command and matching deployment ID.
- Run
terraform destroywith the exact state for this deployment. - Verify no tagged EC2/EBS resources, secret, table, log group, or scale-set registration remains.