Troubleshooting / lifecycle
Find the first broken handoff.
Diagnose from the controller outward: deployment, leadership, GitHub assignment, EC2 launch, runner registration, job execution, then termination. Changing capacity before locating the failed handoff often hides the evidence.
Record the incident start time, deployment generation, fleet name, GitHub job ID, and EC2 instance ID as soon as they are available. Use those identifiers to correlate systems without copying secrets or workflow content into a ticket.
First five minutes
- Check ECS service desired, running, and pending task counts plus the current deployment state.
- Check
/readyz, recentFleetReadytransitions, and structured CloudWatch logs for the affected fleet. - Read—but do not edit—the DynamoDB coordination item. Confirm it has one unexpired owner and the expected deployment generation.
- Inspect the GitHub job queue, matching label, App installation, and App permissions.
- List EC2 instances by Gondola deployment and fleet tags, then compare their launch, expiry, generation, and capacity tags with GitHub runner state.
Symptom index
| Symptom | Inspect first | Safe response |
|---|---|---|
| Controller task repeatedly exits | ECS stopped reason, app logs, secret access, configuration validation | Restore the last known-good digest; do not force two leaders |
/healthz works; /readyz fails | Lease generation and first failing fleet initialization step | Fix the dependency; allow the revision to prove itself |
| Job remains queued; no EC2 instance | Label, scale-set ownership, App permission, fleet maximum, launch errors | Correct the mapping or AWS constraint; do not broaden roles blindly |
| EC2 instance exists; runner stays offline | System log, image architecture, egress/DNS, instance-profile trust, JIT age | Terminate that runner; never recycle it for another job |
| Spot fleet cannot launch | Capacity error type, configured subnets, service-linked role, quotas | Use fallback policy or a reviewed alternate fleet |
| Completed runner remains online | GitHub status, expiry tag, systemd timer, termination errors | Confirm no active job, record IDs, then terminate manually |
| Upgrade rolls back | ECS deployment events and new generation’s fleet readiness | Keep the rollback; diagnose before reapplying the new digest |
Controller is alive but not ready
/healthz confirms the process and HTTP server are alive./readyz is stricter: the current generation must own or observe valid leadership readiness and every fleet must have completed its startup proof. /version identifies the running build. These endpoints are bound to the task’s local health interface; Gondola does not expose them as a public inbound service.
- If ECS cannot start a task, inspect image pull, execution-role, secret, subnet, DNS, and security-group errors.
- If the task waits on leadership during an upgrade, compare its generation with the coordination item and allow the older lease to release or expire.
- If leadership is acquired but readiness never arrives, find the first fleet whose App session, poll, scale-set initialization, or capacity reconciliation failed.
- If one fleet later degrades, use
FleetReadyand that fleet’s logs; other fleet supervisors should continue independently.
Never delete or rewrite an unexpired lease to “unstick” a deployment. If a controller exited because active work ignored cancellation, leave the lease to expire before another leader is forced.
Job is queued but no runner launches
- Confirm the workflow label exactly matches the intended GitHub scale set and that the App is installed for the repository.
- Confirm the scale set carries Gondola’s deployment ownership marker. A same-named unowned set is rejected intentionally.
- Compare assigned, active, minimum, and maximum capacity. A fleet at
max_runnersshould queue rather than over-launch. - Inspect
RunnerLaunchErrorsand the structured AWS error. Check EC2 quotas, subnet addresses, launch-template version, AMI, instance type, IAM pass-role scope, and Spot service-linked role. - Treat ambiguous EC2 request errors differently from definitive capacity errors. Gondola does not try another zone after an ambiguous result because that could create a duplicate instance.
Instance launches but does not register
Inspect the EC2 system log and boot service without placing full user data in a ticket. Verify the AMI and runner container support the fleet architecture, the instance profile trusts EC2, and the subnet can resolve DNS and reach GitHub plus the image registry over HTTPS.
JIT configuration is short-lived. A runner that missed its registration window must be terminated and replaced; it must not be repurposed for a later job. If failures repeat, temporarily route workflows to a known-good fleet or reduce the affected fleet’s maximum to stop new launches while preserving evidence.
Runner does not terminate
Each instance has an independent hard-expiry timer and terminate-on- shutdown behavior. Compare the gondola:expires-at tag, GitHub job state, controller termination logs, and EC2 state. Manually terminate only after verifying that GitHub no longer considers the job active. Retain the job ID, instance ID, generation, and timestamps for post-incident review.
Upgrade or controller failover fails
A graceful leader releases its lease; after abrupt loss, wait at least one configured lease duration before declaring failover broken. If no peer becomes ready, restore the last known-good digest in Terraform, force a new ECS deployment, wait for readiness, and run one disposable job per fleet.
Keep the ECS circuit-breaker rollback as evidence. Do not replace it immediately with the same failing image. First identify whether the new task could pull its image, read its secret, acquire generation-fenced leadership, connect every GitHub session, and reconcile each fleet.
Escalation bundle
Include the following, with credentials and workload data removed:
- Gondola version and exact controller image digest;
- AWS region, deployment name, and generation hash;
- affected fleet, architecture, capacity mode, and instance type;
- UTC incident window, GitHub job ID, and EC2 instance ID;
- ECS deployment events and redacted structured log entries;
- the first failed lifecycle stage and changes made during triage.
Do not include a GitHub App key, JIT configuration, AWS credential, Terraform state, payment-card information, repository source, or full workflow payload. Use the contact route; use the private reporting instructions for suspected vulnerabilities.