Architecture / trust map
A narrow control plane for a deliberately large trust boundary.
Gondola is installed software, not hosted runner capacity. Its job is to turn an approved GitHub runner label into one customer-owned EC2 instance, supervise that lifecycle, and then get out of the way.
Control
One active GitHub listener per deployment
Execution
One ephemeral EC2 instance per assigned job
Boundary
No inbound Gondola service or job proxy
Component map
| Component | Runs in | Responsibility | Persists |
|---|---|---|---|
| Controller | ECS Fargate | Poll GitHub, reconcile desired capacity, launch and retire runners | Structured logs only |
| Fleet catalog | ECS task definition | Bind labels to reviewed image, network, IAM, and capacity policy | Terraform state and task configuration |
| Coordination | DynamoDB | Conditional leader lease, deployment generation, and readiness bit | One small coordination item; no workflow data |
| Runner | EC2 | Consume one short-lived JIT configuration and execute one job | Nothing after instance and root volume termination |
| Credentials | Secrets Manager or SSM | Supply the GitHub App private key directly to ECS | Encrypted secret version |
| Observability | CloudWatch | Logs by default; optional bounded metrics and alarms | According to customer-selected retention |
One job, end to end
- GitHub queues a job whose self-hosted label matches a configured scale set.
- The lease-owning controller receives the assignment over an outbound long-poll session.
- Gondola resolves the scale-set name to a fixed fleet record. Workflow code cannot supply an AMI, subnet, role, or security group.
- The controller obtains a short-lived just-in-time runner configuration and requests the exact launch-template version for that generation.
- EC2 boots in a configured subnet, starts the GitHub runner, and registers for the assigned job.
- The runner executes exactly one workflow job.
- Completion, cancellation, or the hard lifetime limit starts runner retirement. The instance and its root volume terminate.
- Periodic reconciliation detects and removes stale capacity even if a normal completion signal was missed.
JIT configuration is short-lived but still sensitive. It is supplied to the instance during bootstrap and must never appear in controller logs, Terraform state, support bundles, or reusable images.
Leadership and fencing
The default ECS service runs two controller tasks across Availability Zones. Both can be alive, but a conditional DynamoDB lease permits only one to own GitHub message sessions. The active task renews the lease; the warm peer waits to acquire it after graceful release or expiry.
Leadership is fenced by deployment generation. During an upgrade, a new revision cannot inherit readiness from an older revision. It must acquire its own generation’s lease, connect every configured fleet, complete a poll, reconcile desired capacity, and publish active readiness. This creates a short, explicit control-plane maintenance window rather than pretending two generations can safely listen at once.
DynamoDB operations have deadlines shorter than the renewal interval. An independent monotonic watchdog cancels active work before the controller’s locally known lease expiry. If work cannot be proven stopped, the process fails closed and leaves the lease to expire.
Failure behavior
| Failure | Expected behavior | Operator signal |
|---|---|---|
| Active controller exits | Warm peer acquires leadership after release or lease expiry | Task replacement, readiness transition, leader log |
| New revision cannot initialize | ECS circuit breaker restores the previous task definition | Failed deployment and rollback events |
| One fleet listener fails later | Supervisor retries that fleet without blocking other fleets | FleetReady transition and structured error |
| Definitive Spot shortage | Try configured subnets; fallback fleets then use On-Demand | SpotFallbacks and capacity tag |
| Ambiguous EC2 request error | Do not retry in another zone and risk a duplicate launch | Launch error with request context |
| Runner never completes | Independent hard-expiry timer bounds instance lifetime | Expiry tag, termination signal, GitHub job state |
Network and data flows
Gondola opens no inbound internet listener. Controllers require outbound HTTPS to GitHub, AWS APIs, and the controller image registry. Runners require outbound access for GitHub and whatever registries or services their workflows use. The Terraform module attaches to existing VPCs and does not create a NAT gateway or prescribe the customer’s egress design.
- GitHub App private key: secret store to ECS task; never a Terraform value.
- Job assignment and JIT configuration: GitHub to active controller to one runner.
- Repository and workflow data: GitHub to the runner inside customer AWS.
- Operational data: controller and instance events to customer-owned CloudWatch logs and optional metrics.
- Commercial data: website and Stripe only; never required by a running controller or runner.
Deliberate constraints
- AWS and GitHub.com; GitHub Enterprise Server is not supported.
- Linux x64 and ARM64 runner fleets; no Windows or macOS runners.
- One AWS account and region per controller deployment.
- Existing VPCs and customer-selected egress only.
- Exact controller image digests and launch-template versions for reproducible generations.
- One active listener with an optional warm peer, not active/active job assignment.