Configure / Capacity policy

Match capacity to demand.

Choose EC2 alternatives, schedule warm runners, or let the adaptive optimizer adjust warm capacity from observed queue delays. Add a daily allowance to control new launches. Each policy is optional.

Adaptive capacity optimizer · Daily launch allowances · Instance alternatives · Warm schedules

Approve instance alternatives

Merge an ordered list into your existing fleet. Review each type’s architecture, AMI compatibility, CPU, memory, local storage, EBS and networking limits, quotas, regional availability, and price.

fleets = {
  linux-build = {
    scale_set_name            = "gondola-linux-build"
    architecture              = "x64"
    instance_type             = "m7i.large"
    instance_type_alternatives = ["m7a.large", "m6i.large"]
    capacity_mode             = "spot-with-on-demand-fallback"
    subnet_ids                = var.approved_runner_subnets
    min_runners               = 0
    max_runners               = 8
  }
}

Gondola tries the primary type across the configured subnets, then alternatives in the listed order. It advances after a definitive AWS capacity rejection. Ambiguous responses, authorization failures, and configuration errors stop the attempt. Spot-only fleets remain Spot; Spot-with-On-Demand fallback exhausts approved Spot candidates before trying approved On-Demand candidates.

The limit is eight distinct types including the primary and 32 type/subnet combinations per fleet. All types must match the fleet’s architecture. An empty list preserves single-type behavior. Legacy single-fleet installations use runner_instance_type_alternatives.

Selected types, subnets, market, and fallback decisions appear in controller logs. Gondola does not automatically discover cheaper types or switch regions. This launch fallback does not retry a job interrupted after launch. Changes affect future launches, including fleets using S3 caching.

Schedule warm runners

The following window maintains a minimum of two available runners from 12:00 to 22:00 UTC on weekdays, subject to the fleet’s maximum and normal reconciliation. Use the same fleet key as above.

warm_windows = {
  linux-build = [
    {
      days             = ["mon", "tue", "wed", "thu", "fri"]
      start_utc        = "12:00"
      duration_minutes = 600
      min_runners      = 2
    }
  ]
}

Use default as the key for a legacy single fleet. Each fleet accepts up to 16 windows, one through seven unique lowercase weekday names, a UTC HH:MM start, and a duration of 15 through 1,440 whole minutes. The minimum must be between zero and the fleet’s max_runners. Review and apply your Terraform/OpenTofu plan.

Times are always UTC; daylight-saving changes are not applied. Windows can cross midnight or the end of the week. Start is inclusive and end is exclusive. Overlapping windows take the highest minimum and never lower the fleet’s ordinary minimum. Assigned jobs are added to that minimum, capped by max_runners.

Plan for launch time and wind-down

Set windows early enough for EC2 launch and runner registration. They take effect on the next successful reconciliation; an unavailable controller or GitHub connection cannot launch capacity. Logs show effective-minimum changes. With metrics enabled, configured fleets also emit MinimumRunners.

Budget for the higher minimum and delayed wind-down. Alternative types and minimum windows are not spending caps or guarantees of shorter queues. See the AWS cost model and docs/capacity.md in the signed operator-guide bundle.

Adaptive capacity optimizer

Available in controller v0.8.2 with module 0.8.1. Gondola adjusts warm replenishment from GitHub queue-to-assignment delays. It keeps your configured instance types, images, and permissions.

Add this map inside your Gondola module block, using the same fleet key as above. It allows up to two adaptive warm runners and targets a 30-second queue delay, with adjustments no more often than every five minutes.

adaptive_warm_pools = {
  linux-build = {
    max_runners = 2
    target_queue_seconds = 30
    adjustment_interval_minutes = 5
  }
}

The target starts at zero. With at least ten fresh samples, Gondola compares the 90th-percentile delay with your target. It adds one warm runner when the delay exceeds the target and removes one from the target when the delay falls below half. Between those thresholds, it holds.

The effective minimum is the highest of the fleet’s min_runners, its active schedule, and its adaptive target, capped by the fleet’s overall max_runners. Fixed and scheduled minimums can exceed the adaptive maximum. The weekday schedule above therefore keeps the minimum at two during its window.

All three values are required whole numbers. Lower targets stop replenishing excess capacity; they do not terminate existing runners. Learning resets when the listener or controller restarts. After three intervals without valid observations, the target decreases by one per interval toward zero. The queue target is a tuning goal, not a latency guarantee.

Daily launch allowances

Enable an allowance independently of the optimizer. Add this map inside the module block and set max_runner_lifetime = "2h" in the existing linux-build fleet. Replace the example hourly rate with an estimate for your approved instances and region.

fleet_budgets = {
  linux-build = {
    daily_limit_usd = 20
    runner_hourly_rate_usd = 0.10
  }
}

Before every launch, including warm replenishment, Gondola reserves the hourly estimate multiplied by the maximum runner lifetime. This example reserves $0.20 per runner, allowing up to 100 admissions per UTC day. Short jobs and failed or uncertain launches do not refund reservations.

When the allowance runs out, new launches wait; existing jobs continue under their normal lifecycle. A new UTC day opens a new allowance. Ledger errors also block launches until accounting recovers. Fixed, scheduled, and adaptive minimums cannot bypass the allowance.

This is an estimated launch allowance, not an AWS billing cap. Infrastructure charges, actual instance prices, and runners admitted on previous days can add costs. Both amounts must be positive, at most $1,000,000, and use no more than six decimal places. See the AWS cost model.

Enabling an allowance creates a DynamoDB ledger with storage, request, and recovery charges. Usage survives controller restarts and failover. Keep the table and fleet identities to preserve accounting. Removing every fleet_budgets entry plans deletion of the table.

Both adaptive_warm_pools and fleet_budgets default to {}. Omit a fleet to leave that policy off; use default for a legacy single fleet. Review and apply your Terraform or OpenTofu plan to enable them. Policy decisions appear in controller logs. Optional metrics report adaptive capacity, remaining allowance, blocked launches, and admission errors.