Operate / Job and bootstrap diagnostics

Follow a job to its runner instance.

Controller v0.6.0 adds job-to-instance records and optional collection of observed bootstrap stages. Records stay in your existing CloudWatch log group and follow your configured retention policy.

Find a job or instance

Open CloudWatch Logs Insights in your deployment’s AWS region, select the module’s log_group_name, and choose a narrow time range. Replace the example instance ID with the instance you are investigating.

fields @timestamp, fleet, runner.event, runner.job_id,
       runner.workflow_run_id, runner.runner_name,
       runner.instance_id, runner.cleanup_result
| filter runner.instance_id = "i-0123456789abcdef0"
| sort @timestamp asc
| limit 100

To search by the opaque scale-set job ID instead, replace the filter with runner.job_id = "opaque-job-id". Add a workflow run ID filter when available. Gondola records the job-to-runner relationship supplied by GitHub; it does not infer it from launch order or treat scale-set job IDs as numeric GitHub REST job IDs.

Events include runner_launched, job_started, and job_completed. A completion’s termination_requested result means EC2 accepted a request; verify instance state separately. Restarts and missing events can leave gaps. Unknown identifiers are omitted.

Retain observed bootstrap stages

Use matching module 0.6.0 or later and add the following module setting. Review and apply the plan; the option is disabled by default.

bootstrap_diagnostics_enabled = true

This enables scoped EC2 console reads and records fixed bootstrap stages in the controller’s existing CloudWatch logs. It adds no guest agent or new log group. Set log_retention_days for your needs; log ingestion, storage, and queries add AWS charges.

fields @timestamp, bootstrap.fleet, bootstrap.instance_id,
       bootstrap.runner_name, bootstrap.stage, bootstrap.exit_code
| filter bootstrap.event = "bootstrap_stage_observed"
| filter bootstrap.instance_id = "i-0123456789abcdef0"
| sort @timestamp asc
| limit 100

Stages are bootstrap_started, docker_starting, docker_ready, image_pull_started, image_ready, runner_invoked, and exited. The final record contains a numeric exit code. Runner invocation does not prove GitHub registration or job execution.

Collection uses Nitro’s latest console output and is best effort. Buffering, truncation, IAM failures, teardown, and collection limits can leave missing stages. One collector tracks at most 128 instances and makes at most 12 reads per minute. Missing exit records do not prove a leaked instance. Multiple stages received together cannot establish stage durations.

Only validated stage fields are retained. Raw console output, workflow output, local log files, and runner _diag files are not uploaded. Local files disappear with the instance. Observations are diagnostic hints, not a security attestation; jobs with host access can forge console records.

Understand queue timing

When GitHub supplies valid queue and assignment timestamps, logs include queue-to-runner-assignment time. With metrics_enabled = true, the controller also emits JobQueueToRunnerAssignment using existing deployment/fleet/capacity dimensions. Missing timestamps produce no sample. Assignment time is distinct from workflow execution start.

filter runner.event = "job_started" and runner.duplicate = false
| filter ispresent(runner.job_queue_to_runner_assignment_seconds)
| stats count(*) as samples,
        pct(runner.job_queue_to_runner_assignment_seconds, 50) as p50_seconds,
        pct(runner.job_queue_to_runner_assignment_seconds, 95) as p95_seconds
  by fleet, bin(5m)

Existing JobStartLatency measures instance launch to the controller’s observation of job start, including warm-runner idle time and event delay. Review those intervals separately. Use the complete query reference in docs/operations/observability.md in the signed operator-guide bundle, and see operations for alarms and cleanup.