MajorResolved

Pipeline jobs queued longer than usual

Started
Resolved
Duration
2 h 45 min

Affected components

Write-up

Impact

For 2 hours 45 minutes, about one in five pipeline jobs on hosted Linux runners waited more than ten minutes to start. Jobs on self-hosted runners were not affected, and no job results were lost.

Timeline

  1. 14:02 UTC — queue times begin to rise in one region

  2. 14:19 UTC — alert fires; investigation starts

  3. 15:10 UTC — a faulty runner image is identified and rolled back

  4. 16:47 UTC — queue times back to normal

What caused it

A new runner image failed its health check after boot, so runners were created and torn down in a loop instead of taking jobs.

What we changed

Runner images now roll out to one region at a time and stop automatically when boot health checks fail.

Updates

  1. Resolved

    Queue times are back to normal. A full write-up is attached to this incident.

  2. Monitoring

    The rollback is complete and queued jobs are starting. We are watching queue times recover.

  3. Identified

    We have traced the delay to a runner image released this afternoon and are rolling it back.

  4. Investigating

    We are investigating longer than usual queue times for pipeline jobs on hosted runners.