ENGINEERING

Detect Flaky Tests on CI Re-run and Open a Deflake Ticket

When a GitHub Actions job that failed on the first attempt passes on a re-run for the same commit, flag the offending tests as flaky and open a tracked Linear deflake ticket.

CategoryEngineering
Enginesim
Difficultyintermediate
Triggerevent
Steps6
Setup~15 min

How it runs

The automated pipeline, trigger to output.

  • TriggerGitHub workflow_run completedGitHubGitHub
  • LogicIs this a passing re-run of a previously failed SHA?
  • ActionDiff job logs to extract red-to-green testsGitHubGitHub
  • LogicDedupe against open deflake tickets
  • ActionCreate or update Linear deflake ticketLinearLinear
  • OutputPost flake summary to SlackSlack

What it does

It catches the classic flaky-test signature: a CI job fails, someone hits "re-run," and the second attempt passes with no code change. The workflow compares the failed and passed runs for the same commit SHA, extracts the tests that flipped, and files a Linear ticket so the flake is tracked instead of silently re-run forever.

When to use it

Use it when your team's habit is to re-run red CI until it goes green. That habit hides intermittent failures and erodes trust in the suite. This turns every "it passed the second time" into a logged, ownable task.

How it works

  1. 1A GitHub `workflow_run` completed event fires when any CI run finishes.
  2. 2A logic step checks whether this is a re-run (attempt > 1) that succeeded while a prior attempt for the same SHA failed.
  3. 3If so, an action pulls both runs' job logs and diffs the failing test names to isolate which tests flipped red-to-green.
  4. 4A logic step dedupes against open deflake tickets so repeat flakes append rather than spawn duplicates.
  5. 5A Linear action creates or updates a ticket with the test names, run URLs, and commit.
  6. 6A Slack message posts the new flake to the team channel.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect GitHubRepos, issues, pull requests, actions.
  2. 2
    Connect LinearIssues, projects, cycles, triage.
  3. 3
    Connect SlackChannels, DMs, threads, mentions.
  4. 4
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  5. 5
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  6. 6
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.