ENGINEERING

Auto-quarantine tests that fail then pass on rerun

When a GitHub Actions job fails and a retry of the same commit passes, it identifies the offending test, opens a PR that adds a skip/quarantine annotation.

CategoryEngineering
Enginesim
Difficultyadvanced
Triggerevent
Steps5
Setup~25 min

How it runs

The automated pipeline, trigger to output.

  • TriggerGitHub workflow_run completed eventGitHubGitHub
  • LogicDetect fail-then-pass retry on the same commit
  • ActionOpen PR adding the quarantine annotationGitHubGitHub
  • ActionFile Linear issue to track the fixLinearLinear
  • OutputNotify code owner in Slack with the PRSlack

What it does

Turns the classic 'failed, hit retry, went green' signal into automatic quarantine. It compares the failing and passing attempts of the same workflow run, isolates the test that flipped, and proposes a code change that marks it as quarantined so it stops blocking unrelated PRs while a fix is pending.

When to use it

Use it on busy trunk-based repos where a single intermittent test can stall every merge. Best when your test framework supports a skip or quarantine annotation and you want a human to still approve the PR.

How it works

  1. 1A GitHub workflow_run completion event triggers the flow.
  2. 2A logic step checks whether an earlier attempt failed while a later attempt of the same SHA passed; non-matching events stop here.
  3. 3It diffs the attempt results to pin down the exact test that flipped.
  4. 4It opens a GitHub PR adding the quarantine annotation and a link back to the evidence.
  5. 5It files a Linear issue to track the underlying fix and links the PR.
  6. 6It pings the code owner in Slack with the quarantine PR for review.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect GitHubRepos, issues, pull requests, actions.
  2. 2
    Connect LinearIssues, projects, cycles, triage.
  3. 3
    Connect SlackChannels, DMs, threads, mentions.
  4. 4
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  5. 5
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  6. 6
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.