DEVOPS

Quarantine a Flaky Test from CI and File a ClickUp Owner Ticket

When a CI run reports a test that passed on retry, this workflow opens a quarantine PR that skips the test for one cycle, files a ClickUp ticket assigned to the test's code owner.

CategoryDevOps
Enginesim
Difficultyintermediate
Triggerwebhook
Steps5
Setup~15 min

How it runs

The automated pipeline, trigger to output.

  • TriggerGitHub Actions workflow_run completesGitHubGitHub
  • LogicKeep only tests that failed then passed on retry
  • ActionOpen quarantine PR skipping the test for one cycleGitHubGitHub
  • ActionCreate ClickUp ticket assigned to the code ownerClickUpClickUp
  • OutputPost quarantine summary to SlackSlack

What it does

Catches tests that fail then pass on automatic retry (the classic flaky signature), removes them from the blocking suite for exactly one cycle, and makes sure a human owns the fix. The build stays green without anyone silently disabling coverage.

When to use it

Use it when intermittent failures are stalling merges and your team keeps hitting "re-run jobs" instead of fixing the root cause. It turns that reflex into an auditable quarantine with a deadline.

How it works

  1. 1A GitHub Actions webhook fires when a workflow run completes and includes retry metadata.
  2. 2A filter keeps only runs where a test failed on the first attempt but passed on retry.
  3. 3The workflow looks up the test's owner from CODEOWNERS and opens a PR adding a skip annotation tagged with the current cycle.
  4. 4It creates a ClickUp task assigned to that owner, due in one sprint, linking the PR and the failing run.
  5. 5A Slack message posts the quarantine summary to the team channel as the final delivery.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect GitHubRepos, issues, pull requests, actions.
  2. 2
    Connect ClickUpDocs + tasks + chats in one workspace.
  3. 3
    Connect SlackChannels, DMs, threads, mentions.
  4. 4
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  5. 5
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  6. 6
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.