ENGINEERING
GitLab Flaky-Test Quarantine Bot
Watches GitLab CI pipeline failures, counts how often each spec has failed across recent runs, and when a test crosses the repeat-failure threshold it auto-labels it.
How it runs
The automated pipeline, trigger to output.
- TriggerGitLab pipeline failed on default branchGitLab
- ActionFetch JUnit artifact and parse failed testsGitLab
- ActionIncrement per-test failure counter (14-day window)Postgres
- LogicFilter tests at or over the repeat-failure threshold
- ActionOpen GitLab issue labeled flaky::quarantineGitLab
- OutputPost quarantine candidates to on-call Slack channelSlack
What it does
Keeps a running tally of which individual test cases fail on your default-branch pipelines. When a single spec fails three or more times within a rolling window, it flags it as flaky, files a GitLab issue tagged `flaky::quarantine`, and alerts the on-call channel so no one re-runs the pipeline blindly.
When to use it
Use it when intermittent test failures are eroding trust in CI and people have started reflexively hitting retry. It turns "is this flaky or real?" into a tracked, labeled decision instead of tribal knowledge.
How it works
- 1A GitLab pipeline-failed webhook fires for the default branch.
- 2The flow fetches the JUnit report artifact and extracts each failed test name plus its file path.
- 3It increments a per-test failure counter stored in Postgres, scoped to a rolling 14-day window.
- 4A logic step checks each test against the repeat-failure threshold (default 3).
- 5For tests over the line, it creates a GitLab issue labeled `flaky::quarantine` with the failure history and links to the offending pipelines.
- 6A Slack message summarizes the new quarantine candidates with jump links to each issue.
Set it up
What you configure once, before turning it on.
- 1Connect GitLabRepos, MRs, pipelines, registry.
- 2Connect PostgresAny Postgres URL — query, write, migrate.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Upstream Release to Notion Upgrade Brief
When a watched package publishes a new release, fetches the release notes, maps them to the internal modules that depend on it.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.