ENGINEERING
Auto-Quarantine Flaky Tests from Reruns
When a CI test passes only after a rerun on the same commit, files a GitHub issue, tags the test with a quarantine label, and skips it in future runs so it stops blocking merges.
How it runs
The automated pipeline, trigger to output.
- TriggerGitHub workflow run completedGitHub
- LogicRerun passed but original failed on same SHA?
- ActionParse failed test ID from original run logsGitHub
- ActionOpen or update quarantine issue with labelGitHub
- ActionCommit test to skip-list fileGitHub
- OutputPost quarantine summary to SlackSlack
What it does
Detects tests that fail then pass on a rerun of the *same* commit — the clearest signal of flakiness — and quarantines them automatically. It opens a tracking GitHub issue, applies a `flaky-quarantine` label, and adds the test to a skip-list so the next CI run no longer fails on it.
When to use it
Use it when intermittent failures are forcing engineers to mash the rerun button and red builds are eroding trust in CI. It removes the manual judgment call of "is this test actually broken or just flaky?" by keying off the rerun-passed signal.
How it works
- 1A GitHub Actions webhook fires when a workflow run completes.
- 2A filter checks whether the run was a rerun that passed while the original run on the same commit SHA failed.
- 3If so, it parses the failed test identifier from the original run's logs.
- 4It searches existing issues to avoid duplicates, then opens (or comments on) a quarantine issue with run links.
- 5It applies the `flaky-quarantine` label and appends the test to the repo's skip-list file via a commit.
- 6It posts a summary to the team's Slack CI channel.
Set it up
What you configure once, before turning it on.
- 1Connect GitHubRepos, issues, pull requests, actions.
- 2Connect SlackChannels, DMs, threads, mentions.
- 3Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 4Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 5Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Upstream Release to Notion Upgrade Brief
When a watched package publishes a new release, fetches the release notes, maps them to the internal modules that depend on it.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.