ENGINEERING
Flaky Test Detector with Auto-Quarantine PR
Watches CI failures on GitHub, identifies tests that pass on rerun (a flaky signature).
How it runs
The automated pipeline, trigger to output.
- TriggerGitHub workflow run failsGitHub
- ActionFetch run attempts and job logsGitHub
- LogicDetect flaky signature (fail then pass, same SHA)
- ActionCommit quarantine annotation to new branchGitHub
- OutputOpen quarantine pull requestGitHub
What it does
This workflow turns intermittent CI red into a self-healing loop. When a workflow run fails, it inspects whether the same job passed on a rerun. If a test failed then passed without any code change, it marks the test as flaky and proposes quarantining it via an automated pull request, keeping the main pipeline green while the flake is tracked for a real fix.
When to use it
Use this when flaky tests routinely block merges and engineers waste time hitting "re-run jobs." It is ideal for teams with a large suite where a handful of nondeterministic tests cause most of the noise.
How it works
- 1A GitHub Actions workflow_run failure event fires the workflow.
- 2The flow fetches the run's job history and prior attempts from the GitHub API.
- 3A logic step compares attempts: same commit SHA, failed then passed = flaky.
- 4For confirmed flakes, an action edits the test file to add a quarantine annotation and commits to a new branch.
- 5The output step opens a pull request describing the flake evidence and reruns observed.
Set it up
What you configure once, before turning it on.
- 1Connect GitHubRepos, issues, pull requests, actions.
- 2Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 3Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 4Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Scan for deprecated endpoints and email consumers a weekly sunset countdown
On a weekly schedule, scans the OpenAPI spec for endpoints marked deprecated with a sunset date, and emails each consuming team a countdown of how many days remain before removal.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.