ENGINEERING
Flaky Test Quarantine Bot for GitHub Actions
Watches GitHub Actions CI runs, detects tests that fail then pass on retry, auto-tags them as flaky, and opens a tracking issue with the full failure history.
How it runs
The automated pipeline, trigger to output.
- TriggerGitHub Actions workflow_run completedGitHub
- ActionFetch per-attempt job and test resultsGitHub
- LogicClassify: failed-then-passed = flaky
- ActionFind existing issue by test fingerprintGitHub
- OutputCreate or update tracking issue with run historyGitHub
What it does
This workflow listens for completed GitHub Actions CI runs and identifies tests that failed on a first attempt but passed when the job was retried — the classic signature of a flaky test. It adds a `flaky` annotation, opens or updates a GitHub tracking issue, and records each occurrence so engineers can see how often the test misbehaves over time.
When to use it
Reach for this when your team retries failing CI jobs and intermittent tests keep eroding trust in the pipeline. It separates genuine regressions from noise so a red build means something again.
How it works
- 1A GitHub `workflow_run` event fires when a CI run finishes.
- 2The bot fetches the run's jobs and per-attempt test results from the GitHub API.
- 3A decision step compares attempts: a test that failed then passed on rerun is classified as flaky; consistently failing tests are ignored as real failures.
- 4For each flaky test it searches existing issues by a stable fingerprint, then creates a new tracking issue or appends a dated run-history entry to the matching one.
- 5It labels the issue `flaky-test` and posts the updated count, giving the team a single source of truth per flaky test.
Set it up
What you configure once, before turning it on.
- 1Connect GitHubRepos, issues, pull requests, actions.
- 2Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 3Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 4Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Scan for deprecated endpoints and email consumers a weekly sunset countdown
On a weekly schedule, scans the OpenAPI spec for endpoints marked deprecated with a sunset date, and emails each consuming team a countdown of how many days remain before removal.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.