ENGINEERING
Flaky-Test Quarantine Ledger from CI Reruns
Detects tests that pass on rerun after failing in the same CI commit, records each occurrence to a BigQuery ledger.
How it runs
The automated pipeline, trigger to output.
- TriggerGitHub workflow_run completedGitHub
- ActionFetch job results and prior attempts for the SHAGitHub
- LogicTag tests that failed then passed on the same commit
- ActionAppend flake events to BigQuery ledgerBigQuery
- LogicCompute 14-day flake rate and branch on thresholdBigQuery
- OutputNotify team of newly quarantined tests in SlackSlack
What it does
It watches GitHub Actions workflow runs and identifies tests that fail then pass on a retry of the same commit — the classic flaky signature. Every flip is written to a durable BigQuery ledger so flake rate is measured over time, not guessed. When a test's 14-day flake rate crosses your threshold, it is marked quarantined and the team is notified.
When to use it
Run this when reruns are masking instability and you need hard data on which tests are actually flaky before you start spending engineering time deflaking them. It turns "just hit retry" into an auditable trail.
How it works
- 1A GitHub workflow_run completion fires the trigger.
- 2The flow pulls the job and step results plus any prior attempts for the same commit SHA.
- 3Logic compares attempts: a test that was red then green on the same SHA is tagged as a flake event.
- 4Each flake event is appended to a BigQuery ledger table with test ID, SHA, branch, and timestamp.
- 5Logic queries the rolling 14-day flake rate per test and branches when it exceeds the threshold.
- 6Crossing tests are posted to Slack with their flake rate and recent failure samples.
Set it up
What you configure once, before turning it on.
- 1Connect GitHubRepos, issues, pull requests, actions.
- 2Connect BigQueryDatasets, queries, schemas.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Scan for deprecated endpoints and email consumers a weekly sunset countdown
On a weekly schedule, scans the OpenAPI spec for endpoints marked deprecated with a sunset date, and emails each consuming team a countdown of how many days remain before removal.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.