ENGINEERING
Auto-Quarantine a Confirmed Flaky Test via Skip PR with Owner Tracking
When a test is flagged flaky three times in a week, opens a GitHub PR that marks the test as skipped/quarantined, requests review from the code owner.
How it runs
The automated pipeline, trigger to output.
- TriggerFlake report webhook receivedHTTP webhook
- ActionTally recent flake reports for the test
- LogicBranch: weekly threshold crossed vs. record and exit
- ActionOpen quarantine PR and request owner reviewGitHub
- ActionCreate linked un-quarantine ticket with due dateLinear
- OutputNotify owner in Slack with PR and ticket linksSlack
What it does
This workflow acts on repeat offenders. When the same test is reported flaky three or more times within a rolling week, it programmatically opens a GitHub pull request that annotates the test as quarantined (skip marker plus a tracking comment), requests review from the CODEOWNERS owner, and creates a Linear ticket so the quarantine is temporary and tracked to resolution.
When to use it
Use this when chronically flaky tests are blocking merges and you want to remove them from the critical path automatically, while guaranteeing an owner is on the hook to fix and restore them.
How it works
- 1A webhook receives a flake report event (from CI or an upstream detector) carrying the test identifier.
- 2The flow tallies recent flake reports for that test from its store.
- 3A branch checks whether the count crossed the weekly threshold; below it, the run records the report and exits.
- 4Above threshold, the flow opens a GitHub PR adding the skip annotation and a quarantine comment, with the owner requested as reviewer.
- 5A Linear ticket is created and linked to the PR with a due date to un-quarantine.
- 6Slack notifies the owner with the PR and ticket links.
Set it up
What you configure once, before turning it on.
- 1Connect GitHubRepos, issues, pull requests, actions.
- 2Connect LinearIssues, projects, cycles, triage.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Connect HTTP webhookTrigger any URL on agent actions.
- 5Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 6Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 7Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Block PRs that add incompatible Hugging Face model licenses
When a pull request adds or bumps a Hugging Face model dependency, it fetches the model card license, checks it against your org's allowed-license policy.
Quarterly Logging Hygiene Audit Agent
An agent-driven quarterly sweep that surveys all Axiom datasets, builds a logging-hygiene scorecard per service.
Post-Merge Log Volume Recheck After Downsampling PR
After a log-level PR merges, waits a day then re-queries Axiom to confirm the targeted stream's volume actually dropped.
Axiom Ingest Cost Spike to Linear Triage Ticket
When Axiom ingest volume spikes beyond its baseline, identifies which service caused it and files a Linear ticket with the offending log stream, sample lines, and a downsampling…
File a Linear license-review ticket for risky model adds
When a PR introduces a Hugging Face model with a non-permissive or unknown license, it opens a Linear issue assigned to the legal-review team with the model, license.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.