ENGINEERING
Weekly Quarantined-Test Review Board
Once a week, compiles every test currently carrying the flaky-quarantine label into a Notion review board with age, flake history, and owner, and de-quarantines tests that have…
How it runs
The automated pipeline, trigger to output.
- TriggerWeekly schedule
- ActionList issues labeled flaky-quarantineGitHub
- ActionEnrich with Datadog stability + ageDatadog
- LogicFlag tests stable long enough to re-enable
- ActionDe-quarantine: close issue, drop labelGitHub
- OutputPublish review board to Notion + SlackNotion
What it does
Produces a weekly review board of all quarantined tests so they don't rot in skip-lists forever. It pulls every open issue labeled `flaky-quarantine`, enriches each with age and recent stability from Datadog, writes a Notion board, and automatically de-quarantines tests that have passed consistently since being parked.
When to use it
Use it to close the loop on quarantine — the hard part isn't parking flaky tests, it's getting them fixed or safely re-enabled. This gives the team a standing artifact and prevents permanent test debt.
How it works
- 1A weekly schedule trigger starts the review.
- 2It lists all open GitHub issues with the `flaky-quarantine` label.
- 3For each, it pulls recent run stability from Datadog and the issue age.
- 4A branch flags tests stable for N days as ready to re-enable.
- 5For those, it removes the skip-list entry, closes the issue, and drops the label.
- 6It writes the full board (still-quarantined, re-enabled, stale) to a Notion page and pings the team in Slack.
Set it up
What you configure once, before turning it on.
- 1Connect GitHubRepos, issues, pull requests, actions.
- 2Connect DatadogMetrics, traces, log search.
- 3Connect NotionPages, databases, comments.
- 4Connect SlackChannels, DMs, threads, mentions.
- 5Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 6Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 7Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Block PRs that add incompatible Hugging Face model licenses
When a pull request adds or bumps a Hugging Face model dependency, it fetches the model card license, checks it against your org's allowed-license policy.
Quarterly Logging Hygiene Audit Agent
An agent-driven quarterly sweep that surveys all Axiom datasets, builds a logging-hygiene scorecard per service.
Post-Merge Log Volume Recheck After Downsampling PR
After a log-level PR merges, waits a day then re-queries Axiom to confirm the targeted stream's volume actually dropped.
Axiom Ingest Cost Spike to Linear Triage Ticket
When Axiom ingest volume spikes beyond its baseline, identifies which service caused it and files a Linear ticket with the offending log stream, sample lines, and a downsampling…
File a Linear license-review ticket for risky model adds
When a PR introduces a Hugging Face model with a non-permissive or unknown license, it opens a Linear issue assigned to the legal-review team with the model, license.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.