ENGINEERING
Quarantine Tests Exceeding a Datadog Flake-Rate Threshold
On a daily schedule, reads per-test pass/fail metrics from Datadog CI Visibility, and for any test whose intermittent failure rate crosses a threshold.
How it runs
The automated pipeline, trigger to output.
- TriggerDaily schedule
- ActionQuery per-test flake rates from Datadog CI VisibilityDatadog
- LogicSelect tests in the intermittent-failure band
- ActionOpen GitHub PR adding tests to quarantine listGitHub
- ActionFile Linear deflake ticket per testLinear
- OutputReport quarantine batch to SlackSlack
What it does
It turns raw CI telemetry into action. Each day it queries Datadog CI Visibility for per-test flake rates, identifies tests that fail intermittently above your tolerance (for example, 2-15% on the same branch), and both files a Linear ticket and opens a GitHub PR that adds them to a quarantine skip list so they stop blocking the main pipeline.
When to use it
Use it when you already ship test results to Datadog and want a data-driven, no-arguments policy for what gets quarantined. It removes the judgment call about whether a test is "flaky enough."
How it works
- 1A daily schedule triggers the run.
- 2A Datadog action queries CI Visibility for each test's pass and fail counts over the trailing window.
- 3A logic step computes flake rate and selects tests inside the intermittent band, excluding consistently-failing (genuinely broken) tests.
- 4A GitHub action opens a PR adding the selected tests to the quarantine list.
- 5A Linear action files a deflake ticket per test linking the Datadog metric and the quarantine PR.
- 6A Slack message reports the day's quarantine batch.
Set it up
What you configure once, before turning it on.
- 1Connect DatadogMetrics, traces, log search.
- 2Connect GitHubRepos, issues, pull requests, actions.
- 3Connect LinearIssues, projects, cycles, triage.
- 4Connect SlackChannels, DMs, threads, mentions.
- 5Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 6Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 7Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Block PRs that add incompatible Hugging Face model licenses
When a pull request adds or bumps a Hugging Face model dependency, it fetches the model card license, checks it against your org's allowed-license policy.
Quarterly Logging Hygiene Audit Agent
An agent-driven quarterly sweep that surveys all Axiom datasets, builds a logging-hygiene scorecard per service.
Post-Merge Log Volume Recheck After Downsampling PR
After a log-level PR merges, waits a day then re-queries Axiom to confirm the targeted stream's volume actually dropped.
Axiom Ingest Cost Spike to Linear Triage Ticket
When Axiom ingest volume spikes beyond its baseline, identifies which service caused it and files a Linear ticket with the offending log stream, sample lines, and a downsampling…
File a Linear license-review ticket for risky model adds
When a PR introduces a Hugging Face model with a non-permissive or unknown license, it opens a Linear issue assigned to the legal-review team with the model, license.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.