ENGINEERING
Datadog Flake-Rate Threshold to Linear Quarantine
Polls Datadog CI Test Visibility on a schedule, finds tests whose flake rate crosses a threshold, and files a Linear quarantine ticket assigned to the last committer.
How it runs
The automated pipeline, trigger to output.
- TriggerDaily schedule
- ActionQuery Datadog flake rates over 7 daysDatadog
- LogicFilter tests above flake-rate threshold
- ActionResolve last committer per test fileGitLab
- ActionCreate Linear quarantine ticket and assignLinear
- OutputPost daily quarantine summary to SlackSlack
What it does
This workflow watches your Datadog CI Test Visibility metrics and catches tests that have become statistically flaky over a rolling window — not just ones that failed once. When a test's flake rate exceeds your threshold, it opens a Linear quarantine ticket with the trend data and assigns it to whoever last changed the test.
When to use it
Use it when you already ship test results to Datadog and want a data-driven quarantine policy (e.g. "quarantine any test flaking more than 5% over 7 days") rather than reacting to single red builds.
How it works
- 1A daily schedule kicks off the run.
- 2The flow queries Datadog CI Test Visibility for per-test flake rates over the last 7 days.
- 3A logic step selects tests above the configured flake-rate threshold and excludes ones already quarantined.
- 4It looks up the last committer for each test file via the GitLab API.
- 5It creates a Linear issue in the Quarantine project, tagged with the flake rate and a Datadog dashboard link, assigned to that committer.
- 6It drops a summary of the day's new quarantines into a Slack channel.
Set it up
What you configure once, before turning it on.
- 1Connect DatadogMetrics, traces, log search.
- 2Connect LinearIssues, projects, cycles, triage.
- 3Connect GitLabRepos, MRs, pipelines, registry.
- 4Connect SlackChannels, DMs, threads, mentions.
- 5Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 6Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 7Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Block PRs that add incompatible Hugging Face model licenses
When a pull request adds or bumps a Hugging Face model dependency, it fetches the model card license, checks it against your org's allowed-license policy.
Quarterly Logging Hygiene Audit Agent
An agent-driven quarterly sweep that surveys all Axiom datasets, builds a logging-hygiene scorecard per service.
Post-Merge Log Volume Recheck After Downsampling PR
After a log-level PR merges, waits a day then re-queries Axiom to confirm the targeted stream's volume actually dropped.
Axiom Ingest Cost Spike to Linear Triage Ticket
When Axiom ingest volume spikes beyond its baseline, identifies which service caused it and files a Linear ticket with the offending log stream, sample lines, and a downsampling…
File a Linear license-review ticket for risky model adds
When a PR introduces a Hugging Face model with a non-permissive or unknown license, it opens a Linear issue assigned to the legal-review team with the model, license.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.