ENGINEERING
Escalate High-Frequency Flaky Tests from Datadog CI Visibility
Polls Datadog CI Visibility for flaky-test rates, and when a test's flakiness breaches the SLO it pages the owning team in PagerDuty and files a tracking issue.
How it runs
The automated pipeline, trigger to output.
- TriggerScheduled flakiness SLO check
- ActionQuery Datadog CI Visibility for flake ratesDatadog
- LogicFlag tests breaching SLO with min-runs guard
- ActionPage owning team via PagerDutyPagerDuty
- OutputOpen or update GitHub tracking issueGitHub
What it does
This workflow treats chronic flakiness as an operational incident. On a schedule it queries Datadog CI Visibility for per-test flake rates, compares each against your reliability SLO, and for any test breaching the budget it pages the owning team via PagerDuty and opens a GitHub issue to anchor the fix.
When to use it
Use it when a few persistently flaky tests are eroding developer trust in CI and a passive backlog is not enough. It is for platform or DevEx teams who already emit CI metrics to Datadog and want flakiness governed by an SLO with real escalation.
How it works
- 1A scheduled trigger runs the check at a fixed cadence.
- 2The flow queries Datadog CI Visibility for flaky-test rate and run volume over the trailing window.
- 3A branch evaluates each test against the SLO and the minimum-runs guard so rarely-run tests do not page.
- 4Tests in breach trigger a PagerDuty event routed to the owning team's service.
- 5The flow opens or updates a GitHub issue capturing the flake rate, sample failing runs, and the SLO breach for tracking to resolution.
Set it up
What you configure once, before turning it on.
- 1Connect DatadogMetrics, traces, log search.
- 2Connect PagerDutyIncidents, on-call, escalations.
- 3Connect GitHubRepos, issues, pull requests, actions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Upstream Release to Notion Upgrade Brief
When a watched package publishes a new release, fetches the release notes, maps them to the internal modules that depend on it.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.