ENGINEERING
Datadog CI Visibility Flaky Spike to PagerDuty
Subscribes to Datadog CI Visibility flaky-test events and, when a test's flaky rate spikes past a guardrail, pages the owning team via PagerDuty and files a GitHub issue…
How it runs
The automated pipeline, trigger to output.
- TriggerDatadog CI Visibility flaky eventDatadog
- LogicCompare spike to on-call guardrail
- ActionPage owning team on breachPagerDuty
- OutputFile GitHub issue with Datadog trace linkGitHub
What it does
This workflow reacts to flaky-test signals from Datadog CI Visibility. When Datadog reports a test whose flaky rate has spiked beyond a guardrail, the bot decides whether it crosses the on-call threshold, pages the owning team through PagerDuty if so, and always files a GitHub tracking issue linking back to the Datadog test page and recent failed traces.
When to use it
Use this when flakiness in critical paths needs an immediate human, not just a backlog ticket. It routes high-impact spikes to on-call while routing everything else to a tracked issue.
How it works
- 1A Datadog CI Visibility webhook event fires for a flaky test.
- 2The bot reads the test's flaky rate, owning team tag, and recent failure traces from the event payload.
- 3A decision step checks the spike against the guardrail and the team's on-call policy.
- 4If it breaches the on-call threshold, it triggers a PagerDuty incident routed to the owning team's service.
- 5Regardless, it creates a GitHub issue with the Datadog deep link, flaky rate, and trace references for the team to triage.
Set it up
What you configure once, before turning it on.
- 1Connect DatadogMetrics, traces, log search.
- 2Connect PagerDutyIncidents, on-call, escalations.
- 3Connect GitHubRepos, issues, pull requests, actions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Upstream Release to Notion Upgrade Brief
When a watched package publishes a new release, fetches the release notes, maps them to the internal modules that depend on it.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.