DEVOPS
Classify CI failures and auto-quarantine flaky tests
On every failed CI run, decides whether a test failure is a real regression or intermittent flake, and tags repeat offenders for quarantine so they stop blocking merges.
How it runs
The automated pipeline, trigger to output.
- TriggerGitHub workflow_run failedGitHub
- ActionFetch failed jobs and parse failing testsGitHub
- LogicClassify flake vs regression by history
- ActionLabel flaky tests and update tracking issueGitHub
- OutputPass regressions through to block merge
What it does
When a CI workflow run fails, this template pulls the failing test names and compares them against recent history. If a test has flipped between pass and fail on the same commit or across reruns, it is classified as flaky rather than a genuine regression, labeled in GitHub, and recorded so the team can track it.
When to use it
Use it when intermittent failures are eroding trust in CI and engineers are blindly clicking re-run. It separates signal (real breakage) from noise (flakes) automatically, on every failed run.
How it works
- 1A GitHub `workflow_run` completion event with a failed conclusion fires the trigger.
- 2The flow fetches the run's failed jobs and parses the failing test identifiers from the logs.
- 3A logic step checks each test's recent pass/fail pattern: alternating results or a passing rerun marks it flaky; a consistent failure marks it a regression.
- 4For flaky tests, GitHub applies a `flaky` label and adds the test to a quarantine tracking issue.
- 5Real regressions are passed through untouched so they still block the merge as intended.
Set it up
What you configure once, before turning it on.
- 1Connect GitHubRepos, issues, pull requests, actions.
- 2Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 3Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 4Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More DevOps workflows
Slack-approved pause for idle Hugging Face Spaces
On a daily scan it finds idle paid Spaces and posts an interactive Slack approval; on approve it pauses the Space and logs the decision to a GitHub issue audit trail.
Block costly Hugging Face Space hardware upgrades in PR review
When a pull request changes a Space's hardware config, it estimates the new monthly cost and posts a GitHub PR comment that flags upgrades crossing a budget ceiling.
Hugging Face Spaces idle-runtime sweep with auto-pause
On a schedule, scans all Hugging Face Spaces for ones running idle past a threshold, pauses them to stop billing, and posts a Slack summary with the estimated monthly savings.
Open a Zoom war-room from a Datadog multi-alert storm
When a Datadog monitor crosses a critical threshold, this workflow dedupes against active incidents, and only for a genuinely new outage it creates a Zoom bridge.
Auto-spin a Zoom war-room when PagerDuty hits SEV-1
When a PagerDuty incident escalates to a critical severity, this workflow creates a dedicated Zoom meeting and posts the bridge link to the incident's Slack channel so responders…
Spin up a war-room on demand from a Slack slash command
When an engineer runs a Slack command, this workflow creates a Zoom bridge, opens a tracking Sentry-linked incident, files a Linear issue for follow-up.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.