ENGINEERING
Page on-call when flaky failures spike into a suite meltdown
Monitors the rate of newly quarantined specs and total CI failure volume; when both spike past a threshold in a short window, it pages the on-call engineer and posts a meltdown…
How it runs
The automated pipeline, trigger to output.
- TriggerWebhook: spec quarantined eventHTTP webhook
- ActionQuery trailing-window quarantine and failure countsPostgres
- LogicGate: both metrics exceed meltdown thresholds
- ActionTrigger PagerDuty incident for on-callPagerDuty
- OutputPost meltdown alert to SlackSlack
What it does
Distinguishes ordinary background flakiness from a sudden systemic failure — a bad dependency bump, a broken shared fixture, or infra degradation — that masquerades as a flood of "flaky" tests. When the quarantine rate and overall failure volume both spike together, it escalates instead of silently quarantining everything.
When to use it
Use this as a safety net on top of an automated quarantine program, so the system never quietly hides a real outage by quarantining dozens of specs at once.
How it works
- 1A webhook fires each time a spec is quarantined, carrying the running counts.
- 2The flow queries Postgres for the number of newly quarantined specs and total CI failures in the trailing time window.
- 3A logic gate checks whether both the quarantine rate and failure volume exceed their meltdown thresholds simultaneously.
- 4If the gate trips, it triggers a PagerDuty incident for the on-call engineer.
- 5It also posts a meltdown alert to Slack summarizing the spike and the affected specs so the team can converge immediately.
Set it up
What you configure once, before turning it on.
- 1Connect HTTP webhookTrigger any URL on agent actions.
- 2Connect PostgresAny Postgres URL — query, write, migrate.
- 3Connect PagerDutyIncidents, on-call, escalations.
- 4Connect SlackChannels, DMs, threads, mentions.
- 5Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 6Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 7Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Block PRs that add incompatible Hugging Face model licenses
When a pull request adds or bumps a Hugging Face model dependency, it fetches the model card license, checks it against your org's allowed-license policy.
Quarterly Logging Hygiene Audit Agent
An agent-driven quarterly sweep that surveys all Axiom datasets, builds a logging-hygiene scorecard per service.
Post-Merge Log Volume Recheck After Downsampling PR
After a log-level PR merges, waits a day then re-queries Axiom to confirm the targeted stream's volume actually dropped.
Axiom Ingest Cost Spike to Linear Triage Ticket
When Axiom ingest volume spikes beyond its baseline, identifies which service caused it and files a Linear ticket with the offending log stream, sample lines, and a downsampling…
File a Linear license-review ticket for risky model adds
When a PR introduces a Hugging Face model with a non-permissive or unknown license, it opens a Linear issue assigned to the legal-review team with the model, license.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.

Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.
