AI AGENTS
Honeycomb Anomaly Severity Router with PagerDuty Escalation
On a Honeycomb anomaly, the agent classifies severity from the trace blast radius and routes it — low severity to a Slack triage channel with a fix suggestion.
How it runs
The automated pipeline, trigger to output.
- TriggerHoneycomb detects anomalyHoneycomb
- ActionPull spans and estimate impactHoneycomb
- LogicClassify severity by blast radius
- ActionLow: post triage note to SlackSlack
- OutputHigh: page on-call via PagerDuty with rollback runbookPagerDuty
What it does
Decides how loud an anomaly should be before it wakes anyone. The agent reads the trace, estimates blast radius and customer impact, then branches: minor blips get a quiet Slack note with a suggested fix, while severe regressions trigger a PagerDuty page carrying a rollback-first remediation plan.
When to use it
Use it when alert fatigue is real and not every Honeycomb anomaly deserves a page. This applies consistent severity judgment so the right channel gets the right urgency.
How it works
- 1Honeycomb fires on a detected anomaly.
- 2The agent pulls the affected spans, error rate, and span volume to estimate impact.
- 3A severity branch classifies the event as low or high based on blast radius and thresholds.
- 4Low severity: it posts a Slack triage message with a suggested fix and rollback note.
- 5High severity: it opens a PagerDuty incident pre-loaded with a rollback-first runbook.
- 6Either path records the classification rationale for later tuning.
Set it up
What you configure once, before turning it on.
- 1Connect HoneycombDistributed traces and queries.
- 2Connect PagerDutyIncidents, on-call, escalations.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More AI Agents workflows
Custom Metrics Cardinality Spike Pager
A webhook from a Datadog monitor fires when custom-metric cardinality jumps; an agent pinpoints the offending metric and tag, estimates the added cost.
Sentry-to-Confluence Runbook Updater
When a Sentry issue is resolved, the agent finds the matching Confluence runbook page and proposes an inline update with the verified fix.
Stale Doc-PR Chaser for Runbook Gaps
On a daily schedule the agent finds runbook doc PRs that were opened from resolved incidents but never reviewed, summarizes what each one fixes.
Resolved Incident to Public Troubleshooting Doc
For customer-facing errors resolved in Sentry, the agent drafts a sanitized troubleshooting entry and opens a PR to your ReadMe documentation.
On-Call Runbook Gap Closer: Resolved Sentry Issues to Doc PRs
An agent reads each newly resolved Sentry issue, compares the actual fix against your existing runbook, and opens a GitHub PR adding the missing remediation steps.
Weekly On-Call Doc-Gap Digest
Each week the agent reviews every Sentry issue resolved in the last 7 days, ranks the ones whose runbook coverage is missing or thin.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.