AI AGENTS
Severe Log-Cost Spike → PagerDuty Escalation with Triage Note
A scheduled check measures Axiom ingestion against budget; on a severe overage it pages on-call via PagerDuty with a pre-built triage note naming the noisy service.
How it runs
The automated pipeline, trigger to output.
- TriggerScheduled budget check runs through the day
- ActionQuery Axiom ingestion and project burn vs budgetAxiom
- LogicClassify overage: none, minor, or severe
- ActionMinor: post noisy-service heads-up to SlackSlack
- OutputSevere: page on-call via PagerDuty with triage notePagerDuty
What it does
Splits log-cost spikes into two response tiers. It checks Axiom ingestion against a daily budget; a severe overage pages on-call through PagerDuty with a triage note that already names the noisy service and a stopgap sampling action, while a minor overage just drops a heads-up in Slack. Engineers only get woken up when the burn rate justifies it.
When to use it
Use it when uncontrolled log volume can blow the monthly budget in days and you need real escalation, not another muted alert. Best for teams with PagerDuty rotations that want severity-aware routing tied to actual cost.
How it works
- 1A schedule runs the budget check several times a day.
- 2It queries Axiom for cumulative ingestion and projects the daily burn against budget.
- 3A logic step classifies the overage as none, minor, or severe.
- 4For minor overages it queries the top noisy service and posts a Slack heads-up.
- 5For severe overages the agent builds a triage note naming the service and a stopgap sampling rule.
- 6It triggers a PagerDuty incident carrying that triage note to on-call.
Set it up
What you configure once, before turning it on.
- 1Connect AxiomLog streams, queries, dashboards.
- 2Connect PagerDutyIncidents, on-call, escalations.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More AI Agents workflows
Custom Metrics Cardinality Spike Pager
A webhook from a Datadog monitor fires when custom-metric cardinality jumps; an agent pinpoints the offending metric and tag, estimates the added cost.
Sentry-to-Confluence Runbook Updater
When a Sentry issue is resolved, the agent finds the matching Confluence runbook page and proposes an inline update with the verified fix.
Stale Doc-PR Chaser for Runbook Gaps
On a daily schedule the agent finds runbook doc PRs that were opened from resolved incidents but never reviewed, summarizes what each one fixes.
Resolved Incident to Public Troubleshooting Doc
For customer-facing errors resolved in Sentry, the agent drafts a sanitized troubleshooting entry and opens a PR to your ReadMe documentation.
On-Call Runbook Gap Closer: Resolved Sentry Issues to Doc PRs
An agent reads each newly resolved Sentry issue, compares the actual fix against your existing runbook, and opens a GitHub PR adding the missing remediation steps.
Weekly On-Call Doc-Gap Digest
Each week the agent reviews every Sentry issue resolved in the last 7 days, ranks the ones whose runbook coverage is missing or thin.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.