AI AGENTS
On-call agent: Datadog monitor to MCP action with two-key approval
A firing Datadog monitor triggers an agent that proposes a remediation via a custom MCP server and requires two distinct Slack approvals before the action executes.
How it runs
The automated pipeline, trigger to output.
- TriggerDatadog monitor firesDatadog
- ActionQuery custom MCP for state and toolsCustom MCP server
- LogicSelect one MCP action, state blast radius
- ActionRequest two distinct Slack approvalsSlack
- LogicGate: require two different approvers
- ActionExecute approved MCP remediationCustom MCP server
- OutputReport outcome and rollback handle to SlackSlack
What it does
Bridges Datadog alerting to your internal tooling through a custom MCP server, with a stricter two-person rule. The agent proposes one MCP action (e.g. rotate a node, drain a queue) and only runs it after two different engineers approve.
When to use it
Use it for higher-blast-radius operations where one approval is not enough. Ideal when remediation lives behind an internal MCP server rather than raw shell.
How it works
- 1A Datadog monitor webhook posts the alert, metric, and tags.
- 2The agent queries the custom MCP server for current resource state and available remediation tools.
- 3It selects one MCP action and explains the blast radius and rollback path.
- 4The proposal goes to Slack requiring two distinct approver clicks.
- 5The gate enforces both approvals come from different users before proceeding.
- 6The MCP action executes and the agent reports the outcome and any rollback handle back to Slack.
Set it up
What you configure once, before turning it on.
- 1Connect DatadogMetrics, traces, log search.
- 2Connect Custom MCP serverConnect any MCP-compatible tool you own.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More AI Agents workflows
Custom Metrics Cardinality Spike Pager
A webhook from a Datadog monitor fires when custom-metric cardinality jumps; an agent pinpoints the offending metric and tag, estimates the added cost.
Sentry-to-Confluence Runbook Updater
When a Sentry issue is resolved, the agent finds the matching Confluence runbook page and proposes an inline update with the verified fix.
Stale Doc-PR Chaser for Runbook Gaps
On a daily schedule the agent finds runbook doc PRs that were opened from resolved incidents but never reviewed, summarizes what each one fixes.
Resolved Incident to Public Troubleshooting Doc
For customer-facing errors resolved in Sentry, the agent drafts a sanitized troubleshooting entry and opens a PR to your ReadMe documentation.
On-Call Runbook Gap Closer: Resolved Sentry Issues to Doc PRs
An agent reads each newly resolved Sentry issue, compares the actual fix against your existing runbook, and opens a GitHub PR adding the missing remediation steps.
Weekly On-Call Doc-Gap Digest
Each week the agent reviews every Sentry issue resolved in the last 7 days, ranks the ones whose runbook coverage is missing or thin.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.