SECOPS

Page on-call and roll back the last deploy on extreme burn

When a Honeycomb burn-rate alert hits critical, it pages the on-call via PagerDuty, triggers a rollback of the most recent deployment.

CategorySecOps
Enginesim
Difficultyadvanced
Triggerwebhook
Steps5
Setup~25 min

How it runs

The automated pipeline, trigger to output.

  • TriggerHoneycomb critical burn-rate alert webhookHoneycomb
  • LogicConfirm critical tier before escalating
  • ActionPage on-call via PagerDutyPagerDutyPagerDuty
  • ActionRoll back to last known-good deploymentGitHubGitHub
  • OutputOpen Slack incident thread with full contextSlack

What it does

This handles the worst case: budget burning so fast that a freeze alone isn't enough. On a critical fast-burn alert it pages the responsible engineer, kicks off a rollback of the latest GitHub deployment, and spins up a coordinated Slack thread — compressing the first minutes of incident response into one automated flow.

When to use it

Reserve this for your highest-tier services where a sustained critical burn is a customer-impacting incident and an automatic rollback is the agreed safe action.

How it works

  1. 1A Honeycomb webhook fires on a critical-severity burn-rate alert.
  2. 2A logic step confirms the alert is critical tier (not a warning) before escalating.
  3. 3An action triggers a PagerDuty incident routed to the service's on-call rotation.
  4. 4An action calls GitHub to revert to the last known-good deployment for the affected service.
  5. 5The output opens a Slack incident thread with the SLO, burn rate, rollback target, and the PagerDuty link.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect HoneycombDistributed traces and queries.
  2. 2
    Connect PagerDutyIncidents, on-call, escalations.
  3. 3
    Connect GitHubRepos, issues, pull requests, actions.
  4. 4
    Connect SlackChannels, DMs, threads, mentions.
  5. 5
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  6. 6
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  7. 7
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.