ENGINEERING

Recovery agent reopens GitLab merges after budget recovers

When Honeycomb signals an SLO has recovered and held above budget for a sustained window, an agent verifies the recovery is stable, lifts the GitLab merge-freeze.

CategoryEngineering
Enginepaperclip
Difficultyadvanced
Triggerwebhook
Steps5
Setup~25 min

How it runs

The automated pipeline, trigger to output.

  • TriggerHoneycomb SLO recovery webhookHoneycomb
  • LogicVerify recovery held over stabilization window
  • ActionRemove GitLab merge-freeze labelGitLabGitLab
  • ActionNotify blocked MR authors in SlackSlack
  • OutputLog freeze duration to LinearLinearLinear

What it does

Handles the unfreeze side with judgment. A naive automation can unfreeze the instant burn dips, only to re-freeze minutes later. This agent waits for a sustained recovery window, double-checks the Honeycomb signal isn't noisy, then lifts the GitLab freeze, notifies the people whose merge requests were blocked, and records the incident's freeze duration as a Linear issue for the reliability backlog.

When to use it

Use it when flapping freezes are eroding trust and you want a deliberate, auditable unfreeze with a human-readable trail. Best paired with the burn-triggered freeze workflow.

How it works

  1. 1A Honeycomb recovery webhook triggers the run.
  2. 2The agent checks that burn has stayed under threshold for the configured stabilization window and isn't trending back up.
  3. 3If stable, it removes the GitLab `merge-freeze` label and restores merge settings.
  4. 4It comments on the previously blocked merge requests telling authors they can merge.
  5. 5It creates a Linear issue capturing the affected service, total freeze duration, and budget consumed for the reliability review.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect HoneycombDistributed traces and queries.
  2. 2
    Connect GitLabRepos, MRs, pipelines, registry.
  3. 3
    Connect LinearIssues, projects, cycles, triage.
  4. 4
    Connect SlackChannels, DMs, threads, mentions.
  5. 5
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  6. 6
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  7. 7
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.