AI & RAG

Runbook Gap Detector from Unanswered On-Call Questions

When the answer bot can't ground a metric question in any existing Confluence runbook, it files a Linear ticket capturing the question, the Datadog evidence it found.

CategoryAI & RAG
Enginesim
Difficultyintermediate
Triggerchat
Steps6
Setup~15 min

How it runs

The automated pipeline, trigger to output.

  • TriggerEngineer asks the bot a metric question in SlackSlack
  • ActionSearch Confluence for a runbook covering the metricConfluenceConfluence
  • LogicBranch: exit if a runbook exists, continue if the metric is undocumented
  • ActionQuery Datadog for the metric's value and baseline as evidenceDatadogDatadog
  • ActionDraft a runbook stub from the question and baseline data (OpenAI)OpenAI
  • OutputFile a Linear ticket with the stub and evidence for documentation triageLinearLinear

What it does

This workflow catches the moments when on-call knowledge is missing. When an engineer asks the bot about a metric and no Confluence runbook covers it, instead of guessing, the bot files a Linear ticket. The ticket includes the original question, the Datadog baseline it was able to compute on its own, and an OpenAI-drafted runbook stub the team can review and publish. Over time it turns every unanswered question into documented knowledge.

When to use it

Use it to close the long tail of undocumented metrics. It's ideal for teams whose runbooks lag behind their dashboards and who want every "we should write that down" moment captured automatically.

How it works

  1. 1A Slack question to the bot triggers the flow.
  2. 2Confluence is searched for a runbook covering the referenced metric.
  3. 3A logic branch checks whether a relevant runbook was found; if yes, the flow exits (the answer bot handles it).
  4. 4When no runbook exists, Datadog is queried for the metric's value and baseline as supporting evidence.
  5. 5OpenAI drafts a runbook stub with the question, the baseline data, and suggested healthy ranges.
  6. 6A Linear ticket is created with the stub and evidence, tagged for documentation triage.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect SlackChannels, DMs, threads, mentions.
  2. 2
    Connect ConfluenceSpaces, pages, blueprints.
  3. 3
    Connect DatadogMetrics, traces, log search.
  4. 4
    Connect OpenAIModels, embeddings, files.
  5. 5
    Connect LinearIssues, projects, cycles, triage.
  6. 6
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  7. 7
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  8. 8
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.