AI & RAG

Weekly Runbook Coverage Gap Reporter

Each week it reviews unanswered or low-confidence on-call questions from Discord, checks them against existing Confluence runbooks.

CategoryAI & RAG
Enginesim
Difficultyintermediate
Triggerschedule
Steps6
Setup~15 min

How it runs

The automated pipeline, trigger to output.

  • TriggerWeekly schedule fires
  • ActionPull past week of on-call questionsDiscordDiscord
  • ActionScore each against Confluence coverageConfluenceConfluence
  • LogicKeep uncovered questions and cluster
  • ActionRank gaps and suggest runbook titlesOpenAI
  • OutputPost coverage-gap digest to DiscordDiscordDiscord

What it does

Finds the holes in your runbook library. It scans the past week of on-call questions in Discord, attempts retrieval against Confluence for each one, and flags the questions that returned no confident match. The output is a ranked list of missing or weak runbook topics, posted as a weekly digest.

When to use it

Use it when you want your documentation to improve from real on-call demand rather than guesswork. Ideal for ops leads who run a weekly review and want a concrete, evidence-backed list of what to document next, ordered by how often engineers actually asked.

How it works

  1. 1A weekly schedule triggers the run.
  2. 2The flow pulls the past seven days of questions from the on-call Discord channel.
  3. 3Each question is tested against Confluence retrieval to score coverage.
  4. 4A logic step keeps only questions with no confident runbook match and clusters duplicates.
  5. 5An OpenAI step ranks the gaps by frequency and drafts suggested runbook titles.
  6. 6A digest of uncovered topics is posted to the team's ops channel in Discord.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect DiscordCommunity channels + voice + bots.
  2. 2
    Connect ConfluenceSpaces, pages, blueprints.
  3. 3
    Connect OpenAIModels, embeddings, files.
  4. 4
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  5. 5
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  6. 6
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.