AI & RAG

Runbook Coverage Gap Detector from Unanswered Questions

Aggregates questions the answer bot couldn't ground, clusters them by topic, and files a Linear issue per gap proposing which runbook section is missing or thin.

CategoryAI & RAG
Enginesim
Difficultyintermediate
Triggerschedule
Steps5
Setup~15 min

How it runs

The automated pipeline, trigger to output.

  • TriggerDaily schedule triggers gap analysis
  • ActionRead logged unanswered and low-confidence questions from PostgresPostgreSQLPostgres
  • ActionCluster questions into missing-topic themes with OpenAIOpenAI
  • LogicFilter out clusters below the frequency threshold
  • OutputCreate a Linear issue per gap with sample questions and a suggested sectionLinearLinear

What it does

Closes the loop on what your runbooks don't cover. It collects every question the grounded bot answered with low confidence or 'no relevant source,' clusters the recurring themes, and turns each cluster into an actionable documentation task so the wiki improves where engineers actually struggle.

When to use it

Use it when you want your runbook backlog driven by real demand signals instead of guesswork, and to prove the value of new docs by showing how many real questions a gap caused.

How it works

  1. 1A daily schedule triggers the analysis.
  2. 2The flow reads logged unanswered or low-confidence questions from Postgres.
  3. 3OpenAI clusters them into topics and summarizes the apparent missing-knowledge theme for each cluster.
  4. 4A branch drops clusters below a frequency threshold to avoid noise.
  5. 5For each remaining cluster, a Linear issue is created describing the gap, sample questions, and a suggested runbook section to author.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect PostgresAny Postgres URL — query, write, migrate.
  2. 2
    Connect OpenAIModels, embeddings, files.
  3. 3
    Connect LinearIssues, projects, cycles, triage.
  4. 4
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  5. 5
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  6. 6
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.