AI & RAG

Pull request runbook gap detector

On each opened PR, checks whether the changed code is covered by an existing runbook page and comments on the PR when a procedure is missing or its citation has gone stale.

CategoryAI & RAG
Enginesim
Difficultyadvanced
Triggerwebhook
Steps5
Setup~25 min

How it runs

The automated pipeline, trigger to output.

  • TriggerGitHub PR opened webhook with changed filesGitHubGitHub
  • LogicFilter to operationally significant code paths
  • ActionMatch paths against runbook vector index in PostgresPostgreSQLPostgres
  • LogicClassify each path as covered, stale, or undocumentedOpenAI
  • OutputPost consolidated doc-gap comment on the PRGitHubGitHub

What it does

Closes the loop between code and docs at review time. When a PR touches operationally significant paths, it asks the runbook index whether a matching procedure exists, and if the docs are missing or outdated it leaves an actionable comment on the PR so the author fixes it before merge.

When to use it

Use it to enforce docs-as-you-go on critical services, when reviewers routinely forget to check that runbooks keep pace with shipping code and you want the reminder to land directly in the PR.

How it works

  1. 1A GitHub PR-opened webhook delivers the changed file list.
  2. 2A filter keeps only paths mapped to operational concerns (deploy, migrations, infra).
  3. 3Each in-scope path is matched against the runbook vector index in Postgres to find covering pages.
  4. 4A decision step classifies each path as covered, stale, or undocumented using OpenAI plus the page's freshness stamp.
  5. 5The workflow posts a single consolidated review comment on the PR listing gaps and stale citations to address.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect GitHubRepos, issues, pull requests, actions.
  2. 2
    Connect PostgresAny Postgres URL — query, write, migrate.
  3. 3
    Connect OpenAIModels, embeddings, files.
  4. 4
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  5. 5
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  6. 6
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.