AI & RAG

Weekly Stale-Spec Audit from On-Call Questions

Weekly, analyzes the on-call questions the assistant couldn't answer well, cross-references them against the Dropbox spec corpus.

CategoryAI & RAG
Enginesim
Difficultyadvanced
Triggerschedule
Steps5
Setup~25 min

How it runs

The automated pipeline, trigger to output.

  • TriggerWeekly scheduled audit run
  • ActionPull low-confidence questions from Postgres answer logPostgreSQLPostgres
  • ActionCluster questions and identify uncovered topicsOpenAI
  • LogicVerify gaps against existing Dropbox spec pathsDropboxDropbox
  • OutputFile a Linear ticket per documentation gapLinearLinear

What it does

Closes the loop between what on-call engineers ask and what the docs actually cover. Each week it reviews the questions that triggered low-confidence or escalated answers, clusters them by topic, checks the Dropbox spec corpus for missing or thin coverage, and opens Linear tickets describing each documentation gap.

When to use it

Run it to drive continuous improvement of your spec corpus. Over time the most-asked-but-undocumented topics surface as a prioritized backlog instead of living only in Slack history.

How it works

  1. 1A weekly scheduled trigger fires.
  2. 2The week's unanswered or low-confidence questions are pulled from the Postgres answer log.
  3. 3OpenAI clusters them into themes and judges which the Dropbox corpus fails to cover.
  4. 4Each confirmed gap is checked against existing Dropbox spec paths to avoid false positives.
  5. 5A Linear ticket is filed per gap with the example questions and suggested doc to write.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect PostgresAny Postgres URL — query, write, migrate.
  2. 2
    Connect OpenAIModels, embeddings, files.
  3. 3
    Connect DropboxFiles and folders.
  4. 4
    Connect LinearIssues, projects, cycles, triage.
  5. 5
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  6. 6
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  7. 7
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.