AI & RAG

Low-confidence answer gap logger and triage

Captures every runbook question the answer bot couldn't confidently ground, clusters recurring gaps.

CategoryAI & RAG
Enginesim
Difficultyintermediate
Triggerwebhook
Steps5
Setup~15 min

How it runs

The automated pipeline, trigger to output.

  • TriggerWebhook receives low-confidence answer eventsHTTP webhook
  • ActionStore question and retrieval score in Postgres gaps tablePostgreSQLPostgres
  • ActionCluster accumulated gaps with OpenAI on a daily runOpenAI
  • LogicPromote clusters above the frequency threshold
  • OutputFile or update a Linear ticket per missing topicLinearLinear

What it does

Turns the answer bot's failures into a prioritized docs backlog. Each time the bot abstains or answers with low confidence, the question is logged; recurring themes are clustered so the docs team writes the pages users actually need instead of guessing.

When to use it

Use it once your grounded answer bot is live and you want a data-driven way to grow the runbook, closing the most-asked, least-documented gaps first rather than expanding the wiki at random.

How it works

  1. 1A webhook receives every low-confidence or abstained answer event the bot emits.
  2. 2The question and its top retrieval score are stored in a Postgres gaps table.
  3. 3On a daily schedule, accumulated gaps are embedded and clustered with OpenAI to group near-duplicate questions.
  4. 4A threshold step promotes only clusters above a frequency cutoff to actionable status.
  5. 5For each promoted cluster, the workflow files or updates a Linear ticket describing the missing runbook topic and how often it was asked.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect HTTP webhookTrigger any URL on agent actions.
  2. 2
    Connect PostgresAny Postgres URL — query, write, migrate.
  3. 3
    Connect OpenAIModels, embeddings, files.
  4. 4
    Connect LinearIssues, projects, cycles, triage.
  5. 5
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  6. 6
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  7. 7
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.