AI & RAG

Weekly Audit of Unanswerable Policy Questions and Coverage Gaps

Reviews the week's contract Q&A queries that returned weak or no grounding, clusters them into themes, and posts a prioritized report of policy library gaps to the team.

CategoryAI & RAG
Enginesim
Difficultyintermediate
Triggerschedule
Steps5
Setup~15 min

How it runs

The automated pipeline, trigger to output.

  • TriggerWeekly schedule fires
  • ActionPull low-confidence and no-match Q&A logs from SupabaseSupabaseSupabase
  • ActionCluster failed questions into themes with OpenAIOpenAI
  • LogicRank gaps by frequency and recency
  • OutputPost prioritized coverage-gap report to SlackSlack

What it does

Closes the loop on your grounded Q&A system by surfacing what it could not answer. On a schedule, it pulls logged questions that retrieved low-confidence or empty matches, groups them by topic, and produces a ranked list of missing or thin policy areas your team should document in Confluence.

When to use it

Use it when you run a contract clause Q&A assistant and want to systematically improve coverage instead of guessing what to write next. Ideal for legal-ops owners who maintain the policy library and need evidence-based priorities.

How it works

  1. 1A weekly schedule triggers the flow.
  2. 2It queries Supabase for the past week's Q&A logs where retrieval confidence fell below threshold or returned nothing.
  3. 3OpenAI clusters the failed questions into themes and drafts a suggested policy topic for each gap.
  4. 4A logic step ranks clusters by frequency and recency to set priority.
  5. 5The ranked gap report posts to a Slack channel for the policy owner to triage into new Confluence pages.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect SupabaseTables, auth, storage, edge functions.
  2. 2
    Connect OpenAIModels, embeddings, files.
  3. 3
    Connect SlackChannels, DMs, threads, mentions.
  4. 4
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  5. 5
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  6. 6
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.