AI & RAG
Weekly Audit of Answer-Bot Grounding and Citations
Samples the past week of answer-bot responses, re-verifies each cited claim against the frozen corpus with an LLM judge.
How it runs
The automated pipeline, trigger to output.
- TriggerWeekly scheduled audit run
- ActionRead sampled answers and cited chunk IDs from Supabase logSupabase
- ActionScore citation faithfulness with OpenAI judgeOpenAI
- LogicCollect answers below the faithfulness threshold
- OutputPost flagged-answers report to SlackSlack
What it does
Keeps your grounded answer bots honest. Each week it pulls a sample of logged answers, re-checks whether every cited passage truly supports the claim it backs, and scores each answer for faithfulness. Answers that drift from their sources are flagged so Compliance can review before users are misled.
When to use it
Use it as an ongoing quality gate once an answer bot is live — to catch citation hallucinations, stale references, and over-confident answers that should have been refusals. Pairs well with the corpus-freeze indexer for a closed audit loop.
How it works
- 1A weekly scheduled run starts the audit.
- 2A sample of recent answers and their cited chunk IDs is read from the Supabase answer log.
- 3For each answer, the cited passages are re-fetched and an OpenAI judge scores whether they genuinely support the claim.
- 4A logic step collects answers scoring below the faithfulness bar.
- 5A Slack report posts the flagged answers with scores and links for human review.
Set it up
What you configure once, before turning it on.
- 1Connect SupabaseTables, auth, storage, edge functions.
- 2Connect OpenAIModels, embeddings, files.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More AI & RAG workflows
Publish a Grounded API FAQ Page to Confluence Weekly
Each week, clusters the top unanswered or repeated API questions, generates spec-grounded answers with citations.
Detect Breaking API Changes from Spec Diffs and Alert Owners
Compares the new OpenAPI spec against the previous version on each GitLab merge, uses retrieval over the changelog to classify whether changes are breaking.
Pre-meeting prep brief grounded in Coda and CRM
Before each booked sales meeting, builds a one-page prep brief by combining the account's HubSpot context with grounded talking points and objection responses pulled from your…
Coda-grounded sales answer bot with citations in Slack
Reps ask product, pricing, or competitive questions in Slack and get an answer drawn only from your Coda knowledge hub, with links to the exact docs and rows it pulled from.
Weekly knowledge-gap digest from unanswered rep questions
Each week, scans rep questions the answer bot couldn't ground in Coda, clusters the recurring gaps.
RFP and security questionnaire drafter grounded in Coda
Drafts answers to inbound RFP and security questionnaire questions by retrieving approved language from your Coda hub, then files the cited draft for review before a rep sends it.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.