AI & RAG
RAG Answer Faithfulness Judge with Hallucination Escalation
After the bot answers, an LLM judge scores whether each claim is supported by the cited runbook sections, logs the score.
How it runs
The automated pipeline, trigger to output.
- TriggerAnswer-generated event fires from the answer bot
- ActionLoad question, answer, and cited source sections from PostgresPostgres
- ActionJudge faithfulness and citation accuracy with OpenAIOpenAI
- ActionWrite the faithfulness score and verdict back to PostgresPostgres
- LogicBranch on the faithfulness threshold
- OutputEscalate ungrounded answers to a Slack review channelSlack
What it does
Adds a quality gate behind your grounded answer bot. Each time an answer is produced, this flow re-checks it against the exact source sections that were retrieved, scoring faithfulness and citation accuracy. Unsupported claims are flagged so a hallucinated rollback step never goes unnoticed.
When to use it
Use it when runbook answers carry operational risk and you need an auditable record of how grounded each response was, plus a fast path to catch and correct the rare ungrounded answer.
How it works
- 1An answer-generated event (emitted by the answer bot) triggers the judge.
- 2The flow loads the question, the generated answer, and the cited source sections from Postgres.
- 3OpenAI acts as a judge, scoring faithfulness, citation correctness, and noting any unsupported claim.
- 4The score and verdict are written back to Postgres for trend tracking.
- 5A branch escalates any answer below the faithfulness threshold to a Slack review channel with the offending claims highlighted.
Set it up
What you configure once, before turning it on.
- 1Connect PostgresAny Postgres URL — query, write, migrate.
- 2Connect OpenAIModels, embeddings, files.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More AI & RAG workflows
Publish a Grounded API FAQ Page to Confluence Weekly
Each week, clusters the top unanswered or repeated API questions, generates spec-grounded answers with citations.
Detect Breaking API Changes from Spec Diffs and Alert Owners
Compares the new OpenAPI spec against the previous version on each GitLab merge, uses retrieval over the changelog to classify whether changes are breaking.
Pre-meeting prep brief grounded in Coda and CRM
Before each booked sales meeting, builds a one-page prep brief by combining the account's HubSpot context with grounded talking points and objection responses pulled from your…
Coda-grounded sales answer bot with citations in Slack
Reps ask product, pricing, or competitive questions in Slack and get an answer drawn only from your Coda knowledge hub, with links to the exact docs and rows it pulled from.
Weekly knowledge-gap digest from unanswered rep questions
Each week, scans rep questions the answer bot couldn't ground in Coda, clusters the recurring gaps.
RFP and security questionnaire drafter grounded in Coda
Drafts answers to inbound RFP and security questionnaire questions by retrieving approved language from your Coda hub, then files the cited draft for review before a rep sends it.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.