AI & RAG
Grade runbook answers and escalate weak citations
After the RAG assistant answers a remediation question, an LLM judge grades the answer's grounding and citation quality, logs the score.
How it runs
The automated pipeline, trigger to output.
- TriggerWebhook fires with answer, question, and retrieved sourcesHTTP webhook
- ActionLLM judge scores faithfulness and citation accuracyOpenAI
- ActionLog scores and rationale to Postgres evaluation tablePostgres
- LogicBranch when grounding score falls below threshold
- OutputEscalate weak answers to Slack review channelSlack
What it does
Adds a quality gate behind your on-call RAG assistant. Every answer is scored by an LLM judge for whether its claims are actually supported by the cited postmortems, then logged for trend analysis — and any poorly grounded answer is flagged for human review before it misleads a responder.
When to use it
Use it when you need confidence that the assistant is grounded, not hallucinating remediation steps under pressure. Essential before you let an RAG bot influence real incident response.
How it works
- 1A webhook fires when the assistant emits an answer, carrying the question, answer, and retrieved sources.
- 2An LLM judge scores faithfulness, citation accuracy, and completeness against the source text.
- 3Scores and rationale are written to a Postgres evaluation table for dashboards and trends.
- 4If the grounding score falls below threshold, the workflow branches to escalation.
- 5Low-scoring answers are posted to a Slack review channel tagging the on-call docs owner.
Set it up
What you configure once, before turning it on.
- 1Connect HTTP webhookTrigger any URL on agent actions.
- 2Connect OpenAIModels, embeddings, files.
- 3Connect PostgresAny Postgres URL — query, write, migrate.
- 4Connect SlackChannels, DMs, threads, mentions.
- 5Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 6Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 7Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More AI & RAG workflows
RFP and security questionnaire drafter grounded in Coda
Drafts answers to inbound RFP and security questionnaire questions by retrieving approved language from your Coda hub, then files the cited draft for review before a rep sends it.
Detect Breaking API Changes from Spec Diffs and Alert Owners
Compares the new OpenAPI spec against the previous version on each GitLab merge, uses retrieval over the changelog to classify whether changes are breaking.
Grounded reply suggestions for inbound sales email
Reads inbound prospect emails, retrieves the matching answers from your Coda hub.
Coda-grounded sales answer bot with citations in Slack
Reps ask product, pricing, or competitive questions in Slack and get an answer drawn only from your Coda knowledge hub, with links to the exact docs and rows it pulled from.
On-Call Spec Answerer from Dropbox Engineering Corpus
Answers on-call questions posted in a Slack channel by retrieving the most relevant Dropbox engineering specs and replying with a grounded, source-cited answer in the thread.
Agentic Deep-Dive API Researcher for Hard Spec Questions
An agent fielded via webhook that answers multi-part API questions by iteratively searching OpenAPI specs, changelogs, and Confluence runbooks.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.