AI & RAG

Audit answer-bank citation freshness against live Confluence docs

Walks every entry in the Postgres answer bank, re-checks the Confluence page each answer cites, and flags or quarantines entries whose source doc changed, moved, or was deleted…

CategoryAI & RAG
Enginesim
Difficultyadvanced
Triggerschedule
Steps7
Setup~25 min

How it runs

The automated pipeline, trigger to output.

  • TriggerSchedule: nightly
  • ActionPage through active answer_bank entriesPostgreSQLPostgres
  • ActionFetch cited Confluence page + versionConfluenceConfluence
  • LogicCompare stored vs current version hash
  • ActionLLM judges if answer still matches sourceOpenAI
  • ActionFlag failing entries as stale in PostgresPostgreSQLPostgres
  • OutputPost freshness summary to SlackSlack

What it does

Keeps your RAG answer bank honest. Each stored answer is linked to a Confluence source page; this workflow periodically re-reads those pages, compares the current content against what the answer claims, and uses an LLM to decide whether the citation is still valid. Entries backed by changed or missing docs are marked stale so retrieval can skip them.

When to use it

When support docs evolve faster than your knowledge base and you risk serving confidently wrong answers. Run nightly to guarantee every retrievable answer still points at a live, consistent source.

How it works

  1. 1A nightly schedule starts the audit.
  2. 2The workflow pages through all active entries in the Postgres answer bank.
  3. 3For each entry, it fetches the cited Confluence page by ID and reads its current version and last-updated date.
  4. 4A logic step compares stored vs. current version hashes to find pages that changed or 404'd.
  5. 5For changed pages, an LLM judges whether the answer still matches the source.
  6. 6Entries that fail verification are flagged `stale` in Postgres and a freshness summary is posted to a Slack review channel.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect PostgresAny Postgres URL — query, write, migrate.
  2. 2
    Connect ConfluenceSpaces, pages, blueprints.
  3. 3
    Connect OpenAIModels, embeddings, files.
  4. 4
    Connect SlackChannels, DMs, threads, mentions.
  5. 5
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  6. 6
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  7. 7
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.