AI & RAG

Verify a rewritten article actually fixes its retrieval misses after publish

When a help article is updated in Intercom, replays the historical user phrasings that previously missed it and confirms retrieval now ranks the article first; if it still…

CategoryAI & RAG
Enginesim
Difficultyadvanced
Triggerwebhook
Steps5
Setup~25 min

How it runs

The automated pipeline, trigger to output.

  • TriggerIntercom article-updated webhookIntercomIntercom
  • ActionLoad article's historical missed phrasingsPostgreSQLPostgres
  • ActionRe-run retrieval scoring on new contentOpenAI
  • LogicBranch on whether all phrasings now rank top
  • OutputReopen Linear ticket if still missingLinearLinear

What it does

It closes the loop on article fixes. When an article is published or edited in Intercom, the workflow replays the real user phrasings that previously failed to retrieve it and checks whether the updated content now ranks first. This proves the rewrite worked instead of assuming it did.

When to use it

Use it right after you ship article rewrites driven by your retrieval audits. It prevents 'fixed' articles from silently still missing the queries they were supposed to catch, and it keeps your Linear board honest.

How it works

  1. 1An Intercom webhook fires when an article is published or updated.
  2. 2Load that article's historical missed phrasings from Postgres.
  3. 3An OpenAI step re-runs retrieval scoring for each phrasing against the updated content.
  4. 4A logic branch checks whether the article now ranks top for every phrasing.
  5. 5If all pass, mark the Postgres record resolved and the loop ends.
  6. 6If any still miss, reopen the Linear ticket with the remaining failing phrasings and current ranks.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect IntercomConversations, contacts, articles.
  2. 2
    Connect PostgresAny Postgres URL — query, write, migrate.
  3. 3
    Connect OpenAIModels, embeddings, files.
  4. 4
    Connect LinearIssues, projects, cycles, triage.
  5. 5
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  6. 6
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  7. 7
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.