ENGINEERING

Gate Replicate endpoint promotion on a PR-triggered regression bench

When a model PR is merged, runs the new Replicate model version against a fixed golden eval set and only promotes it to the production endpoint alias if accuracy and latency stay…

CategoryEngineering
Enginesim
Difficultyintermediate
Triggerevent
Steps6
Setup~15 min

How it runs

The automated pipeline, trigger to output.

  • TriggerModel-release PR mergedGitHubGitHub
  • ActionRead candidate version reference from PRGitHubGitHub
  • ActionRun golden eval set on candidate versionReplicateReplicate
  • LogicCompare accuracy + p95 latency vs baseline thresholds
  • ActionPromote endpoint alias if passingReplicateReplicate
  • OutputPost go/no-go verdict to SlackSlack

What it does

Turns every merged model-update PR into an automatic go/no-go decision. It runs your held-out eval set through the newly pushed Replicate version, compares the scores against the version currently serving production, and only repoints the production endpoint alias when the candidate is at least as good.

When to use it

Use it when an ML team ships model changes through GitHub and you want a hard quality gate before any new weights reach live inference. It removes the manual "did we check the bench?" step and prevents silent accuracy or latency regressions from reaching users.

How it works

  1. 1A merged PR labeled `model-release` fires the GitHub trigger with the new version SHA.
  2. 2The flow reads the candidate Replicate version reference from the PR body.
  3. 3It runs the golden eval set against the candidate version on Replicate and captures per-case scores plus p95 latency.
  4. 4A logic step compares candidate accuracy and latency to the live version's recorded baseline against your thresholds.
  5. 5If it passes, it updates the production endpoint alias to the candidate version on Replicate.
  6. 6It posts the verdict, score delta, and decision to the release Slack channel.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect GitHubRepos, issues, pull requests, actions.
  2. 2
    Connect ReplicateImage, video, and model inference.
  3. 3
    Connect SlackChannels, DMs, threads, mentions.
  4. 4
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  5. 5
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  6. 6
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.