ENGINEERING
Gate Replicate endpoint promotion on a PR-triggered regression bench
When a model PR is merged, runs the new Replicate model version against a fixed golden eval set and only promotes it to the production endpoint alias if accuracy and latency stay…
How it runs
The automated pipeline, trigger to output.
- TriggerModel-release PR mergedGitHub
- ActionRead candidate version reference from PRGitHub
- ActionRun golden eval set on candidate versionReplicate
- LogicCompare accuracy + p95 latency vs baseline thresholds
- ActionPromote endpoint alias if passingReplicate
- OutputPost go/no-go verdict to SlackSlack
What it does
Turns every merged model-update PR into an automatic go/no-go decision. It runs your held-out eval set through the newly pushed Replicate version, compares the scores against the version currently serving production, and only repoints the production endpoint alias when the candidate is at least as good.
When to use it
Use it when an ML team ships model changes through GitHub and you want a hard quality gate before any new weights reach live inference. It removes the manual "did we check the bench?" step and prevents silent accuracy or latency regressions from reaching users.
How it works
- 1A merged PR labeled `model-release` fires the GitHub trigger with the new version SHA.
- 2The flow reads the candidate Replicate version reference from the PR body.
- 3It runs the golden eval set against the candidate version on Replicate and captures per-case scores plus p95 latency.
- 4A logic step compares candidate accuracy and latency to the live version's recorded baseline against your thresholds.
- 5If it passes, it updates the production endpoint alias to the candidate version on Replicate.
- 6It posts the verdict, score delta, and decision to the release Slack channel.
Set it up
What you configure once, before turning it on.
- 1Connect GitHubRepos, issues, pull requests, actions.
- 2Connect ReplicateImage, video, and model inference.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Upstream Release to Notion Upgrade Brief
When a watched package publishes a new release, fetches the release notes, maps them to the internal modules that depend on it.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.

Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.
