ENGINEERING
Nightly Replicate inference drift watch with auto-rollback
Each night replays a canary prompt suite against the live Replicate endpoint, detects output drift or latency regression versus a stored baseline.
How it runs
The automated pipeline, trigger to output.
- TriggerNightly schedule
- ActionReplay canary suite on live versionReplicate
- LogicDetect drift / latency regression vs baseline
- ActionRoll endpoint alias back to last-known-goodReplicate
- ActionOpen incident with failing casesPagerDuty
- OutputPost drift summary to SlackSlack
What it does
Guards a production Replicate endpoint against silent drift over time. Every night it replays a fixed canary suite, scores the outputs against a stored baseline, and if quality or latency has degraded past tolerance it automatically rolls the endpoint alias back to the last-known-good version and escalates.
When to use it
Use it for endpoints that must stay stable between releases, where upstream model changes, autoscaling cold starts, or config drift could quietly degrade results. It catches regressions that no PR introduced.
How it works
- 1A nightly schedule trigger starts the run.
- 2The flow replays the canary prompt suite against the current production Replicate version and records scores and p95 latency.
- 3A logic step diffs the run against the stored baseline to flag accuracy drift or latency regression.
- 4If a regression is detected, it repoints the Replicate endpoint alias back to the last-known-good version.
- 5It opens a PagerDuty incident with the failing cases and the rollback action taken.
- 6It posts a summary with the drift chart link to Slack.
Set it up
What you configure once, before turning it on.
- 1Connect ReplicateImage, video, and model inference.
- 2Connect PagerDutyIncidents, on-call, escalations.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Scan for deprecated endpoints and email consumers a weekly sunset countdown
On a weekly schedule, scans the OpenAPI spec for endpoints marked deprecated with a sunset date, and emails each consuming team a countdown of how many days remain before removal.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.