ENGINEERING
Nightly Flaky-Test Detector that Builds a Deflake Board
Runs the test suite multiple times each night in CI, identifies tests with inconsistent results across runs.
How it runs
The automated pipeline, trigger to output.
- TriggerNightly schedule
- ActionDispatch GitHub workflow to run suite N times on pinned commitGitHub
- LogicCompare outcomes across runs and compute flip counts
- LogicRank flakes and merge with existing board
- ActionUpsert ranked Linear ticket per flaky testLinear
- OutputPost top-flakes digest to SlackSlack
What it does
It actively hunts for flakes rather than waiting for them to appear. Each night it kicks off repeated runs of the suite against the same commit, collects which tests produced different outcomes across identical runs, and feeds a Linear board where each flaky test is a ticket ranked by flip frequency, so the team always knows the worst offenders.
When to use it
Use it when flakes hurt but rarely reproduce on a single CI run. Multi-run detection surfaces low-frequency flakes that re-run-and-move-on never catches, and the ranked board makes triage obvious.
How it works
- 1A nightly schedule triggers the run.
- 2A GitHub action dispatches a workflow that executes the suite N times on the pinned commit and collects the per-run results.
- 3A logic step compares outcomes across runs and computes a flip count for each non-deterministic test.
- 4A logic step ranks flaky tests and merges with existing board entries.
- 5A Linear action upserts a ticket per flaky test with its flip frequency and updates priority.
- 6A Slack digest posts the top flakes for the morning standup.
Set it up
What you configure once, before turning it on.
- 1Connect GitHubRepos, issues, pull requests, actions.
- 2Connect LinearIssues, projects, cycles, triage.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Upstream Release to Notion Upgrade Brief
When a watched package publishes a new release, fetches the release notes, maps them to the internal modules that depend on it.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.