ENGINEERING
Weekly Flaky-Test Trend Report from BigQuery
On a weekly schedule, queries your CI history warehouse in BigQuery for the top flaky specs, ranks them by failure rate and developer hours lost to retries.
How it runs
The automated pipeline, trigger to output.
- TriggerWeekly Monday schedule
- ActionQuery 30-day flakiness aggregates in BigQueryBigQuery
- LogicFilter above threshold and rank top 20
- ActionFormat leaderboard with week-over-week deltas
- OutputDeliver leaderboard to engineering-leads SlackSlack
What it does
Produces a weekly flakiness leaderboard from warehoused pipeline data: which tests fail most, their pass/fail ratio, the suites they live in, and an estimate of retry minutes burned. It gives leads a data-backed view instead of anecdotes about "that one annoying test."
When to use it
Use it when your CI events already land in BigQuery and you want a recurring management-facing signal to prioritize test-stability work in sprint planning.
How it works
- 1A Monday-morning schedule trigger fires.
- 2A BigQuery step runs an aggregation over the last 30 days of pipeline runs, computing per-test failure rate, run count, and retry-time estimate.
- 3A logic step keeps only tests above the flakiness threshold and sorts them into a top-20 ranking.
- 4The flow formats the ranking into a readable table with deltas versus the prior week.
- 5A Slack message delivers the leaderboard to the engineering-leads channel with a note on the worst regressions.
Set it up
What you configure once, before turning it on.
- 1Connect BigQueryDatasets, queries, schemas.
- 2Connect SlackChannels, DMs, threads, mentions.
- 3Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 4Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 5Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Engineering workflows
Gate breaking API PRs behind downstream consumer acknowledgement
When a PR introduces a breaking contract change, comments the impact summary back on the PR, applies a blocking label.
Publish a versioned API changelog to Confluence on each release tag
On a new semver release tag, gathers the contract changes since the last release and writes a clean.
Agent reviews model-license fit and suggests compliant swaps on the PR
When a PR adds a Hugging Face model, an agent reads the model card and license, judges fit against your commercial-use policy.
Upgrade Impact Router to Module Code Owners
Maps a dependency-bump PR's affected modules to their CODEOWNERS, then DMs each owner on Slack with only the changelog slice that touches code they own.
Re-Voice IVR Prompts on Phone-Tree Config Merge
When a phone-tree config change merges in GitHub, regenerates the ElevenLabs audio for any prompt whose script changed in the diff and opens a follow-up PR adding the new audio…
Upstream Release to Notion Upgrade Brief
When a watched package publishes a new release, fetches the release notes, maps them to the internal modules that depend on it.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.