DATA OPS
BigQuery dropped/renamed column sentinel with PagerDuty incident
Detects when a column is dropped or renamed in your governed BigQuery datasets and, because that breaks downstream queries hard, pages the on-call via PagerDuty and posts…
How it runs
The automated pipeline, trigger to output.
- TriggerSchedule triggers periodic check
- ActionList live column inventory from BigQueryBigQuery
- LogicFind columns removed or renamed since last run
- LogicExit if no columns missing
- ActionOpen PagerDuty incident for the on-callPagerDuty
- OutputPost impact summary to Slack on-call channelSlack
What it does
It guards against the most destructive kind of drift: a column disappearing. Each run it lists every column across your governed BigQuery datasets, compares the set to the prior run, and surfaces columns that vanished or were renamed. Because a missing column throws errors rather than producing wrong-but-quiet results, this one escalates: it opens a PagerDuty incident and drops the affected tables into Slack.
When to use it
Use it on production datasets that feed customer-facing reporting or revenue jobs, where a dropped column means failed queries within minutes and you need the on-call engineer woken up, not a backlog ticket.
How it works
- 1A schedule triggers the check every few hours.
- 2Query `INFORMATION_SCHEMA.COLUMNS` across the governed BigQuery datasets to get the live column inventory.
- 3Compare against the previous inventory to find columns that were removed or renamed.
- 4If nothing is missing, save the inventory and exit quietly.
- 5Open a PagerDuty incident describing which tables lost which columns.
- 6Post the same impact summary to the data on-call Slack channel and persist the new inventory.
Set it up
What you configure once, before turning it on.
- 1Connect BigQueryDatasets, queries, schemas.
- 2Connect PagerDutyIncidents, on-call, escalations.
- 3Connect SlackChannels, DMs, threads, mentions.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Data Ops workflows
Snowflake column type-drift sentinel with Linear fix ticket
Snapshots the data types of every column in your tracked Snowflake schemas on a schedule, diffs against the last snapshot.
Daily BigQuery Scheduled-Query Cost Attribution to Owners
Each morning, totals the prior day's on-demand bytes-billed per scheduled query, maps each query to its owner from a label, and posts a per-owner cost leaderboard to Slack.
PR-time Snowflake schema contract check on dbt model changes
When a pull request changes a dbt model, it compares the model's declared output columns against the live Snowflake table it will replace and blocks the merge with a GitHub check…
Agent-triaged warehouse drift with impact analysis and runbook update
On a webhook from your warehouse audit log, an agent investigates the changed column, traces which downstream models and dashboards depend on it.
Cross-warehouse replication schema mismatch reconciler
Compares the column shape of mirrored tables between BigQuery and Snowflake and, when a replicated table has drifted out of sync between the two, opens an Asana task for the data…
Weekly BigQuery Cost Trend Sheet and Exec Digest
Compiles week-over-week BigQuery scheduled-query cost by owner and dataset into a Google Sheet with trend columns.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.