DATA OPS
Nightly Google Drive CSV sweep with Postgres upsert and drift alert
On a nightly schedule, pulls new CSVs from a Google Drive folder, validates them, upserts rows into Postgres by primary key, and pages on-call via PagerDuty if the schema drifts.
How it runs
The automated pipeline, trigger to output.
- TriggerNightly schedule fires
- ActionList and download new CSVs from Drive folderGoogle Drive
- LogicCheck headers against contract; branch on drift
- ActionUpsert conforming rows into Postgres by primary keyPostgres
- OutputOpen PagerDuty incident if schema driftedPagerDuty
What it does
Runs every night to collect any CSVs added to a shared Google Drive folder, validate each against the expected schema, and upsert the rows into a Postgres table keyed on a primary identifier so re-delivered files update rather than duplicate. If a file's header layout drifts from the contract, it raises a PagerDuty incident instead of loading bad data.
When to use it
Use it when teams export reports to a Drive folder on their own cadence and you need the warehouse table refreshed once a day, idempotently, with a hard stop when the source format changes unexpectedly.
How it works
- 1A nightly schedule trigger starts the run.
- 2The pipeline lists the Drive folder and downloads files newer than the last successful run.
- 3A logic step compares each file's headers to the contract; on drift it branches to alerting.
- 4Conforming files are upserted into Postgres by primary key.
- 5If drift was detected, a PagerDuty incident is opened with the offending file and diff.
Set it up
What you configure once, before turning it on.
- 1Connect Google DriveDocs, sheets, slides, files.
- 2Connect PostgresAny Postgres URL — query, write, migrate.
- 3Connect PagerDutyIncidents, on-call, escalations.
- 4Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
- 5Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
- 6Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.
More Data Ops workflows
Weekly BigQuery Cost Trend Sheet and Exec Digest
Compiles week-over-week BigQuery scheduled-query cost by owner and dataset into a Google Sheet with trend columns.
Daily BigQuery Scheduled-Query Cost Attribution to Owners
Each morning, totals the prior day's on-demand bytes-billed per scheduled query, maps each query to its owner from a label, and posts a per-owner cost leaderboard to Slack.
BigQuery Per-Team Budget Breach Alert to PagerDuty
Tracks month-to-date BigQuery scheduled-query spend per team and, when a team crosses its monthly budget, pages the team's on-call in PagerDuty and snapshots the spend breakdown…
dbt source freshness watcher with severity-routed alerts
Checks Snowflake loaded-at timestamps against each dbt source's freshness SLA, then routes warnings to Slack and hard breaches to a PagerDuty incident so stale data never…
dbt orphan model detector with Linear cleanup tickets
Scans your dbt manifest for models that no other model, exposure, or BI tool consumes.
Raw Sensor Telemetry Archive to BigQuery
Captures every incoming building sensor reading via webhook, normalizes the payload into a consistent schema.
Run it inside a business
This workflow drops into a full company template. Import the org, and this is one of the playbooks its agents run.
Run this workflow in your colony.
14-day trial. No DevOps. No Sales call. Provisioned in under a minute.