Telemetry Cost-Spike Triage from PagerDuty Incident

When an observability-cost budget alarm escalates to PagerDuty, this agent does the first triage for the on-call engineer. It looks at the moment the spike began, finds the Datadog metric or Honeycomb dataset whose cardinality or volume jumped, and writes a plain-language root cause plus a candidate drop or aggregate rule, all attached to the incident before a human even opens their laptop.

Use it when telemetry cost spikes page someone and the on-call wastes the first twenty minutes figuring out which service shipped a noisy new tag. It compresses triage into a single incident note.

1. 1A PagerDuty incident with a telemetry-cost service tag triggers the run.
2. 2The agent reads the incident's start time and any included query context.
3. 3It queries both Datadog and Honeycomb around that window to locate the metric or dataset with the abnormal cardinality or volume jump.
4. 4A logic step picks the single most likely culprit and drafts a targeted drop or aggregate recommendation.
5. 5It appends the root-cause summary and proposed rule as a note on the PagerDuty incident.