ENGINEERING

Open a GitHub issue from a Datadog APM latency-spike alert

When a Datadog APM monitor alerts on a latency regression, fetches the slowest trace, extracts the offending span and SQL.

CategoryEngineering
Enginesim
Difficultyintermediate
Triggerwebhook
Steps5
Setup~15 min

How it runs

The automated pipeline, trigger to output.

  • TriggerDatadog APM latency monitor webhook firesDatadogDatadog
  • ActionFetch exemplar trace and span breakdown from DatadogDatadogDatadog
  • LogicParse dominant span; confirm regression is query-bound
  • ActionResolve owning repo and create labeled GitHub issueGitHubGitHub
  • OutputNotify on-call in Slack with issue linkSlack

What it does

Reacts to a Datadog APM latency monitor alert by pulling the exemplar trace behind the spike, identifying the slowest span and its database call, and opening a GitHub issue in the service's repository with everything an engineer needs to reproduce.

When to use it

When your alerting lives in Datadog but your engineers work issues in GitHub. Use it to close the gap between a paging-grade latency alert and a tracked, repo-scoped bug rather than re-investigating from scratch.

How it works

  1. 1A Datadog monitor webhook fires when APM latency breaches the threshold.
  2. 2Fetch the exemplar trace and span breakdown from the Datadog Trace API for the alerting service.
  3. 3A logic step parses out the dominant span and its `db.statement`, confirming the regression is query-bound and not network.
  4. 4Look up the owning GitHub repo from the service tag mapping.
  5. 5Create a GitHub issue with the slow query, latency delta vs. baseline, affected endpoint, and a Datadog trace permalink, labeled `performance`.
  6. 6Notify the on-call engineer in Slack with the issue link.

Set it up

What you configure once, before turning it on.

  1. 1
    Connect DatadogMetrics, traces, log search.
  2. 2
    Connect GitHubRepos, issues, pull requests, actions.
  3. 3
    Connect SlackChannels, DMs, threads, mentions.
  4. 4
    Set each agent's modelWe leave models unset so you pick the tier — fast + cheap, or top-quality.
  5. 5
    Tune it to your dataEdit the prompts, filters, and field mappings so it matches how your team works.
  6. 6
    Test, then turn it onRun once against a sample, confirm the output, then enable the trigger.

Run this workflow in your colony.

14-day trial. No DevOps. No Sales call. Provisioned in under a minute.