Prometheus Alert Rule Generator

Demo: API error-rate alert with context

A useful API alert combines an expression with enough context for the responder. The annotation should say what is happening and where to start, not just repeat the metric name.

• Use a ratio when possible so traffic volume changes do not distort the alert.
• Add a for duration to avoid paging on short deploy blips.
• Include service, severity, and runbook labels consistently.

groups:
  - name: api.rules
    rules:
      - alert: ApiHighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: page
          service: checkout
        annotations:
          summary: "Checkout API 5xx error ratio is above 5%"
          runbook: "Check recent deploys, dependency errors, and upstream payment latency."

The generator should produce YAML that is readable enough to review before promtool validation.

Alert tuning: make generated rules actionable

Generated alert rules should be treated as a starting point. Before enabling them, tune thresholds, for durations, labels, and annotations so the alert tells an on-call engineer what is broken and where to look.

• Use for: to avoid paging on short spikes.
• Keep labels stable enough for routing and deduplication.
• Write annotations with dashboard links, runbook links, and the exact PromQL expression meaning.

- alert: ApiHighErrorRate
  expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
  for: 10m
  labels:
    severity: page