Skip to main content
An incident is a fact about one request, so it needs no baseline and is never subject to the warmup window.

​
The four kinds

The first three are never normal and cannot be switched off.

​
Why 4xx is opt-in

A 4xx is usually the caller’s own bug, which is exactly why an integration tool should be able to catch it. But an endpoint where a 401 is the normal answer would page you all day.
client_error_rate is the share of requests to one endpoint family that must be 4xx before one is reported. Below 20 requests to a family no rate is claimed at all, because the first 4xx is a rate of 1.0 and means nothing.

​
The recording is the reproduction

An incident forces its recording to disk regardless of sampling.rate. Sampling it away would leave you with an alert pointing at something you cannot run.

​
Listing them

incidents is a reader, not a gate: it exits 0 whether or not it found anything. The reproducible until line appears when retention.max_age_hours bounds how long the recording stays on disk. --format json always carries total_matching and truncated. See pikopod incidents.

​
Reproducing one

The incident becomes a scenario pack that arms the same failure in the sandbox and replays the recorded request at it. For a 5xx or an unreachable upstream the fault is armed on method and path, so it reproduces regardless. For a 4xx the body matters, and the body is redacted, so check the pack against what your code actually sends. When the agent runs on another host, pikopod incidents export <fp> writes a bundle that reproduce accepts anywhere. See Reproduce.

​
In an alert

Incidents follow the same dedupe and threshold rules as drift: one alert per fingerprint, after three occurrences in fifteen minutes. See Alerts.