Skip to main content
An incident is a fact about one request, so it needs no baseline and is never subject to the warmup window.

​
The six kinds

The first three are never normal and cannot be switched off. The last two need the inbound webhook tap, below.

​
Why 4xx is opt-in

A 4xx is usually the caller’s own bug, which is exactly why an integration tool should be able to catch it. But an endpoint where a 401 is the normal answer would page you all day.
client_error_rate is the share of requests to one endpoint family that must be 4xx before one is reported. Below 20 requests to a family no rate is claimed at all, because the first 4xx is a rate of 1.0 and means nothing.

​
Inbound webhooks

The agent can also sit in front of your webhook handler. Give the upstream a receiver and point the provider’s webhook URL at the agent’s /hooks/<upstream>:
Each delivery is forwarded to the receiver untouched and the receiver’s response goes back to the provider, byte for byte. Nothing is retried. Recording is asynchronous and bounded, like the proxy, so a wedged recorder drops captures and counts them on /healthz as webhooks_dropped, never a delivery. /hooks/<upstream> is the one route that needs no X-Pikopod-Token and ignores the loopback Host guard, because the provider cannot carry your token and the delivery is authenticated by the provider’s own signature, which your receiver still verifies. What is recorded per delivery, redacted before disk, in <data_dir>/recordings/<upstream>.webhooks.ndjson: the event name, the delivery id as a token, the resource id as a token, the event timestamp, and the order it arrived in. The event name and id come from the provider’s envelope when the sandbox has one declared, from the x-pikopod-webhook-* headers otherwise, and from type, id and created in the body as a last resort.
Both fire on the first occurrence with no warmup. They are not reproduced with reproduce, because there is no request for the sandbox to answer; they are rehearsed with the sandbox’s own misdelivery faults, which is the other half of the same story. See Webhooks.

​
The recording is the reproduction

An incident forces its recording to disk regardless of sampling.rate. Sampling it away would leave you with an alert pointing at something you cannot run. The one exception is a path under record.exclude. Its traffic never reaches disk in any form, so a 5xx there is alerted as a fact and nothing more. Reproducing it says exactly that, with the key to change:
See What never reaches disk at all.

​
Listing them

A webhook incident lists the way to rehearse it instead of a reproduce command:
--only takes incidents, drift or divergence; the third lists what the provider did that the sandbox would not have, see Divergence. agent incidents is a reader, not a gate: it exits 0 whether or not it found anything. The reproducible until line appears when retention.max_age_hours bounds how long the recording stays on disk. --format json always carries total_matching and truncated. See pikopod agent incidents.

​
Reproducing one

The incident becomes a scenario pack that arms the same failure in the sandbox, answering with the recorded response body, and replays the recorded request at it. For a 5xx or an unreachable upstream the fault is armed on method and path, so it reproduces regardless. For a 4xx the request body matters, and it is redacted, so check the pack against what your code actually sends. When the agent runs on another host, pikopod agent incidents export <fp> writes a bundle that reproduce accepts anywhere. See Reproduce.

​
In an alert

Incidents follow the same dedupe and threshold rules as drift: one alert per fingerprint, after three occurrences in fifteen minutes. See Alerts.