Serve first, observe afterwards
The response is written to your client in full before any observation code runs. The request body is copied on the way through into a capped buffer, and the copy never affects the bytes on the wire: past the cap the copy truncates and the pass-through continues. Streaming and server-sent events flush immediately. Observation then runs asynchronously behind a bounded queue:
When the queue is full, the oldest capture is dropped and counted. When the byte budget is exhausted, the new capture is dropped and counted. Every capture stage recovers from panics and counts them.
/healthz reports recordings_dropped and observer_panics, so a struggling observer is visible without ever being felt.
It never retries
An unreachable upstream gets an honest502 with X-Pikopod-Error: upstream-unreachable and no second attempt. A retry in front of a payments API is a double-charge window.
No timeout of its own
The agent sets no upstream timeout, so a legitimately slow or streaming response is never cut short. An incident fires when your caller gives up and the connection dies, not when a stopwatch says so. Only the request header read has a deadline, 20 seconds, so a client that never sends headers cannot hold a connection open forever.Protocol upgrades pass through
WebSocket and other upgrades are handed to the upstream unchanged. A test pins this, because a capture wrapper that broke upgrades would violate fail-open.Measured, not asserted
Two benchmarks serve the same workload. In the second, observation is completely jammed: nothing drains the capture queue, so every request takes the drop path.If the process dies
Traffic stops, because pikopod is in the path. Run it as you would any sidecar: a supervisor that restarts it, and/healthz as the liveness probe. On SIGTERM it drains for three seconds and then closes. The sandbox and the agent are separate listeners and fail independently. See Deployment.