One agent, not a sidecar per replica
Run one agent as a small shared service. Baselines are single-writer: two agents pointed at onedata_dir corrupt each other, and N agents with N data_dirs learn N partial baselines, each warming up on a fraction of the traffic, each raising incidents the others never see, with a fingerprint that exists on only one host. A single agent behind a stable address sees the whole stream and holds the whole history. The extra hop is a loopback-class forward that serves before it observes.
If the process dies, traffic stops
pikopod is in the path, so a crash, an OOM, or a bad deploy takes the proxy with it. Run it as you would any sidecar: a supervisor that restarts it, and/healthz as the liveness probe. /healthz is public for liveness and needs the token for the full payload, so a load balancer can poll it without being handed your upstream inventory.
The sandbox and the agent are separate listeners in one process and fail independently. The sandbox going down does not touch production traffic.
Start with staging
Drift needs a baseline. Letting the agent warm up somewhere low-stakes means the first thing production sees is a tool that has already been quiet for two days. Incidents fire either way, from the first request.data_dir is state, not cache
Put it on a persistent volume, back it up like a database, and give it to exactly one agent. Losing it loses the baselines, the event log and the recordings behind every open incident. Include.salt in the backup and exclude it from anything you share. See Data directory.
Binding beyond loopback
PIKOPOD_TOKEN or a 0600 token file, and send it as X-Pikopod-Token from every client, including your app’s calls through the agent. See Security.
A systemd unit
token_file: /etc/pikopod/token in the config rather than the token in the unit’s environment, so it never appears in systemctl show. On SIGTERM pikopod drains for three seconds, then closes.
Container
The image isFROM scratch, runs as user 65532, and has no shell:
listen: 0.0.0.0 and a token inside the container so the published ports are reachable, and keep the host-side binding on loopback unless you mean otherwise.
How an incident reaches a developer
In rough order of how often they apply:- Export a bundle.
pikopod incidents export <fp>on the agent host writes one JSON file with the event, the already-redacted recording and the contract version.pikopod scenario reproduce ./incident.jsonandpikopod fix ./incident.jsonon a laptop read nothing from that laptop’sdata_dir, and the file keeps working after the origin’s retention has aged the incident out. See Incident bundles. - Commit the generated pack.
reproducewrites an ordinary scenario pack. Once it is in the repository it guards that path forever and needs no recording at all. - Share
data_dirread-only (a mounted volume, an rsync) for a team that wants every incident, not one at a time. - Automate the export:
pikopod incidents export --since 24hin a cron job, attached to an issue or a pull request.
Sizing
Stored per record are the sanitized request and response, so recordings are small and bounded bysampling.rate and retention. The sandbox store is capped at 100 MiB per sandbox. The proxy hot path is a full loopback round trip in about 82 microseconds on one machine, with observation fully jammed or not. See Data plane safety.