agent truthfulness replays those recordings against a fresh copy of the sandbox and scores the answers, leaf by leaf, so the claim has a number next to it.
The scoring rule
Every recorded JSON response is replayed as the request that produced it, with the recorded request body and the sandbox’s own credential. The sandbox’s answer is then compared with the recording:- The status is one leaf.
201recorded and201served reproduces it. - Every scalar field of the body is one leaf, addressed by JSON pointer, so
/data/items/0/amountis a leaf and arrays score element by element. A leaf reproduces when the sandbox serves the same path with the same type and, where the recording kept a readable value, the same value. - A redacted leaf is scored on shape only. The recorder tokenizes identifiers and drops strings it cannot classify before anything reaches disk, so the real value is gone. Such a leaf reproduces when the sandbox serves the path with the right type, and the per-endpoint line counts it as
shape-onlyso you can see how much of the score rests on shape alone. Fields listed as volatile are treated the same way. - A leaf the sandbox serves that the provider did not send counts against it. An invented field is a lie the client would read.
- A
404where the provider answered scores zero for that response, and the endpoint is listed as unanswered. On the spec alone, a fresh sandbox has not stored the resource your recordedGETreads, so this is the usual reason a read endpoint scores0%on day one. Once recordings answer first, a create replayed from a recording teaches the sandbox that the recorded id means the resource it stored, and the chained read lands on that resource instead of a404. See Identifiers across steps.
worst list names the three fields that failed most often and, for each, where the sandbox got its value: enum, convention amount, example, or not served when the sandbox never produced the field at all. Those are the same sources requests --explain prints, so the fix is usually visible in the line: an example in the spec, a rule, or a recording the sandbox should replay.
Three places it shows
pikopod agent truthfulness <upstream>, with--format jsonfor the full breakdown.pikopod agent status, as atruthfulnessblock per upstream, scored over the most recent 200 recordings:
- The end of
pikopod demo, scored over the demo’s own recordings:
status to a value outside the spec’s enum and adds a fee_bearer field halfway through, which is what the number reports.
Before you have production traffic
The number needs responses the provider really sent. Without any, the command exits2 and says so:
The second run
The first number is the spec alone. Once recordings exist for the upstream, the sandbox answers from them before the spec (see Precedence), and the same command scores that:livemode. A sandbox imported before any traffic existed switches to recordings the first time you run pikopod import <name> --update after the agent has recorded some.