Skip to main content
A pack is YAML: name, provider, optional description, and a definition block. The validator is strict on purpose. Unknown keys are rejected, because a pack that references a missing operation, an unresolvable variable, or an impossible assertion is a broken test, and a broken test is worse than no test. The full grammar is in the schema reference.
provider is metadata; scenario run targets whatever sandbox you name on the command line. contractVersion is set by from-drift and reproduce.

Steps

Every step is key, type, config, plus optional description, assertions, capture, continueOnFailure, timeoutMs. Keys are unique and match [A-Za-z0-9_.-]+. Fault kind values and their behaviour are in Faults. times: N fires the fault for the first N matching requests and then recovers deterministically; per scopes that window to global, idempotency-key, or resource.

VERIFY_SEQUENCE

VERIFY_REQUESTS answers “how many, and what was the last one”. VERIFY_SEQUENCE answers “in what order, and how far apart”, which is the retry-storm and double-charge question.
It is an ordered subsequence: unrelated requests between matches are fine, order is not. minGapMs and maxGapMs measure virtual time since the previous match, so a backoff claim is reproducible. The matchers are the assertion. It fails closed: an evicted journal cannot prove an ordered claim, and a matcher touching a header the journal had to truncate is reported as unprovable rather than quietly not matching. Journaled identifiers are tokens, which is what makes “both retries used the same key” provable without the key being readable.

EMIT_WEBHOOK

Fire a declared event that no API call causes, such as money landing in an account. Only declared events can be emitted, and data is overlaid onto the documented payload. See Webhooks.

Assertions

An assertion is target and op, plus optional subject (SANDBOX, the default, CLIENT, or PRODUCTION), path (JSONPath starting $.), key for headers, resourceType and resourceId, match, expected, schemaRef, and soft.
  • Targets: response.status, response.headers, response.body, response.latencyMs, state.resource, state.resourceCount, webhook.delivery, webhook.count, execution.faultApplied, sandbox.requestCount, sandbox.request.
  • Ops: equals, notEquals, contains, notContains, in, matches, matchesSchema, exists, absent, gt, gte, lt, lte, countEquals, lengthEquals, isOneOf.
in and isOneOf are the same check; so are countEquals and lengthEquals. matches rejects an uncompilable regex at validation time, and matchesSchema requires schemaRef.

Captures and variables

capture maps a variable name to source$jsonpath, where the source is response.body, response.headers, webhook.delivery or state.resource: chargeId: response.body$.id. A capture whose path does not resolve fails the step rather than binding silently. Reference variables as {{name}}. Precedence is inputs, then captures, then defaults. A variable must be declared, captured by an earlier step, or a generator: {{seq()}}, {{randomString(n)}}, {{now()}} (the virtual clock, never the wall clock). {{any:string|number|boolean|iso8601|uuid}} stays verbatim in requests and is a matcher in assertions.

Seed data

SEED_STATE seeds resources before the traffic that reads them. Omit resourceKey to autogenerate one; an empty key is rejected. attributes is required.

Limits

1 MiB per file, 200 steps, 50 assertions per step, 1000 assertions total, 1000 seeded resources, 50 inputs, and 90 virtual days of accumulated WAIT and EXPECT_WEBHOOK time. If you hit scenario pack is too large, split the scenario or trim seed data. A pack with thousands of seeded resources is usually a load test wearing a scenario’s clothes.

”pack does not ground against this sandbox’s API”

The pack references operations the sandbox does not have: it was written for another provider, the sandbox was re-imported from a spec that dropped the operation, or the operation id changed upstream, which is itself worth looking at. pikopod scenario list <sandbox> shows what the sandbox exposes. A REQUEST path must match an endpoint in the pinned API and must be sandbox-relative, never an absolute URL.