Running scenarios

pikopod scenario check examplepay declines retry_storm

✓ declines — PASSED (4 assertion(s) passed; 0 not evaluated)
    NOT_EVALUATED  arm-decline      armed error on POST /charges
    PASSED         declined         POST /charges → 400
    NOT_EVALUATED  clear            cleared matching faults
    PASSED         recovered        POST /charges → 201
✓ retry_storm — PASSED (4 assertion(s) passed; 3 matcher(s) matched in order; 0 not evaluated)
    NOT_EVALUATED  arm              armed error on POST /charges
    PASSED         attempt1         POST /charges → 503
    PASSED         attempt2         POST /charges → 503
    PASSED         attempt3         POST /charges → 201
    PASSED         storm-shape      3 matcher(s) matched in order
exercised 1 of 4 declared operations (unexercised: GET /charges, GET /charges/{id}, PATCH /charges/{id})

Names resolve to archetypes first, then to saved packs. Several can run in one command. The last line counts the spec’s operations the run’s request steps reached, so you can see what the bound archetypes did not touch.

Reading the result

Mark	Status	Meaning
`✓`	`PASSED`	Every assertion passed.
`✗`	`FAILED`	An assertion failed. Exit `1`.
`!`	`ERRORED`	A step could not execute. Exit `2`.

Each step line shows the step’s status, key and summary. NOT_EVALUATED marks steps with no assertions of their own, such as arming or clearing a fault. When the sandbox has a webhook sink, a sink summary follows each scenario. A failed request step carries the request as one curl line, so the person who looks into it can send the same request to a running sandbox without reading the pack:

✗ wrong-status — FAILED (expected response.status equals 503, got 201)
    FAILED         create           expected response.status equals 503, got 201
                                  curl -X POST "$PIKOPOD_URL/examplepay/charges" -H "authorization: Bearer $PIKOPOD_CREDENTIAL" -H 'content-type: application/json' -H 'idempotency-key: order-77' --data '{"amount":5000,"currency":"usd"}'

1 scenario(s) failed — exit 1

$PIKOPOD_URL is the sandbox server, http://127.0.0.1:4600 under pikopod up, and $PIKOPOD_CREDENTIAL the credential sandbox list prints. The credential itself never appears in the output. The same line ends every requests --explain, and an incident bundle carries one built from the recording.

Flags

Flag	Meaning
`--bind role=operationId`	Override a role binding. See Binding.
`--input name=value`	Set a declared input. Numbers and booleans are coerced to the declared type.
`--seed <string>`	The run seed. Default: the sandbox’s seed. Same seed, same bytes.
`--persist`	Run against the sandbox’s real store. Seeded state then stands for the served sandbox. Armed faults never outlive the run; arm the served sandbox with `chaos` or `mode` instead.
`--target <url>`	Run against a real endpoint.
`--target-header 'Name:value'`	A header on every remote request. Repeatable.
`--format text\|json\|junit\|githubactions`	The report shape. See Reports a build can be gated on.

Reports a build can be gated on

--format junit writes one testsuite per scenario and one testcase per step. A failed step’s summary is the failure message, and the curl line to reproduce it is the body; NOT_EVALUATED steps are skipped cases. Notes and the exit line go to stderr, so stdout is the artifact:

pikopod scenario check examplepay declines wrong-status --format junit > junit.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="6" failures="1" errors="0">
  <testsuite name="declines" tests="4" failures="0" errors="0" skipped="2">
    <testcase name="arm-decline" classname="declines">
      <skipped></skipped>
    </testcase>
    <testcase name="declined" classname="declines"></testcase>
    <testcase name="clear" classname="declines">
      <skipped></skipped>
    </testcase>
    <testcase name="recovered" classname="declines"></testcase>
  </testsuite>
  <testsuite name="wrong-status" tests="2" failures="1" errors="0" skipped="0" file="scenarios/wrong-status.yaml">
    <testcase name="create" classname="wrong-status"></testcase>
    <testcase name="read-back" classname="wrong-status">
      <failure message="expected response.status equals 503, got 200" type="REQUEST">expected response.status equals 503, got 200&#xA;curl -X GET &#34;$PIKOPOD_URL/examplepay/charges/charges_1&#34; -H &#34;authorization: Bearer $PIKOPOD_CREDENTIAL&#34;</failure>
    </testcase>
  </testsuite>
</testsuites>

--format githubactions prints one ::error per failed step, with the pack file and the line of the step’s key when the scenario is a pack, so the failure lands inline on the pull request. The unexercised operations become one ::notice:

::error file=scenarios/wrong-status.yaml,line=10,title=wrong-status / read-back::expected response.status equals 503, got 200
::notice title=unexercised operations::GET /charges, PATCH /charges/{id}

--format json carries every run’s full result plus failed, errored, exercised and unexercised:

{
  "runs": [
    {
      "name": "declines",
      "result": {
        "status": "PASSED",
        "summary": "4 assertion(s) passed; 0 not evaluated",
        "notEvaluated": 0,
        "resultHash": "69cc9df8150fd7209bfeac4f556d90c85eb80981b6aef49b0f27b3cfe56f49d3",
        "steps": ["…"]
      }
    }
  ],
  "failed": 0,
  "errored": 0,
  "exercised": ["POST /charges"],
  "unexercised": ["GET /charges", "GET /charges/{id}", "PATCH /charges/{id}"]
}

Exit codes are the same in every format. A complete workflow that uploads the JUnit file is under CI integration.

Ephemeral by default

Each run drives its own requests at a throwaway copy of the sandbox, so a test never pollutes the served sandbox’s state, and no pikopod up needs to be running. Faults, seeded state, the virtual clock, the webhook outbox and the request journal all exist inside the run.

Against a real endpoint

pikopod scenario check examplepay checkout-flow --target https://sandbox.examplepay.test --target-header 'Authorization:Bearer …'

Only REQUEST, NOTE and SNAPSHOT steps are allowed. A real endpoint has no fault arming, seeded state, virtual clock, webhook outbox or request journal to consult, so any other step type is refused before anything runs. Header values are never printed.

In CI

pikopod import examplepay --spec ./specs/examplepay.yaml --seed ci-fixed
pikopod scenario check examplepay declines timeouts retry_storm partial_failure incident-14835fa32dfb

Pin --seed so transcripts are byte-identical between runs. See CI integration.

Getting started

Gate: catch declared changes in CI

Sandbox: rehearse failures

Scenarios

Observe: the agent

Reproduce and fix

Operations

Project

Running scenarios

Reading the result

Flags

Reports a build can be gated on

Ephemeral by default

Against a real endpoint

In CI

​Reading the result

​Flags

​Reports a build can be gated on

​Ephemeral by default

​Against a real endpoint

​In CI

Reading the result

Flags

Reports a build can be gated on

Ephemeral by default

Against a real endpoint

In CI