A reproducible replay starts with its contract

Start with the data specification

A reproducible replay begins with a record of the rules it implements: feed format, sequence scope, event semantics, initial state and the exact meaning of each output. This provides the basis for checking whether a repeated result is also consistent with the data specification.

Separate protocol facts from experimental assumptions. A documented replace rule belongs to the former; assuming every order has unit size belongs to the latter. Both may be useful, but a reader should never need to guess which is which.

Keep a small experiment record

experiment: event-order-001
input: three synthetic SET messages
sequence_scope: one channel, one session
initial_state: quantity = 0
update_rule: SET overwrites quantity
source_order_expected: 7
receive_order_expected: 4
network_access: none

Keep this record with the code. For larger inputs, retain a checksum, parser version and data licence or access conditions. Machine-readable records make later changes visible and prevent a familiar filename from silently referring to different data.

Test invariants, not only final values

A final checksum can reveal disagreement without identifying its origin. Per-event checks localise the problem: a known order before cancellation, a positive remaining quantity for a resting order, and agreement between order-level totals and an aggregate built from the same included state.

Each invariant must respect the feed. A message can be a correction, recovery update or session transition. The checks should accommodate these protocol-defined cases, with exceptional transitions represented explicitly in the fixtures.

Record when information becomes available

An offline reconstruction can use later messages to explain an earlier event. That may be valid for diagnosis while unavailable for a contemporaneous decision. Label outputs as retrospective states, feed-observed states or live-available features.

A timestamp comparison alone is insufficient if the timestamp was assigned at a processing stage the consumer had not yet observed. Availability belongs to the observation model and should be tested. Keep diagnostic labels separate from the input used by the decision rule.

What to include in a review package

• An input fixture with provenance and a stable identifier.
• Explicit assumptions and unsupported message types.
• An expected event-by-event trace and a minimal run command.
• Failure states, recovery rules and information-availability checks.
• The scope of the experiment and the interpretation of its results.

This checklist is intended for reviewing a small experiment. The public lab provides a synthetic teaching fixture to demonstrate the workflow.