Scenario roster

DR Scenarios

Worked disaster-recovery scenarios for the XRPL dUNL. Each scenario is built on a common framework (template.md) with four axes:

1. Situation variables — the inputs/dials that define the incident.
2. Detection & monitoring models — the signals, metrics, and dashboards we need to see it (and an explicit list of what we can't see today).
3. What we're proving — a falsifiable claim with a metric + threshold, not a tabletop narrative.
4. Response — mapping onto Proposal 0001's response menu.

How this directory maps onto the exercise layers

Which layers exist, who responds in each, and what each one proves is drills/README.md. This directory holds the artifacts.

The distinction the layout enforces: a simulation runs itself, a live drill does not. A simulation is a JSON the harness executes end to end and verdicts on its own, which makes it the engineer's dev-loop and the mechanical de-risking step. A live drill reuses the same fault and the same verdict criteria with people on the other side.

The how-to-rehearse documents are public and live in drills/: the tabletop run sheet, the drill packets, and the recording run-of-show. This tree is the internal ring and never commits; what stays here is the scenario catalog itself plus the local run material, drill-cost-estimate.md, harness-plan.md, and proving-runs.md.

One spec, both modes. The simulations/NN-<slug>.json file is the live-drill driver — the loadtester's drill.mode flag switches it. simulate (default) fires the whole timeline and verdicts with no human. live fires only the fault (step 1), provisions and tears down the network itself, and leaves the recovery (step 2) to the on-deck operators while still measuring how long they took. Recovery steps are tagged phase: recover (the recovery verbs default to it), so the specs here are live-ready without edits. See the loadtester's drill-report spec for the mode/phase/provision contract.

How a live drill is set up, run, and torn down — it runs on our own private network, never on operators' production validators:

• Setup (us) — select the on-deck validators per drills/README.md, stand up a distributed, datagram-emitting xrpld-lab network, give each participant scoped access to their own node with a key delivered per drill-packets.md, pre-stage background load, bring up the live XDGM board, and pick a random, unannounced date.
• Run — Part 1 (us): at the random time, the facilitator injects the scenario's fault and fires the page. Part 2 (them): the on-deck operators pick it up cold — detect, diagnose, work the runbook, coordinate through the Incident Coordinator, and drive it to the harness verdict. That unwarned hand-off from our injection to their response is the test; the harness records the whole run.
• Teardown (us) — xrpld-lab destroys the network, which invalidates every issued key; any retained host has authorized_keys cleared. Output is two artifacts: the harness verdict and the retrospective.

The simulation layer is grounded in xrpld-lab (build a real multi-validator network + inject the fault) and xrpld-loadtester (drive background traffic + measure the outcome); the loadtester's drill-report spec defines what a drill report shows versus a plain load-test report.

Status: all Draft. Built from deep research; not yet drilled. Simulation specs for the UNL-visible scenarios (02–12) are staged in simulations/ and validated to parse against the loadtester RunSpec; most are partial (the central fault injects today, but some claims wait on lab primitives the specs list in meta.blocked_on). The point of the harness is that "Draft → Drilled" is a matter of running the experiment.

Visibility: who sees what

The repo is private, so committing a file makes it visible to the UNL, not the world. Three rings, from most to least exposed:

• World — eligible for eventual publication at xrpl.org/security. Reading the runbook doesn't help an attacker: the incident is a safe state (a halt) and the recovery is operational. Marked Exposure: Public on the scenario.
• UNL — committed here so the operators who run the response can see the playbook, but not for public mirroring: publishing the detection thresholds, coordination timings, or response sequencing would hand an attacker a roadmap. Marked Exposure: Private on the scenario. Controlled transparency for these is the post-incident retrospective (0001 §3), not this forward-looking playbook.
• Internal — things the UNL does not see: exploit reproductions and the publisher-key procedures that are themselves an attack surface. These are git-ignored (scenarios/**/*-private*, the 90–99 range) and never committed; the executable drill assertions for them live in the private xrpld-lab, not here.

An Exposure field on each committed scenario marks only the World-vs-UNL line — every committed scenario is UNL-visible regardless.

World-eligible: 02, 03, 08, 09, 12, 14. UNL-only: 04, 05, 06, 07, 10, 11, 13.

Scenario 13 is the deliberate counterpart to 11, drawn from the 2026-08-01 peer-connection-flood incident. 11's thesis is that a coordinated config gate restores health without a binary; 13 is the exception where that is false, because the abuse is pre-consensus at the socket and no runtime knob gates it. 13 also owns building the connection-flood injector that 11 lists as its top missing primitive, and establishes that the injector is a standalone external binary, not a compile-gated adversary node.

Internal ring (git-ignored, not in the table above):

• 97 — UNL Publisher Master-Key Replacement. The coordinated fleet-wide [validator_list_keys] edit. Internal because the procedure is the social-engineering surface (convincing 35 operators to trust a new master key is the attack in 99 when the key is the attacker's).
• 98 — UNL Publisher Signing-Key Rotation. Publisher-side ops; a rotation is transparent to the fleet (a manifest seq bump, master key unchanged, zero operator action), so there is nothing for the UNL or the IC to do.
• 99 — UNL Publisher Key Theft. The overlay-injection exploit reproduction; never mirrored publicly.

Scenario 12 and internal 97 are the validator-identity / publisher-identity continuity pair from the June 21 2026 validator-migration incident: 12 covers an operator preserving their own validator identity across a host move; 97 covers a publisher replacing the key the whole fleet pins. Both are the operational/recovery counterparts to integrity attacks — 12 to 06, 97 to the internal publisher key theft case (99), whose open Q4 (fleet-wide [validator_list_keys] update time and owner) 97's master-key-replacement drill answers.

Monitoring substrate: XDGM (UDP datagram), not Grafana

The "models we need for reviewing and monitoring" axis is served by XDGM, the existing UDP datagram telemetry path — no Grafana, Prometheus, or StatsD stack.

• Transport. Each node emits a binary XDGM datagram (magic 0x4D474458) once per second to a single collector; xrpld-lab --datagram_monitor "HOST PORT" injects the [datagram_monitor] stanza into every node's config (protocol-agnostic in the lab), and loadtester xdgm.py decodes it into an in-memory dict keyed by node public key.
• What the packet already carries (so we don't build collectors for it): server_state, ledger_seq, ledger_hash, version_string, validation_quorum, proposer_count, peer_count, converge_time_ms, load_factor, warning_flags (amendment-blocked / not-synced), plus CPU / load-avg / memory / disk / net+disk I/O rates and cache hit rates.
• Two prerequisites (both decided by us, not blockers): 1. The node must emit XDGM. Stock rippled does not ([datagram_monitor] is a Xahau / dangell7/datagram-branch feature). The DR lab network must be built from a datagram-capable build: xrpld-lab create:network --build_server <src> --build_version dangell7/datagram (or a rebase of it onto the target rippled release). Fallback if we don't want the custom build: a 1 s server_info + get_counts RPC poller synthesizing the same fields — more work, same data. 2. xdgm.py is post-run aggregation today, not live. It already holds every node's latest sample in memory, updated every 1 s; exposing that as a live SSE/WebSocket or terminal (rich/curses) view is a thin build — that is the Grafana replacement.

This substrate directly supplies several scenario detection signals, which is why the detection backlog below is shorter than it first looked.

Lab backlog — "network-building on the lab"

The scenarios surfaced what xrpld-lab / xrpld-loadtester can already do vs. what must be built before each scenario is fully drillable. This is the concrete backlog for putting the DR-drill harness on the lab.

Already there (reused as-is): create:network (N validators + peers + signed UNL), up/down, node:stall (silent-failure halt), node:restart (crash), update:node (binary upgrade/rollback), enable:amendment (activation), docker compose stop (outage/partition), health polling (time-to-recovery / time-to-supermajority), and loadtester tps_validated (ledger-time ground truth), p95/over_5s (stress), tefPAST_SEQ + correlation hit-rate (fork/partition), fee monitor, XDGM per-node CPU/mem.

Notable free win: [amendment_majority_time] has a 15-minute floor enforced in Config.cpp, settable via create:network --config_overrides. That compresses the 2-week amendment clock to ~15 min, making the amendment scenarios (04, 10) actually drillable. Backlog: package as a reusable "fast-amendment" lab profile.

Must be built (grouped; → scenarios that need it):

Detection / measurement layer (highest leverage — these are the "monitoring models" the scenarios keep asking for):

Delivered by consuming XDGM live (build = the thin live view over xdgm.py, once nodes emit XDGM — see Monitoring substrate above; no per-signal collector needed): - Cross-node fork detection — compare ledger_hash across nodes at a common ledger_seq; divergence = fork. (Replaces the standalone "cross-endpoint hash comparator.") → 07. - Version-spread / time-to-supermajority panel — version_string + server_state per node; the curve that collapses to the patched build during an upgrade. → 03, 05. - Quorum-headroom view — validation_quorum + proposer_count + server_state, so "approaching quorum loss" is a live gauge. → 02. - Consensus-stress view — converge_time_ms + load_factor + I/O rates (complements loadtester over_5s). → 11.

Still real builds (XDGM does not carry these): - Value-at-risk / abnormal-value-movement detector — per-ledger drops-moved + continuous supply-invariant monitor + per-amendment baseline/threshold. Transaction-semantic, not in XDGM; the gating capability for the worst-case exploit claim and the biggest build. → 10. - Validations-stream agreement metric — subscribe to the validations stream (which trusted validator disagrees, with amendment/flags), beyond the per-node ledger_hash XDGM already gives. → 06. - Phantom-proposer / network-attribution check — cross-node test of whether a validator that self-reports proposing actually has its master key in other nodes' accepted-validation set. The migrating node's own XDGM says healthy; only a cross-node/validations-stream view exposes a dropped signing key (stale wallet.db or un-propagated/stale-sequence manifest). → 12 (shares the validations-stream build with 06). - Publisher-list / sequence / [validator_list_keys]-state watcher — RPC poll surfacing each node's trusted publisher master key(s), per-publisher sequence + expiration, and fleet-wide config-edit progress (XDGM carries none of this). → 97, 99, 08, 14. - feature admin-RPC reader — scrape support-fraction, admin-only veto-margin fields, and the majority/enabled trajectory. → 04. - Fleet-correlation detector — automatic "N validators failed on the same input" instead of a human noticing in chat. → 05, 02. - Surface UNL health as harness metrics — validator_list_expires, served-list sequence, trusted-set size (not in XDGM). → 08.

Fault-injection primitives: - health negative/expected-state assertions (e.g. --expect-state amendment_blocked) so traps fail the drill closed instead of by-hand. → 03. - Parameterizable fault knob — "halt after N ledgers" / "crash on tx pattern X" — so drills don't depend on supplying a real broken binary. → 03, 05. - Hot config-reload primitive — edit [reduce_relay]/[overlay]/fee/queue on a running node and restart in place with history preserved (today only update:node, which wipes lib/ and forces a full resync — wrong path for the seconds-fast config-gate this proves). → 11. - Overlay-level malicious-peer / peer-flood injector — loadtester speaks only client WS; it can't generate true TMTransaction/peer-protocol floods. → 11, 06. - node:corrupt-ledger injection — real same-tx-set/different-hash silent corruption, vs. today's partition approximation. → 07. - Correlated-fault primitive + provider/region/ASN metadata (kill:region/kill:provider) and a diversity/concentration metric (HHI / max-validators-per-domain) — so "correlated outage" is enforced/measured, not hand-picked. → 02. - Provision + UNL-enroll a fresh replacement validator (not just revive a stopped host), to time the worst-case recovery branch. → 02, 12. - Validator-identity-migration injectors — seed a node with a stale wallet.db (mismatched manifest/node identity) and rotate [validator_token] with a non-incrementing/un-propagated manifest sequence, plus a validator-keys.json sequence-state handle, so the two phantom-proposer faults are reproducible rather than hand-built. → 12. - Publisher master-key replacement + fleet-wide config-edit orchestrator — stand up a new publisher master key and stagger [validator_list_keys] edits across nodes on an operator-response cadence (today: hand-stitched config overrides + restarts), plus publisher signing-key rotation and revocation-manifest (0xFFFFFFFF) injection. → 97. Pairs with a short-TTL / near-expiry UNL (--unl_ttl) primitive to force the expiry race. → 97, 08.

Orchestration / fidelity: - One-shot embargo-drill orchestrator — stagger update:node across the fleet on a realistic operator-response cadence and emit a single 4h pass/fail artifact (the 0001 Success Criterion) instead of hand-stitched health polls. → 05. - Signed-binary / hash-verification step in update:node — so the supply-chain half of the rollout (operators verifying before installing) is actually exercised. → 05. - Runtime feature-disable — enable:amendment only votes on; there's no real kill-switch primitive, so kill-switch claims currently lean on workload-stream separation as a proxy. → 05, 10. - Short-TTL / near-expiry UNL + clock-skew primitive (--unl_ttl) — to exercise the "outage outlives the cache" worst case. → 08. - Two-publisher lab network (--num_publishers 2) with an effective-aware re-sign and revocation injection, so the threshold-1 union, the pending-list rotation, and the unavailable-publisher quorum lockout can be driven on non-validating peers. → 14, 08, 97.

Out of lab scope (flagged honestly): Scenario 09 (comms-channel loss) is a human-coordination failure with no network component — the lab can only prove the chain keeps validating without the humans. Its tooling (channel health checks, an out-of-band status page, a comms-failover tabletop) belongs outside the network-fault lab, and it has no simulation spec.

Relationship to the proposals

• These files are new content, not the 0002 §6 migration of existing README material — so creating them here doesn't pre-empt the migration decision. On 0002 acceptance the World-eligible scenarios move to runbooks/<slug>.md (noted in each file's header).
• Each scenario's §5 cross-links open questions back to Proposal 0001 (response menu, 4h supermajority, kill-switch authorization/inventory, non-responder backstop) and Proposal 0002 (teams, tiered alerting, Engineering Response Team). Several scenarios independently re-derive the same gaps already listed in those proposals' Open Questions — useful corroboration.
• Proposal 0003 is the live-drill (Rung 3) participation model: how the simulation's fault and verdict get run as an all-hands rehearsal across 35 operators.

Rendered from scenarios/README.md in the private repo. Where this page and the markdown disagree, the markdown is current.