The whole picture
The Incident Board
Every lane at once, in order: the Coordinator's board, and the full picture for anyone who wants one. It is not required reading for a live incident - your own role page is complete. Where this board and a role page disagree, the role page is current.
The board at a glance
- Detect the issue and fire the page operators + monitoring
Whoever saw it first, in whichever channel they were already in. Any operator and any paged seat can fire the page on a correlated pattern; a false page costs nothing.
- The page rings every seat at once; the Coordinator declares IC
Each seat makes its first move without waiting. Denis declares, takes command, names Brett as UNL Leader in #dunl-operators-only, opens the bridge, and orders both tracks.
- The working channel opens IRT
#unl-validators takes reports and logs. The embargo starts: nothing leaves it and
xrpld-privateuntil the Coordinator clears it. - Two tracks in parallel advisors + engineers
The damage report and the root cause, both ordered in the declaration. Advisors own damage, engineers own cause; the Coordinator holds both and says which he is waiting on.
- Interim response if damage is accruing advisors recommend, operators execute
Config gate, kill switch, or halt - called off the damage report, without waiting for the root cause.
- Confirmed reproduction: decide the fix, recommend the response IRT
Engineers own the fix; advisors weigh the response and speak one recommendation.
- Build and sign the patched binary engineers
If the response needs one: restricted pipeline, M-of-N custody, hashes operators can fetch independently.
- Execute the response operators
One instruction, one of five actions, pinned by Brett in #dunl-operators-only, countersigned where unsigned. Operators verify, act on their own nodes, and post
done; Brett counts and chases the silent. - Close out IC + COMMS
De-escalate and unwind so the monitoring window runs clean, then stand-down on the four conditions. Advisory, CVE, and retrospective follow.
Throughout: the Steering Group is dormant, COMMS drafts in the background, and the Coordinator keeps the record - every declaration, order, interrupt, decision, and handoff with its time. There is no scribe.
Every clock in the plan
A warn lands on a dashboard, nobody is woken. A page rings phones - on a correlated signal it rings every paged seat at once. The acknowledgement clocks exist for the phone that goes unanswered: after 5 minutes the seat's second is paged and takes the seat rather than asking.
| Clock | Starts when | Length | When it runs out |
|---|---|---|---|
| IC acknowledgement | The page fires | 5 min | The secondary is paged, and 5 minutes later the tertiary. Whoever takes it declares immediately |
| Engineer Leader ack | The page fires | 5 min | The second takes the seat, saying so in channel. Nothing on the cause track starts until someone holds it |
| Advisor Leader ack | The page fires | 5 min | The second takes the seat, saying so in channel |
| Order acknowledgement | The declaration's two orders land | 3 min per leader | The silent leader's second is paged and takes the immediate deliverable |
| Damage report | The Advisor Leader's one-word ack | 15 min | The Coordinator asks for it before anything on the cause track, unknowns taken as answers |
| Rollout deadline | An instruction is pinned - T+0 | 4 h | 80% of the fleet is the supermajority standard; miss it and the Coordinator extends, accepts lower, or changes the response |
| First count | The pin | T+30 min | Count posted; the chase opens on everyone silent |
| Early warning | The pin | T+60 min | Under 40% acknowledged, the Coordinator hears it now |
| Second contact | The pin | T+90 min | The still-silent are contacted on a different channel than the first attempt |
| Reasons attached | The pin | T+2 h | Every outstanding name has a reason: blocked, declined, unreachable, or in progress |
| Emergency contacts | The pin | T+2 h 30 | ICE contacts asked to reach anyone still unreached, out of band |
| Projection | The pin | T+3 h | Will the target be met by the deadline, with the reason mix - what the Coordinator can still act on |
| Coordinated-restart consent | An agreement thread opens (coordinated regime only) | 30 min | Without majority agreement of responding operators, each operator may act individually |
| Monitoring window | De-escalation, or the last recurrence | 24 h | Stand-down becomes declarable. A recurrence inside it resets the window on the same incident |
The phases, step by step
Phase 0 - the signalsteps 1-9
| # | Who | Where | What happens |
|---|---|---|---|
| 1 | whoever sees it | wherever they are | The first observation is the monitoring page tier or a human already in the logs; a human signal lands raw, usually in #dunl-operators-only |
| 2 | monitoring, or the observer | alerting tier | The correlation rule decides who is woken: one validator's alert routes to its own operator; the network-wide check, or the same fault on three or more validators inside five minutes, fires the page to every seat. Any paged seat and any operator may fire it directly |
| 3 | the page | every paged seat | All simultaneously. Each seat's first move is fixed and none waits on the Coordinator's acknowledgement |
| 4 | UNL Leader | contact registry | Opens the registry now - the chase starts 30 minutes after any pin |
| 5 | COMMS | the bridge | Joins whatever call exists, catches up from the record, takes the door |
| 6a | Engineer Leader | telemetry, logs | Reads the leads. First discriminator: do the affected validators answer server_info? Reachable-but-stuck is software; unreachable is infrastructure |
| 6b | Advisor Leader | the damage report | Starts filling it at the page, minutes ahead of his own clock |
| 6c | org on-call engineer | telemetry, logs | Owes the Advisor Leader the first read of the affected surface. Nothing else: no declaring, no instructing, no speaking |
| 7 | IC | the page | Acknowledges inside 5 minutes, checks correlation: a correlated pattern is an incident |
| 8 | whoever acknowledged first | #unl-validators | Until a Coordinator acknowledges: post the raw signal timestamped and start the record. Not a declaration; no pinning, because no one can authorize an instruction |
| 9 | escalation | alerting tier | 5 minutes silent pages the secondary; 5 more the tertiary. All three silent is a paging failure and the retro's primary finding |
Phase 1 - declarationsteps 10-19
| # | Who | Where | What happens |
|---|---|---|---|
| 10 | IC | #unl-validators | Declares on the fixed body: condition off the seven-word list, severity, damage line or its due time, Cause: not yet established, bridge link, UNL Leader named, authorization regime, next update. Unknown is a complete condition |
| 11 | IC | the bridge | Opens the room with the standing pre-shared link in the same minute. If it fails: any room, link posted in channel under the poster's own name, never by DM |
| 12 | IC | #unl-validators | Both orders in one message: the damage report due 15 minutes from acknowledgement, and the root cause with the formation named. Plus the embargo |
| 13 | both leaders | #unl-validators | Each acknowledges in one word. Either silent for 3 minutes gets their second paged, who takes the deliverable |
| 14 | IC | #dunl-operators-only | The Coordinator's only post there: incident declared, the named UNL Leader is the only person who will pin here, keep nodes running as normal |
| 15 | UNL Leader | #dunl-operators-only | Takes the net: instructions come pinned, from this seat, nowhere else; done in the thread; names the backup or a fallback. The fallback is never the Coordinator |
| 16 | COMMS | #disaster-recovery-news | Mirrors the declaration body verbatim. Trimming goes back to the Coordinator, never through edit |
| 17 | COMMS | public channels | Publishes the pre-approved holding statement on his own timing. No cause, no response, no timing |
| 18 | IC | #unl-validators | Pins the interrupt rule and the cadence: leaders interrupt directly when it cannot wait; everything else reports by exception; leaders report, not members |
| 19 | COMMS | the bridge | Works the door and the public channels in parallel from here on |
Phase 2 - two trackssteps 20-33
| # | Who | Where | What happens |
|---|---|---|---|
| 20 | Engineer Leader | #unl-validators | Names the formation in one message: telemetry read, swarm, split the map, or one driver, with named owners per surface |
| 21 | Engineer Leader | #unl-validators | Staffs the listeners in the same message: named people who read every channel and investigate nothing |
| 22 | engineers | xrpld-private | Run the loop: logs, code, replicate with a test, push the failing test, team review. Reproduction detail never appears in #unl-validators |
| 23 | Engineer Leader | #unl-validators | Feeds the Advisor Leader the early surface read, labelled low confidence. Shape, not cause |
| 24 | Advisor Leader | #unl-validators | Posts the damage report inside the 15 minutes, alone if the advisors have not assembled: class, how much, accruing, reversible, surface, worst case in one hour |
| 25 | Advisor Leader | the bridge | Pulls advisors from the pre-cleared pool, who arrive into a report that exists and sharpen it |
| 26 | advisors | the bridge | Work the response menu against the posted class, costing each lever. The discussion does not open until the class is posted |
| 27 | COMMS | #unl-validators | Routes public-side posts one at a time, by name, original link attached; separately, the public-temperature digest on the cadence |
| 28 | UNL Leader, or one named helper | #dunl-operators-only | Reads and routes the operator channel. One reader per channel; a route is a link, not a paraphrase |
| 29 | IC | #unl-validators | Asks each track for its specific missing fact, one question to one named leader. Never "any update?" |
| 30 | IC | #unl-validators | If the damage report misses its 15 minutes, asks for it before anything on the cause track |
| 31 | Advisor Leader | #unl-validators | Refuses to guess a line: names the missing signal, who can supply it, and the conservative assumption |
| 32 | both leaders | interrupt path | Four events interrupt instead of waiting for the cadence: damage stops growing, damage becomes permanent, a failing test reproduces the bug, or the release estimate moves past its threshold |
| 33 | IC | #unl-validators | Holds the one synthesized picture, appends the record, and states which track is being waited on every time that changes |
Phase 3 - the interim responsesteps 34-50
| # | Who | Where | What happens |
|---|---|---|---|
| 34 | Advisor Leader | #unl-validators | Recommends acting ahead of root cause while damage accrues: the rate, the measure, the cost if wrong, the time to reverse |
| 35 | IC | #unl-validators | Decides off two lines of the report: accruing and irreversible, act now; accruing and reversible, act if the rate is material; stopped, let the hunt run and say so aloud |
| 36 | IC | #unl-validators | Asks the Engineer Leader whether the measure makes the fix harder; tells the UNL Leader to prepare the instruction unpinned |
| 37 | Engineer Leader | to the IC | If the measure conflicts with the fix: the conflict in one sentence, an alternative, the Coordinator's call. Said once |
| 38 | IC | #unl-validators | Hands the UNL Leader the instruction as one of the five actions, with the do-not line, verification, target, deadline, and stall threshold |
| 39 | UNL Leader | before pinning | Checks five preconditions: authorized, verifiable, do-not named, bounded, reversible. Any missing: back to the Coordinator before pinning |
| 40 | UNL Leader | #dunl-operators-only | Pins INSTRUCTION n: from him by name, authorized by name, the action, the do-not, verify before acting, target and deadline, done in this thread |
| 41 | IC | the instruction thread | Countersigns every instruction shipping no signed artifact, restating the action in his own words. Two independent names, or operators do not act |
| 42 | operators | own hosts | Run the three checks; all pass, do exactly what it says, post done. Under the incident regime done is an acknowledgement, not a vote |
| 43 | an operator | #dunl-operators-only | A failed check stops that operator cold. Cannot comply: said in the channel with the reason. Silence is the one failure |
| 44 | UNL Leader | #dunl-operators-only | Counts two numbers, reports the lower: acknowledged from the thread, observed from the network via the Engineer Leader. Acknowledged-but-not-observed is a failed upgrade and interrupts the Coordinator |
| 45 | UNL Leader | #dunl-operators-only | Runs the rollout clock, posting the count on every mark even when it has not moved, and saying so |
| 46 | UNL Leader | opted-in channels | Chases the silent in fixed order: pin, chosen channel, backup channel, emergency contact, then the Coordinator. A chase never carries the instruction |
| 47 | UNL Leader | #dunl-operators-only | Records each decline with its reason and keeps the ceiling; the ceiling dropping below the target interrupts the Coordinator immediately |
| 48 | UNL Leader | #dunl-operators-only | On a damage report from an operator, grades the response: node down or spreading, full HOLD; degraded but followed as pinned, pause new starts; a named deviation, nothing held and the do-not restated to the whole channel |
| 49 | UNL Leader | #dunl-operators-only | On a spoofed instruction: everyone stops, everything real is pinned here, whoever acted says so with no consequence, and the Coordinator is interrupted |
| 50 | UNL Leader | #dunl-operators-only | Announces the threshold the moment it lands and keeps contacting the tail past it |
Phase 4 - the fixsteps 51-57
| # | Who | Where | What happens |
|---|---|---|---|
| 51 | Engineer Leader | interrupt path | Reports root cause confirmed: failing test path and commit, reviewer, cause in one sentence, the trigger, fix approach, build estimate as a range |
| 52 | Engineer Leader | #unl-validators | States the release ETA: earliest to latest, confidence, the gating step. The only ETA in the incident |
| 53 | IC | #unl-validators | Restates the ETA and the count outward, each with its source's timestamp. Everyone else quotes him; no third figure exists |
| 54 | Advisor Leader | #unl-validators | Speaks the one recommendation, with the engineers' concur or dissent inside it. Two competing recommendations never reach the Coordinator |
| 55 | engineers | restricted pipeline | Build and sign under M-of-N custody, hashes produced, nothing unreleased touching general-purpose CI |
| 56 | Engineer Leader | #unl-validators | Reports release readiness: tag, custody, hashes, what was tested, what was skipped and the risk |
| 57 | all | steps 38-50 | The chosen response executes on the identical instruction path as the interim: same actions, checks, countersign, counting, and chasing |
Phase 5 - the picture changessteps 58-61, whenever triggered
| # | Who | Where | What happens |
|---|---|---|---|
| 58 | IC | #unl-validators | Re-declares on any severity or condition change, immediately, carrying only the line that moved. Moving off Unknown is a re-declaration, not a correction |
| 59 | COMMS | #disaster-recovery-news | Mirrors the escalation notice the moment it exists, outside the cadence. Every routine update carries condition, severity, and damage even when unchanged |
| 60 | IC | #unl-validators | Hands over with the written snapshot; the incoming Coordinator accepts by name in writing. No acceptance, no handoff |
| 61 | UNL Leader | #dunl-operators-only | Does not rotate with the Coordinator. Hands the net to his own backup on his own timing, carrying the running chase state |
Phase 6 - the endsteps 62-69
| # | Who | Where | What happens |
|---|---|---|---|
| 62 | IC | #unl-validators | De-escalates while the network is healthy and work remains, and orders the unwind here, in reverse order, so the 24-hour window runs with the measures off. The bridge closes here |
| 63 | UNL Leader | #dunl-operators-only | Pins the revert as a numbered instruction to the identical verification bar, counted to the same threshold as the rollout |
| 64 | IC | #unl-validators | Declares stand-down only when the four conditions hold as facts, with the cause line: established with the failing test named, or open with the person and place carrying it |
| 65 | COMMS | public + news feed | Publishes the stand-down notice naming the revert time and any retained measure |
| 66 | COMMS | public | The embargo lifts at stand-down. The advisory and CVE publish on his timing, then the retrospective and community session |
| 67 | UNL Leader | #dunl-operators-only | Closes the rollout: final count, last acknowledgement time, names now tracked outside the incident |
| 68 | UNL Leader | the rollout record | Hands the retro the four timestamps and the per-operator table. Held out of the public repo: a map of who answers slowly is a targeting list |
| 69 | all | retro | The retrospective runs on the incident template, and what the run exposed lands as tasks |
The board with every rule linked to its owner: governance/timeline.md