What you get at the end
From my own experiment, with no customer involved.
Akka.NET is a library for servers that work together. Its issue 8508 was a bug, later fixed. Below is the report Molina gives for the commit before the fix, as your engineers would receive it.
- The promise tested
A healthy cluster client is never stopped by a stray reconnect timer. A cluster client talks to a group of servers, and a stray timer fires when it should not.
- The starting state
Akka.NET at the commit before the fix, e3401469d. A server and a cluster client ran in two virtual computers, each program running for 83 s and each virtual computer lasting 101,000 ms of Molina's clock.
- The failure I caused
Replay code 1 cut the server's network connection starting at 3,509 ms, for 60,000 ms of Molina's clock.
- The sequence of events
- 2.6 s: the client's last pong, a reply from the server.
- 3,509 ms: the cut starts.
- 30.1 s: the client stops.
- The observed violation
The client stopped at 30.1 s, 2.5 s before 32.6 s, the earliest time a client that behaves would stop. The saved output counts 0 stray timers reaching the client, so the violation is read from the early stop. On the commit with the fix, the same replay code stopped the client at its own deadline, 48.1 s.
- The replay code
Replay code 1. The failure was replayed 3 times, and each replay gave the same verdict, to the millisecond.
- The smallest set of failures that still breaks it
Before: a cut from 3,509 ms to 63,509 ms, with the programs running 83 s. After: a cut from 3,509 ms to 17,509 ms, with the programs running 13 s. Shrinking took 27 questions asked of virtual computers. The smaller set stopped the client in 3 of 3 re-runs, and none of its 3 one-step neighbors did. Each moved one second: the start of the cut, its end or the run length.
- The limits of the experiment
- I wrote this schedule from the bug's own mechanism, so it says nothing about finding a bug nobody has described.
- The schedule sets a reconnect timeout on the client, and the Akka.NET release leaves that timeout off.
- The cuts came from one measured connect time, on this machine, this build of Molina and these two Akka.NET commits.
- The smallest set is smallest only against the runs I made.
- Two runs were set aside and run again after Molina refused to stop a virtual computer at an exact point.
- The saved output of this run does not include the recording, the client's settings or the messages between the two programs.
A second example
Postgres 17 ran on a disk that says saved before it is, and the power was cut at request 500. The application was told that 89 saved writes were stored, and the database held 88 rows after recovery. The 30 skipped flushes shrank to 1, flush 155, in 13 runs, and that flush lost a saved write in 3 of 3 re-runs. The result is a bundle of 5 files that refuses one changed byte. Exact replay of Postgres is not shown by that run: its background writers and timers are not under control.
How the experiment was recorded
A gate in Molina's repository built Akka.NET at two commits, searched replay codes, replayed the one it found, shrank the schedule and ran the fix. Its output is saved as 20261103T221543Z-readiness-rediscovery.out, and the runner's ledger holds a passing row for that gate. The second example's output is saved as 20261102T174714Z-finding-replay.out.