molina/how-it-works.md

docs

How Molina works

After an incident nobody could reproduce, engineers argue about the cause. I turn that argument into an experiment, and you get a result either way, with a recording your engineers can read and check.

A run starts with a container image and ends with a result. I have run experiments on a disk that says saved before it is, a cluster of servers with its network split, and a program with racing threads.

A run from a container image to a resultFour steps from top to bottom. Your container image goes into a virtual computer on my machine, whose disk, network and clock are Molina's and which runs your software as it is. I cause failures there: a disk that says saved before it is, the network between servers cut, the power cut at a point in your workload that I choose. Your promises are checked after each failure. When a promise breaks, you get a replay code, the smallest set of failures that still breaks it, and a recording.Your container imageyour software, as it isA virtual computer on my machineDiskNetworkClockall three are Molina'sYour softwareI cause failures here:a disk says saved before it isthe network between servers cutthe power cut where I chooseYour promises are checkedWhen a promise breaks, you get:a replay codethe smallest set of failuresa recording to read and checkI run it again on request
  1. You hand me your software as a container image, which is your software packaged to run on Linux.
  2. Your software runs inside a virtual computer on my machine, and the disk, the network and the clock of that computer are Molina's.
  3. I cause failures there, such as a disk that says saved before it is, the network between servers cut, or the power cut at a point in your workload that I choose. After each one I check the promises you and I agreed on.
  4. When a promise breaks, you get a replay code, a recording you can read and check, and, where the failure can be reduced, the smallest set of failures that still breaks the promise.

In more detail

Nothing is installed on your side. You cannot run Molina yourself yet.

If nothing breaks, you get a recording of what ran. A clean report is still a result: what was explored held up, and nothing more.

A replay code names one failure, and I run it again on request. Replay matched exactly in five of my runs. Replay is not yet measured on your software until I run it. Shrinking takes failures out one at a time while the promise still breaks. I have measured shrinking on a cluster bug and on the disk that lied about flushes. For a compose file, molina campaign --compose reduces an isolate and a stop.

Replay and shrinking, with the figures from my runsThree panels, from three different runs. First, a bug in a cluster library, Akka.NET issue 8508: a failure schedule with nodes running for 83 s shrank to one with nodes running for 13 s. The 13 s schedule stopped the cluster client in 3 of 3 re-runs. Second, the same replay code run again 3 times, and each replay gave the same verdict. Third, a database on a disk that said saved before it was: 30 skipped flushes shrank to 1, in 13 runs, and 3 of 3 re-runs of the one flush lost a saved write.A bug in a cluster libraryAkka.NET issue 8508: how long nodes ran83 sshrunk13 s3 of 3 re-runs stopped the client.None of its 3 one-step neighbors did.The same replay code, run againfoundagain 1again 2again 3Replayed 3 times: the same verdict each time.A disk that lied about flushes30 skipped flushes1 flush still loses a saved write13 runs. 3 of 3 re-runs lost a saved write.
  • Akka.NET issue 8508, a bug in a library for servers that work together: a failure schedule with the nodes running for 83 s shrank to one with the nodes running for 13 s. The 13 s schedule stopped the cluster client in 3 of 3 re-runs, and none of its 3 one-step neighbors did. I wrote that schedule from the bug's own mechanism, so it shows that replay and shrinking work and says nothing about finding unknown bugs. The record
  • The same schedule was replayed 3 times, and each replay gave the same verdict on the client, to the millisecond.
  • A database on the disk that lied about flushes: 30 skipped flushes shrank to 1, in 13 runs. Each of 3 re-runs of the one flush lost a saved write. Exact replay was not shown on those runs. The record

These are my own runs on one machine. How exact is the replay? What you get at the end

The short version, on the Molina page

Book the free 30-minute call
:SageTech --ready