Molina documentation
Molina is a service I run on my own machine. You hand me your software as a container image, and nothing is installed on yours. I make its disk, network and clock fail the way real ones do, then check the promises your software has to keep.
So far I have run experiments on a disk that says saved before it is, a cluster of servers with its network split, a program with racing threads, and a database whose power was cut at 40 different points.
Molina is for a business owner or head of engineering whose product depends on a database or on several servers working together, and for the engineers who run that product. They are responsible for databases, distributed systems, infrastructure and data durability, where losing data a customer was told was saved, breaking a consistency promise or being unable to reproduce a failure has serious consequences.
The sales page is short. These pages hold the detail.
- How it works: from a container image to a replay code, in four steps and one picture.
- What you get at the end: a worked example, from my own experiment, in the order your engineers would read it.
- FAQ: the questions buyers ask me, in plain words.
- Glossary: replay code, promise, network split, saved write and the other words, in everyday terms.
- How to read a result: what the labels on a record mean.
- What it cannot do: the software I cannot run and what I cannot promise.
- How I treat your data: what I do with your files and what can read them.
How exact is the replay?
Replay matched exactly in five of my runs. Replay is not yet measured on your software until I run it.
- An Akka.NET cluster with its network split: in 3 of 3 pairs, all 1,476 network messages had the same outcome, delivery time and contents, and the same 42 were cut. Standing: MIGHT. The record
- A .NET program with four racing threads: 96 restored copies, 90 counted, none differed. Standing: MIGHT. The record
- Postgres on a disk that says saved before it is, in exact mode: 3 of 3 replays byte for byte identical in console, timings and switch log. The switch log was empty: no network message crossed the switch. Standing: MIGHT. The record
- A NATS pair with a split in virtual time: 3 of 3 replays, all five files byte for byte identical; with another replay code, 0 of 5 files matched. Standing: MIGHT. The record
- Akka's own test, NodeUpSpec: 36 of 36 copies byte for byte identical on an idle host and 36 of 36 on a loaded one. Standing: UNRATED. The record
What I ran
Each record says what it shows and what it does not show, and gives the conditions where they fit.
- Postgres, a database, on a disk that says data is stored when it is only in memory
- QuestDB, a database, power cut at forty different points
- A program with racing threads, restored from a saved starting point
- Akka.NET, one server cut off from its group
- Akka.NET, a network split, replayed
- Akka.NET issue 8508, found again from the commit before its fix
- NATS, a message broker, partitioned in virtual time and run again on one seed
- Akka.NET, a node joining a cluster, run again from a snapshot
- Postgres, a lying disk, shrunk to its smallest set of lying flushes
- TCP, the shortest partition window found for one exchange
- Akka.NET issue 8508, its plan shrunk