Support smart network fault injections through predicates

[aoak]

Problem: To simulate certain failure scenarios, it is important to have control over when network failure happens. For example, the application layer sends message A and B in succession, and we want the network partition to kick in after A is sent but before B is sent on a TCP connection. Equivalent on UDP maybe something like when A and B are sent, drop A but allow B.

Currently it is hard to simulate those kinds of failures. It would be great to add a support to take a predicate that is run on message stream and when the predicate returns true, a fault is introduced.

For example, following would introduce a partition between host 1 and 2 after a particular message is sent from host1 to host2

turmoil::partition_after("host1", "host2", |msg| matches!(msg, Msg::PrepareAck);