Document guarantees provided by Management API

[qnikst]

It worth documenting what are the guarantees that management API provides:

1. could messages be lost
2. could messages be reordered
3. and more complicated: is there any ordering/reliability guarantees between different Agents.

[hyperthunk]

There are no ordering guarantees between Agents, no. The event bus is serial and single threaded, so we should have some guarantees, but not many. Anything that is published by the NC will be treated in FIFO order. There are no ordering guarantees pertaining to entries published to the event bus by other processes or agents though.

Can messages be lost? Locally, no, I don't think so. Unless the forkIO thread on which the event bus relies is killed, in which case we've got bigger problems anyway. Perhaps all the event state should be isolated further? This is similar to the problem of "what happens if the NC thread terminates?" I think.

Could messages be reordered: No, I don't see how they could.

[hyperthunk]

@qnikst - how do you want this documented, and where? I'd like to close this down so that we can move on with blocked PRs...

[hyperthunk]

@qnikst ... ping! I'd like to know where you want these things documented, and I'll provide a PR today or tomorrow. I'd like to get on with #265 and #184 so we can look at a minor/patch release, so I can stabilise the version of CH on which various other libraries rely (i.e., to resolve the discussion in haskell-distributed/distributed-process-extras#6 (comment)).

More importantly, do you agree with all of the following assertions about this....

1. since the event bus uses STM broadcast channels to communicate with agents, no message written to the bus successfully can be lost
2. agents can also receive messages via their mailboxes - these are subject to the same guarantees as all inter-process message sending
3. messages dispatched on an STM broadcast channel (i.e., management event bus) are guaranteed to be delivered with the same FIFO ordering guarantees that exist between two communicating processes, such that communication from the node controller threads (i.e., MxEvent's) will never be re-ordered, but messages dispatched to the event bus by other processes (including, but not limited to agents) are only guaranteed to be ordered between one sender and one receiver.
4. no guarantee exists for the ordering in which messages sent to an agent's mailbox will be delivered, vs messages dispatched via the event bus

I can point to the code fragments which I believe back this up, if that's useful...

[qnikst]

I think I agree with all of the points, and current implementation match those.

[hyperthunk]

Okay awesome, I'll add those points to the docs for Debug/Management and submit a PR.