I describe here the high level strategy that we adopted for MP, as it solved some related problems that were already crucial for single processor versions.

These ideas are much like schemes for transactions where an attempt is made that may turn out to be infeasible and must then leave no side effects. This is commonly called commit-abort logic. We were vaguely aware of this discipline and I am sure that it was not original with us. Still some of the details of our implementation may be of interest.

Perhaps ideally a CPU spends most of its time executing user code in user mode. This code will occasionally invoke the kernel to change some part of the world that is not represented in the memory space of any domain. Most such kernel invocations are feasible and the code must handle the success case quickly, but be prepared for possible obstacles. There are several forms of obstacles:

• The semantics of the invocation is at fault and the operation must not be done and the invocation rejected. In particular the untrusted user code may designate two kernel objects with the requirement that they be distinct. The kernel must ensure that they are distinct lest the system become corrupted. A domain may try to jump to itself or a malformed domain may be invoked.
• The operand (page or node) is not in RAM and must be summoned; meanwhile the domain must be stalled.
• The mode of preparation of the operand is inappropriate for the code available to preform the transaction, and the operand must first be suitably prepared.
• Another CPU is just now operating on the operand and some sort of delay, at the very least, is required. Upon discovering this unusual event, some fairness strategy might be required to avoid a possible dog fight.