The NX Bit

NX is a sort of memory state from which the CPU will not fetch instructions. The page table entry of some recent CPU’s include a bit instructing the CPU not to obey instructions from virtual addresses defined by that entry. Until recently I had considered this hardware feature nice but unnecessary for Keykos.

Kurt Rodarmer convinced me that there were difficult circumstances where NX can provide a valuable security backup, even for Keykos. These circumstances include code, G, that decodes necessary messages from untrusted sources. It is always possible to get G right whence NX is unnecessary. It is possible but famously difficult. It may be strategic to borrow G from legacy software. It may be necessary, or highly convenient to give G significant authority X. It is error prone and a significant burden to verify this G is not gullible. In either case flaws in G may result in “maliciously crafted messages” including code that then wields authority X.

Typically G will be chartered to deliver messages to other system components after decoding some external message. If code from malicious messages wields X then we have merely extended the perimeter and are logically back to having a new source of messages that cannot be trusted.

There is a bright spot here that must be noted: The format of these secondary messages is under our control and thus they conform to binary message patterns whose format is designed to convey missives that the system is designed to safely act upon. Decoding these secondary messages is far easier!

Perhaps it is never necessary to give G dangerous authority but if so this in an undeveloped art.

This does not counter the attack where the original message is written over the part of the stack which holds the return address; the CPU does not execute instructions on the stack in this case and the attacker must thus branch to some address within trusted code that supports the attack. I have not explored that art. Trusted code is seldom written with that attack in mind.

Often we can afford to write the code that decodes messages in a type safe language.


This same issue applies as well to virtual machines within the system. I see no reason why QEMU cannot deliver the NX function of a virtual machine. There may be an easily followed discipline where interpreters need no more authority than the code that they “obey”. If so this needs to be elaborated and explained clearly. Keykos has long speculated about variant domain creators that would make domains that obeyed user mode instructions of some non-native ISA. Putting those ideas together may be feasible and sufficient. Perhaps this could be extended to less conventional VM’s such as JVM.