My concerns on using display hardware in a secure system is that largely undocumented function joins the TCB, at the very bottom.

Here are some concrete worries:

The display card is either a PCI card or its equivalent and has RW access to all of RAM. It may have documented function but what is its behavior when accepting non conforming signals? Very old display technologies defined “display lists” that were constructed by the application and interpreted by the hardware as it computed pixels. Some of these hardware display devices were general purpose computers. Bill Weiher wrote a calculator program for the PDP-10 at Stanford that would work with the main CPU stopped. Are today’s display cards less general?

One technique is to arrange that only TCB code produces display lists for the display hardware. In that manner only documented signals (display list contents) goes to the card and it suffices that the card have no undocumented side effects to valid display lists.

On the 370, KeyKos followed the general approach to the I/O system hardware used by all of IBM’s operating systems, and especially VM/370. User mode code would prepare a “channel program” that looked very like a real channel program except addresses in the program were virtual instead of real. The trusted kernel would translate the programs into real programs statically enforcing the protection rules. Translation was done each time the channel program was started.

The same thing could be carried out by trusted Eros/Keykos code but there would probably be a greater performance penalty and this trusted code would be very dependent on the particular display card. The code would be more complex due to the special security rules regarding perimeters of visible pixels. What to do when processing a command to draw a circle where several parts of the circle are actually visible and intervening parts are not.

Another separate issue is that the GUI introduces several security problems in its very definition. If the user moves a window and thus reveals space in an partly obscured window some systems will signal the latter to redraw.

Here is a draconian plan for a simple windowing foundation.