Barrelfish
It has been several years since I glanced at Barrelfish.
I am plowing into this document.
CPU drivers are single-threaded and non-preemptible.
They run with interrupts disabled on their core, and share no state with other cores. CPU drivers can be thought of as serially executing exception handlers for events such as interrupts, faults, and system calls from user-space tasks.
Each such handler executes in bounded time and runs to completion.
If there are no such handlers to execute, the CPU driver executes a user-space task.
These are good rules.
There are different good rules.
It is a good sign that they begin here.
Keykos said early on: the kernel runs privileged, unmapped, without interrupts, even even without page faults.
All other code runs mapped, interrupts enabled and with demand paging.
Contrast with Keykos
I like their style of description and I will thus contrast Barrelfish with Keykos.
MP
MP is more important today than when Keykos was developed.
We had detailed plans for MP but no code.
We planned shared RAM holding shared mutable data and a system of classic shared and exclusive data locks.
The Barrelfish ‘CPU driver’ is much like the code that defines the behavior of the Keykos domain kernel object.
“If there are no such handlers to execute, the CPU driver executes a user-space task.”
This suggests a queue of messages to the kernel.
This is one convenient mechanism of inter-core signals.
Keykos has not settled on a solution to this but another Keykos kernel rule is that all storage (including that for message queues) be ascribed to a user account and instantly recoverable (like space in a cache).
I await hearing about the discipline to avoid queue overflow.
Keykos and Barrelfish kernels are alike in that there is no kernel stack state except during one of the kernel events.
A dispatcher can be thought of as the local component of an application (often called a domain in Barrelfish) on a particular core.
An application which spans multiple cores (for example, an OpenMP program) has a dispatcher on each core that it might potentially execute on.
It isn’t clear which the domain is: the dispatcher or application.
The Keykos domain can run on any available CPU in a symmetric system.
Some hardware imposes a cost in switching in that domain specific data is physically associated with a particular core.
Cache and floating vector state comes to mind.
We have found that keeping it logically separated is easy, while admitting some execution cost.
It sounds as if application logic is necessary for homogeneous CPUs to serve a population of users.
From the description of the relation between the kernel and the dispatcher it sounds more complex to me than the Keykos model; perhaps an illusion.
Operating system tasks in Barrelfish are always single-core by design, and therefore have only one
dispatcher. Applications, however, frequently have more, one on each core.
How comes it that a system task (performed by kernel code?) has any dispatchers?
Is a dispatcher code?
Does it run privileged?
They refer to K42 in which a ‘dispatcher’ is code that has its own address space some of which is pinned.
My opinion is that classic symmetric MP with coherent memory is strongly limited by hardware considerations to no more than 8 CPUs.
However even four homogeneous CPUs with a shared CPU ready queue are an important configuration to support.
The dispatcher design with distributed CPU ready queues don’t seem to support this.
Perhaps coherence between slower caches is strategic.
Back to Barrelfish:
Keykos planned on a type of (Keykos) domain for each sort of CPU.
Calls between types would necessarily be between CPUs perhaps via some sort of kernel message queue which we had hitherto avoided.
Hopefully these would find their data mutually comprehensible.
Heterogeneous domain types could even be supported for imaginary ISA's given a user mode CPU simulator.