Bugs in such code can lead to security breaches. Typically this requires the penetrator to be aware of and understand the nature of the bug in some detail. The bug’s behavior is probably non deterministic. Nonetheless bugs have been seen and fixed that would have allowed exploits.
There is no division between “semantics code” and “performance code”; they are as mixed as an omelet. There is reason to believe, however, that the efficient kernel is scarcely more vulnerable than an unoptimized version.
There is yet more code, called “CHECK” that checks each of the optimizations of the kernel, which are each carefully documented. CHECK is run before each checkpoint and even more often shortly after deploying a new version of the kernel. Virtually any error in optimization logic is vulnerable to being caught by CHECK if CHECK runs soon enough after the error.
There are, however, many dynamic optimizations of disk hardware that are not subject to such discipline. A combination of hardware storage protection and CHECK tends to detect blatant storage corruption by this code. This optimized code is trusted with such sensitive matters as page identity, however.
This is also a reason to be optimistic about robustness of an open source version of the kernel. Modifications to CHECK are a red flag warning of any short cuts taken regarding the optimized data structures.
A feature that we did not implement would be kernel primitives, the kernel irritation key, to drive the kernel thru routines normally invoked for reasons of performance. It would be reasonable for user mode diagnostic code to be able to cause the unpreparing of a node, an act with no semantic impact unless there are bugs in that code. Such hooks would facilitate driving the kernel into corner cases. They might be either closely held or rescindable in order that diagnostic code could not impact performance in normal use.