The impetus for this note is to collect aspects of modern computer display technology with an eye towards organizing the software along capability lines. I hope to accommodate these patterns.

Notes on computer display technology

Early raster displays, such as experimental PARC machines and the Lisa and Macintosh, were accessible to software by storing directly into the refresh buffer (video memory, frame buffer) which was accessed as RAM. Hardware that was not closely associated with the CPU would continuously copy from the buffer to the screen, whose image would otherwise last only a fraction of a second. Primitive graphics routines were cognizant of “clipping paths” so as to modify only pixels owned by the current caller. These primitives were hierarchical and higher levels would compute bounding boxes and perhaps discover that the entire request, such as to draw a circle or line segment, was entirely outside the current window. Thus was screen space allocated and protected.

A clipping path was often a rectangle, but another window might partially obscure the one into which the current image was being drawn. When a window was moved it was necessary to call an application’s draw code again to redraw the pixels that had been revealed by the move. Pixels already on the screen in a moved window were merely copied.

Another early Mac trick was to temporarily displace pixels over which a menu appeared. They would be replaced when the menu was dismissed. This required that the real owner of that space not write for that would deface the menu and leave the displaced pixels unmodified. During these times the Mac would allow no application drawing whatsoever. Savvier drawing primitives might have updated the displaced pixels but this seems too complex perhaps.

When images were rendered directly into the refresh buffer in a windowed system a clipping path advised drawing primitives where not to draw lest other windows be corrupted. One result is that the pixels that resulted from a draw operation might not have been captured and were thus unavailable for inspection. You might call this blind drawing. It is a sort of write only memory.

That drawing was predominately blind allowed another significant Mac trick which was a mode where some level of drawing calls were recorded rather than performed. The application could request this mode and take delivery of the record. The system could perform from the record. The Mac PICT file format was mainly such a record. OpenGL has a similar mechanism.

Assignment semantics is an implicit assumption of all of the above. The most recently stored pixel is what is seen. This assumption pervades most programming and is thus natural, but not always convenient. One place where this is inconvenient is in hidden line and surface logic where the frontmost pixel is to be displayed and not the last pixel to be processed for a particular point in a raster. Sometimes it is convenient to suppress creation of all but the frontmost pixel. This saves processing time but requires logic that may be complex or slow. A common alternative is to include “z-value” with the pixel. Placing such a pixel in a raster takes place only when the z-value of the new pixel is less than that of the z-value already in the raster.

A similar variation on simple assignment semantics is pixels with alpha values which do not obliterate the pixels that they displace. Instead the brightness values for each of the several colors of the new pixel are linear combinations of the old and new pixels. This provides the effect of translucent images thru which “lower” images can still be seen.

An early hardware enhancement was the sprite which caused some small image to displace some variable location on the screen. This processing was done during refresh between the frame buffer and the screen. It its main use was to draw a cursor that moved as a mouse moved. Early Macs, at least, lacked a sprite. The cursor became invisible whenever an application was allowed to write in the frame buffer lest the cursor be written on instead of the application’s logical image.

The Graphic Processing Unit

Hardware GPUs often take over much of the pixel pushing work from the CPU. They copy one 2D pixel array to another with a raster to raster conversion. Both arrays are decomposed into triangles so as to approach a continuous nearly differentiable map. The interior of each triangle undergoes an affine transformation. The Mac’s “Genie effect” uses this to morph windows to and from the “Dock”. I think that the target is scanned by triangle. The source triangle is identified and fetched so the target pixels can be computed, perhaps by interpolation. Somewhat orthogonal to this is computing the coördinates of the target triangle by means of 3D projective transformations so that one small change to the transformation parameters can move or rotate the image of a 3d object. For this purpose 3D coördinates are provided to the GPU. This is consistent with the z-buffer logic, of course.

GPUs may also do surface reflection and lighting calculations by computing or being given the normal to a 3D surface.

All this function is available to application code thru OpenGL. I suspect that all of this function is also available via software, perhaps using vector SIMD features of modern CPUs. There may be an energy cost here to consider.

I doubt that this logic is done in the refresh loop for there is no simple limit on the RAM bandwidth required when a large region is mapped to a small region. Perhaps power is required only when either the source image or the map changes or is changing.

The high performance copying available from a GPU supports the idea that graphics routines need no longer be aware of clipping paths. Clipping logic is relegated to the copy logic and not dispersed in the many drawing primitives. This would simplify them and make them faster. Faster, that is, except for those situations where the drawn image has no presence on the screen. Besides being obscured by another window, the image may be scrolled entirely off of the page. Some truncation of invisible drawing, especially in animation, seems strategic. Hidden surface logic is presumably integrated here before the real work begins.

Some reasons to skip pixel pushing:

I expect there are others.

Image construction is typically done thru many subroutine layers. The system or even the hardware typically provides the lower layers. Implicit in most of this traditional design is that later drawing displaces earlier drawing—that computing and placing of a pixel is like an assignment statement and that is indeed heavily exploited.

Timeliness

There is also the issue of time. Most of the above is about producing an image on the screen. A movie is a timed sequence of images (=frames). It is sometimes strategic for successive images to share pixel work, meaning that not all pixels need be rendered each frame. Also when the user moves a window about the screen with the mouse the pixels of the window are expected to remain visible within the moving window. Time issues for the image get tangled up with issues of when image generation code must run. This is perhaps the main impetus for call backs wherein the application delivers a procedure to lower levels which is to be invoked when there is indeed a real need to present the pixels. See Continuously Evaluated Functions for more details.

I speculate here about how to organize window software with capability discipline.