After reviewing the history of the bottom layer of comm protocols I think that there is no point to the blocks of packets that was part of the previous plan. The new idea is that payload input from the fiber goes continuously to a ring (circular buffer) in uncached DRAM. Headers are still preceded by an HS (new header symbol). Headers are limited to 128 bytes and the hardware deposits them in a consecutive cached memory locations, in another ring. As in format C the headers are augmented by the hardware to point to the payloads. Unlike format C the program can find the headers because the hardware puts the header length before the header. The payload size and DRAM address follow the header. See more comments on hardware here.

Output to the fiber is driven from a similar ring of headers written by software and read by hardware.

Error Control

Among the possible elements of the output header ring is a command to the hardware to send the accumulated error syndrome. The offset of this element is recorded in the ELR, a ring of such values and the index of that ring is included with the syndrome. This causes the receiver to get error status in his input ring which covers the data transmitted since the previous syndrome. Acks from downstream refer to this index and this normally allows the space of a block of sequential packets to be freed. The ELR leads to the output headers.

On the receiving side the receiving hardware deposits the error syndrome in the input header ring and updates that ring’s cursor.

Format D Fiber Form (datagram on the fiber):

This is a minimal version. The special symbols might be more than 10 baud if necessary to reduce the probability of seeing them when they were not transmitted. Other redundancy might be worthwhile like the payload length in the header. Depending on hardware engineering, we might include error control information following each PM and perhaps HS too. In case of forward error control an error syndrome might follow the header for the software to apply to the header and/or payload.

Format D: IHR = Input Header Ring (header deposited by hardware in CPU’s cache):

 The software must verify that there is enough room in a header for the turn ops promised by the turn op count. The software and hardware might find it mutually agreeable to align the PL field on some power of 2 like 4 or 8. See this about allocation of DRAM for payloads.

Format D: OHR = Output Header Ring (Prepared by software to drive transmission):

 Formats C and D both assume that the large majority of transmitted bits are in payloads.

Variations on the Header

Both header rings: Datagrams, Circuit packets, Error control blocks, Needles (for new circuits), Pause, and poll, Dissolve circuit. too The “packet-on-link count” helps get back in sync after a loss of sync. 8 bits are probably not enough. The header is getting crowded.

The writer of a Pause header does so when there are no other headers to write. (???)

The rings

Three sorts of ring (circular buffer) are proposed above:
  1. Uncached DRAM for the consecutive payloads of some particular fiber, written by hardware, read by hardware on the way out to some other fiber,
  2. Cached memory for the consecutive incoming headers and other administrative stuff from some particular fiber, written by hardware, read by software,
  3. Cached memory for the consecutive outgoing headers and other administrative stuff for some particular fiber, written by software, read by hardware.
Each of these would be allocated on a multiple of their size which would be a power of two. For each the software would be able to read the current location of of the hardware buffer cursor and perhaps detect being lapped (horse or auto racing jargon). Being lapped is probably a common failure during system debugging or system overload. Circular buffers ending on page boundaries can be mapped with conventional memory maps so that even data that wraps the circular buffer appears to the CPU at contiguous virtual addresses.
Another idea is to alleviate DRAM load by having the channels get and put full DRAM pages. CPU access to such DRAM could be allowed (via the cache) if that access did not erase the DRAM page being accessed by the channel.

Another unrelated idea is for transmission hardware to look ahead in the output ring and select packets on the basis of being in DRAM that currently has the capacity to deliver them. This might violate unspecified order delivery rules. Packets from circuits would need special attention in such a design.

Perhaps some packets can be marked by the sending node as owning DRAM at the destination. This would be conveyed to the sending hardware to put a different special header mark so that traffic with reservations will not be lost to stand-by DRAM overflow. The reserved space for quota traffic can be shared among traffic with quota. Reserved DRAM can be allocated in a ring at the same addresses as the stand-by rings and this avoids ruining any preplanned DRAM bandwidth arrangements.

More generally there might be a hardware nexus that allocated DRAM capacity about a millisecond ahead. Such a plan might avoid DRAM congestion and concomitant packet loss. Stealing from the SDLC world we can put idle flags on the fiber when the DRAM cannot deliver payload as fast as committed.

Further details.