Perhaps the program counter is the seminal element of the von Neumann computer.† Turing certainly had the idea earlier. It is too simple an idea to deserve accounting for its invention. It came about with storing programs in the same sort of memory as the data where information is stored in numbered cells. Remembering a number thus sufficed for the computer to remember what it was doing as it finishes obeying an instruction from some program in memory. It reminded the rest of the machine what instruction it was working on just now. The most basic operation of the machine was to fetch the instruction indicated by the PC, and concurrently:
Actually the program counter would be better named the instruction address register and some architectures call it that.