Self modifying code was once the only way to go. Modern machines almost eliminate the practice. Yet there is the recently popular Just in Time Compiler, the JIT. For several decades efficient sort routines have for each invocation of the sort, generated new machine instructions for the comparator function that examines two records to determine which one comes first in the result.

Algol 60, Pascal, Algol68 and PL/I all provide nested procedure definitions whose body may refer to parameters or variables defined in the outer procedure. These languages also have allowed procedures as parameters to other procedures. C’s syntax allows nested procedures but the official language definition disallows it. Most C compilers do not allow it either. In this situation it has become standard for C compilers to represent a procedure pointer in one word which is the address of the code that performs the procedure.

The languages that have classically allowed nested procedures represent a procedure pointer in two words, sometimes knows as “fat function pointers”. One word locates the code and the other locates the stack frame for the invocation of the outer procedure where the procedure pointer was created. This is necessary when there are variables or parameters that occur freely in the procedure body. These must be declared in a block that contains the function definition. The stack location value locates the generation of such free identifiers in effect where the procedure pointer was produced. The code must refer to these stack values by means of a pointer to their frame. There are at least two different ways to accommodate references to yet outermore procedures.

gcc and some of IBM’s C compilers allow nested procedures. These compilers adhere to the rule that the yield of different compilers should be compatible when loaded together. gcc sought a scheme to represent the procedure pointer in one word so that its output would be compatible with older compilers that used one word to locate the code that performed the procedure. In official C functions are not nested and so the compiled code never needs access to outer scopes. I thought there was no such scheme for accessing outer scopes until I saw the claim that gcc supported nested procedures. When gcc must compile code to produce a procedure pointer to a procedure whose body mentions stack variables of other frames, it compiles code to compile code on the stack and then produces the address of the new stack code for the procedure pointer. The stack code knows where it is and loads its own address into a register that will be used by the nested procedure code, to which it then branches. Here is a note on the gcc compiler logic that does this. I am not chagrined at having thought it impossible.

Here is some code that illustrates this most excellent hack:

#include <stdio.h>
typedef int w(int);
int x(w);
int main(){
  int m = printf("Hello\n");
  int z(int q){return q+m;}
  printf("%d\n", x(z));
  return 0;}
together with the separately compiled
typedef int w(int);
int x(w q){return 12+q(3);}
Here is one of several patterns that use nested functions in an essential way. This is sometimes referred to as the funarg problem. Here procedure nesting supports local security.