Memory technologies (memristor) are on the near horizon where write count limitation which increments and decrements to a reference count would be fatal. The object table would seem required there. Swift is not yet confronted by these.
References to counters is complex in case of multiprogramming. Keykos has untested plans, but there is a cost.
An idea for reference counting. My objection to referencing count has been that objects banished to memory that is fast to read and slow to write is that incrementing and decrementing the counters is too expensive.
It now occurs to me that the decision to put something is seldom mutable memory is accompanied with a decision to form a unique pointer to the real new address and next to that pointer keep the counter. All accesses to the object are indirect thru the unique pointer.
Of course this is reminiscent of an old Apple table of pointers thru which you “always go”. The problem is that some subroutines may change the real address as a side effect and any change must invalidate all caches of the real address, for instance on the stack.
To address that we assume that languages do not reify that counter space. Compiler optimizations to bypass counter space are known to the compiler and direct pointers on the stack or other continuations to objects or sub-objects can be found there by stack frame type savvy GC code. All it needs to know is what words in the frame are pointers. This means that any direct pointer into an object must have an offset which is not negative!