An 8-byte timestamp is not a lot of state. You could have a million concurrent connections and use only 8 MB for a file descriptor -> timestamp lookup table. The TCP send/receive buffers will exhaust your system memory long before the timestamps do.