I'm co-founder at a hedge fund startup. Much of our code may not be extremely sensitive to small delays, but picking the right datastructures makes a big difference even on small workloads. And we do have some big simulations where performance definitely counts. But mostly, picking the right datastructure isn't that hard most of the time, and is a painless kind of optimization.