On very fast local systems and networks, 'rough idling' can go on for a very long time, until it comes time to scale.
One time I found the cause of a program that was run at regular intervals and was taking a few minutes to complete running. It would run approximately 42,000 SQL select calls every time it dropped into a new directory to parse some files. It only took 10 minutes to process the entire directory structure. So the program was actually incredibly fast, considering all the work it was doing.
The catch? It only needed to run SQL select 42 times per directory; the other 41,958 calls were not necessary. As far as I know it was never fixed, because it only took 10 minutes to run.
One time I found the cause of a program that was run at regular intervals and was taking a few minutes to complete running. It would run approximately 42,000 SQL select calls every time it dropped into a new directory to parse some files. It only took 10 minutes to process the entire directory structure. So the program was actually incredibly fast, considering all the work it was doing.
The catch? It only needed to run SQL select 42 times per directory; the other 41,958 calls were not necessary. As far as I know it was never fixed, because it only took 10 minutes to run.