One thing to add is our test wizards have also conjured up a system we call the "try server", which allows the execution of all the test suites asynchronously on fast machines before you commit. So, you don't have to wait for your laptop or machine you're working on to run the tests, you just kick it off and get an email with your results in a few minutes. This makes it so it's painless to be sure you'll never commit a red build.
Yeah, I was thinking about doing a similar thing here at Cloudkick.
Currently, our test suite is not that large and it takes around 6-8 minutes to complete, but with testing every minute counts (testing is generally not that fun and a slow test suite just makes it more painful).
We have a two types of tests:
- Twisted tests - this tests runs asynchronously and finish pretty fast so they are generally not that problematic
- Django tests - Django tests don't run in parallel so they are pretty slow. Recently, I was playing around with the Django test runner and I have made some modification to it to run the tests in parallel. Now the Django tests finish around 50% faster.
The only problem with this solution is that it is a "hack" and it requires some modifications to the Django core (I guess I should play more with the nose parallel test runner).
We also use some other "tricks" which make tests run faster - for example, MySQL data directory on a test server is stored on a ram disk.
Coupled with slow tests is the fact that unless you have something like the try server it ties up your machine while the tests are running. Being able to just kick it off and continue working on the next feature or bug goes a long way to reducing the pain.
