How an old tie with a new suit reminded me of proper testing strategies

A few months ago I purchased two new suits for myself.  A blue one and a gray one.  They are a bit more modern, in the fashionable sense, and my wife was relieved I'm finally wearing something that looks somewhat reasonable.  A few weeks ago I put on the gray suit, and picked out a yellow tie, thinking "gray and yellow go together, right?  suuurree..."  Turns out I was right, and the old yellow tie that never got any feedback, suddenly became the catalyst for comments such as "nice suit!" and "is the tie new?"  Actually, its not.  I've had it for years, but apparently it looked so nice with this suit, it really made an impact.

Testing software (and hardware for that matter) is a lot like matching the right tie to the right suit - except in this case you're looking to break it.  You see, in testing we run the same tests over and over again.  Often we run on the same environment and configurations.  Once we've passed the initial bug possibilities, the next few runs will very unlikely find anything.  But if you choose to change something slightly - a different version number, a small configuration change, a slight modification to the scale or loading rate - all of a sudden you can come across a bug that will block - and those are the best ones. 

Case - Once I was testing an installer.  I've tested this installer fifty times at least, and the developer swore this build was the one that will be ready for release.  So I prepared the server and noticed the configuration was using an upgraded version rather than what we tested in the past - a clean install.  "That's different." I thought to myself.  So I tried it on the upgraded component instead.  Lo and behold we discovered a hardcoded verification function that failed miserably.  We had to remove the verification (proving it wasn't necessary) and create yet another build.  If I hadn't changed the configuration of the test, I never would have found the bug.  

Sometimes you have to try the old tie (ie - test case) with a new suit (new configuration) to get the most out of it (find the bugs).  And that's why it's always good to shake things up a bit.