Performance Testing Taxonomies
July 28th, 2008 in Performance Testing
I wanted to review some of the books I had on performance testing, so I decided to share some of the results here. Specifically, I was looking for some taxonomies related to performance bugs, tests, and risks:
- Performance Testing Guidance for Web Applications by Meier, Farre, Bansode, Barber, and Rea has an excellent chapter (Chapter 3) on performance test risks.
- Testing Computer Software by Kaner, Falk, and Nguyen has an Outline of Common Software Errors (Appendix A) that has a couple of issues related to performance.
- Testing Applications on the Web by Nguyen, Johnson, and Hackett has a chapter on performance testing (Chapter 19) which has some good lists. There’s one on key factors that effect response time or performance, one on common hardware problems, one on common software problems, and a list for optimization considerations.
- While it probably wasn’t intended for this use, I found that Appendix E on HTTP Response Codes in The Web Testing Companion by Lydia Ash gave me a lot of performance testing ideas. Appendix H also had a short but useful list for Performance questions.
Overall, I flipped through 14 hardback books. That was the best list I could come up with (which depressed me a bit - perhaps I have the wrong books, but I don’t think so). After that, I went online and I was able to find the following:
- A Taxonomy of E-Commerce Risks and Failures by Vijayaraghavan (which includes an outline of FMEA).
- Similar to the previous paper, Bugs in the Brave New Unwired World: A Failure mode catalog of risks and bugs in mobile applications by Jha and Kaner has a lot of items that relate to performance.
- A really good performance testing risk taxonomy on Roland Stens’ site.
- A less directly related, but no less helpful, reference that I always use (even when performance testing) is Bach’s Heuristic Test Strategy Model.
I be interested in knowing about some taxonomies I may have missed. I’m looking specifically for performance defects and/or risks, but I’m fairly good at transferring from material that’s similar but not directly related.
Mike:
It seems like I get asked the question every few months, what would I recommend for reading about perf testing. I believe what you’re trying to do is list what’s helpful and thought I would share my list. This list is literally pasted out of an email I sent to a professional friend so it’s written as such.
***
Resources for learning:
* segue’s unpublished books - they have two if you call them and you’re very nice and somewhat lucky you can ask them to send you a copy. Ask for both books. The beauty of these two books is they’re non-commerical - no sales pitch just a solid overview. I have to confess I read them ages ago when I was starting out in perf testing back in 1998 - 2000 but my memory was that the overview and pictures was helpful. I’ve handed out all the extra copies I had. Once after some thorough digging on Amazon, I found one. Don’t forget Segue has become Borland.
* I recommend - of course - Scott Barber’s work - off the top I can’t recall which articles in particular but I do have a printout about an inch deep on my bookshelf and could get the titles. I especially like his article on “investigating.” Dig through past STP mags for the pieces. Scott’s Google video on perf testing is great and is well, Scott as he is in person. Good stuff.
* At the risk of being obnoxious, I like the piece I wrote ages ago for STQE magazine on using web logs. Logigear reprinted an updated version recently in their company newsletter. You can also find the pdf off my website (www.karennjohnson.com ). I don’t see much in print on weblog or analytics but I think both are important related topics.
* I was doing perf testing at the start of the dot com, at the time kicking around the STAR conferences - Alberto Savoia and myself were talking performance. I used to listen to him at STAR’s and we’d talk but its been some years now and he’s moved on to build a company. Run a search on Savio on the Sticky Minds site and look for his work in the early 2000, 2001 and you’ll find his perf test stuff there.
* Recently Scott has published a book through MS Press. You can find the pdf online and a hardcopy will be coming out soon. I previewed some chunks of the book but have to confess I haven’t read the whole piece front to back or the final version.
* I also recommend a book I read last year I picked up “Integrated Approach to Web Performance Testing: A Practitioner’s Guide.” You can find this book on Amazon. I’d like to meet/exchange with the author because I think there are some good bits in the book.
* My paper from CAST conference (2006) on compiling perf test results is also on my site.
***
Hope this list helps. What’s interesting is that most of the resources aren’t singular books you can just order.
Karen
Good info, thanks, Karen!