Some time ago I met with another tester here in Indy to provide advice on performance testing. A while ago I published some articles on the notes from that meeting, and a couple months ago I published some scans of my notes from the discussion.
I don’t see many examples of notes, so I thought I’d link to them here as well:
- The first page was an interactive attempt to understand the problem.
- The second page was an attempt to understand the goals of the performance testing.
- The third page was an attempt to document the initial usage model.
- The fourth page was an attempt to illustrate how I would approach organizing my work and how to present results to the various stakeholders.
- The fifth page was a description of a performance degradation curve and how to use it.
I went to Ford Motor Company’s website today to build and price an F-150. While I was there I found numerous issues with their site, including errors for links with no valid destination and a rather annoying Web 2.0 app that I had to close and restart no fewer than three times to get to a completed F-150. For whatever reason, it stopped accepting inputs three different times. There was also an issue with a gigantic cursor blinking on the left-side of the screen and some fields that disappears and reappeared randomly. And if you tried to navigate using your browser’s history, you’d get some interesting partially-loaded pages with error messages and you’d never get redirected to a page where the app would try to reload.
While I was doing this testing, I tried to discern a lesson or two for the QuickTestingTips audience. I couldn’t. I wasn’t trying to test. I was trying to build a truck. The software just didn’t work . It wasn’t even close to working. If I were Ford, I would be embarrassed. I’m pricing a $40,000 item, and I can’t even use the online tool you provide me to learn about the product and it’s options!
After I finally got my truck built and submitted to the dealer for pricing, I got a couple of follow up emails confirming the submission. That was about the only thing that went well with my experience. As I reflected on if there were possible testing lessons I could draw from this experience, I came up with nothing. The interface had so many problems it didn’t even occur to me to test for possible issues. If I were a tester, I would have spent the entire day just writing up the issues I saw while doing (what I hope is) the happy path.
After building the F-150, I switched over to Toyota’s site and built a Tundra. I had a price on the truck I wanted in about five minutes. For $20,000 to $40,000 a truck, you’d think Ford would take some time to get something so simple as an online options builder correct. As for me, I still need to think of today’s tip, and I’m no closer to an idea. Perhaps I’ll go test a Google app… those always give me interesting ideas. (That’s because they don’t crash as soon as I start using them.)
While doing a search on the WSJ website, I noticed that they can see the future:
That’s an interesting superpower. I imagine it’s fairly handy for a news site. Heh…
I suspect I understand the business driver behind it. I’m sure the article goes in tomorrow’s paper and thus is marked tomorrow’s date. It just looks really odd. Especially since I’m not even close to midnight on the East Coast. Took me back a bit when I saw it.
Click on an article for the June 18th date, and you get a June 18th dateline in the article. Kinda wierd.
Last month I had the pleasure to facilitate the Workshop on Regulated Software Testing . At the workshop I had the pleasure to see something I’ve not previously seen at a workshop. During the workshop Geordie Keitt , the head of the AST eVoting SIG , interviewed Jim Nilius former Senior Director of Voting System Test Lab at SysTest Labs Incorporated. It was a full-on Barbra Walters style interview on the topic of testing voting systems. And it was awesome.
For peer workshops of that type, you typically have two or three types of presentations. The most common is an experience report. That’s where someone in attendance gets up and tells a story about a real project they worked on or a real problem they solved. You can tell it’s an experience report because they use words like "I" and "we" a lot. The idea is that through sharing of actual experiences, followed by open and honest questioning, everyone can learn more about what works and what doesn’t, and why.
Other types of presentations can include problem-solving opportunities where an attendee relates an issue they are struggling with right now and attendees try to help generate ideas for what to try next. Workshops can also include research reports (describing original research that you conducted or significantly participated in) or position papers (which might be on a topic you feel strongly about, but may not have a specific experience to share).
What made the interview so great was that Geordie controlled how we learned about the topic. Through his series of planned and ad-hoc questions, he drew out Jim’s stories and experience. It was also entertaining (since both of them have a health sense of humor). After the interview, Jim remained in front of the workshop for a session of open-season questioning where any attendee could as a question they thought Geordie had missed.
I suspect I’ll be adding the interview format to the IWST website . I encourage other workshop organizers to do the same. I’m not sure everyone would be as successful and Geordie and Jim, but it worked really well – and it was a nice change of format. I’m thinking it might also be a great way for someone who might not be comfortable enough to share an experience report to still share their experience.
I believe Geordie and Jim will have some follow up work on the topic to publish at some point. I’ll like to it when it comes out.
Someone was silly enough to let me test on Friday. I know, don’t let kids run with scissors, don’t play in traffic, and don’t let test managers actually test. I get it.
In an effort to get a full 45 minute test session together, I decided to group some similar bugfixes together. It was around five tickets, all in the same component. I noticed something that I did on accident while testing that paid off. On reflection, I should have done it on purpose, but I didn’t.
The first bugfix I tested worked. I tried a couple of scenarios around the original bug, those worked too. Happy day. I checked of that ticket and moved on to the next one. I tested the second ticket. It worked too. I tried a couple of scenarios around that ticket, those passed.
After testing the final scenario for the second ticket, it occurred to me that I was in a wonderful position to exercise the function from the first ticket. So, I clicked the button, and was rewarded with an uncaught Java exception. The scenario for my second bugfix was a perfect stress test for the first feature I tested.
The new bug was a different bug than the original one. So that fix still worked. The issue wasn’t that I missed something I should have caught while testing for the first fix. What struck me was that I was lucky enough in my random selection of order for retesting the fixes that the system was in a state for me to get some simple extra tests in for the previous feature.
Had I been thinking, I would have ordered the features that way on purpose. That’s the piece I missed. It’s easy to think that testing bugfixes is less intensive work than setting out to test something that’s previously untested. But that’s not true. There’s just as much opportunity for test design and thinking about the problem.
It was a good reminder for me.
