Mike Kelly’s blog

In April we held the first IWST workshop for 2008. The topic of the all day workshop was ‘SOA Testing.’ We were able to get through five experience reports (with open season questioning) and we looked at some specific examples in iTKO LISA. Thanks to Mobuis Labs for hosting us and providing breakfast. Attendees of the workshop were:

• Ken Ahrens
• Michael Goempel
• Ingrid Grant
• Michael Kelly
• John McConda
• Matt Osgatharp
• Eddie Robinson
• Jeff White
• Jeffrey Woodard
• Christina Zaza

The first ER came from Ken Ahrens and in it, Ken related an experience where he participated in an SOA evaluation project for a government agency. Ken gave a great overview of the technologies involved, and was kind enough to set the stage for some of the beginners in the room by defining all the acronyms (SOAP, JMS, MQ, ESB, etc…). He also talked a bit about some of the different models for SOA (pub/sub, sync/async, etc…). In the project, they compared different SOA technologies and products to determine which was the best fit for the client. For example, they tested different ESBs and ESB configurations to determine which would perform best in the client’s context.

During the presentation, Ken spent some time talking about the current state of SOA and some of the common challenges he’s seen testing in SOA environments. I particularly found his discussion of some of the common themes and “Three C’s” interesting. Dave Christiansen and I gave a webinar on some of the challenges of testing SOA a few months ago and if I could go back, I would frame some of the challenges using the “Three C’s” instead of the way we did it. It’s a useful model for talking about how SOA testing is different from some more traditional manual testing contexts.

After Ken’s talk, Tina Zaza presented her experience doing component level testing for a financial services application. Tina and her team did manual testing of several services during her first project as a tester. She used Excel to manage test coverage and traceability. Tina struggled with some of the short-term thinking of the contract staff she worked with; which lead her to rework and lost time. She found that inter-team communication was initially a big challenge (getting the BAs, Devs, and Testers to all talk). They have since tried to solve that challenge by getting the developers to review the test case XML before the initial test cycle, asking “Is this the right XML?” They also were challenged with managing the volume of XML, encountering challenges with naming, tracking versions, showing relationships, and backup/recovery.

Tina’s talk got praise from almost everyone during checkout. It resonated with me because many of the challenges Tina faced on the project are universal challenges and aren’t specific to SOA. It highlighted what common testing problems SOA fixes, creates, and doesn’t change at all. Tina is also very candid, which made it easy to ask questions.

After Tina’s talk, Eddie Robinson presented an experience report on testing for a big-three automaker centered on software for supplier management. Eddie presented a tool be built in Excel to convert a largely manual process to an automated execution and test management framework. In his project, there was a nightly FTP from the client. A VBS program pulled those files and another program persisted those changes into a staging database. Eddie’s spreadsheet then executed a large number of queries (each query representing a test case) and checked those actual results against expected results. A summary tab in the spreadsheet rolled up the results and presented a summary of execution to date. It also had some columns to help track issues logged in UAT as well as functional testing.

After Eddie’s experience report I talked about a project where Jason Horn and I did what we affectionately call “bulk testing” with some simple Ruby scripts. The concept is relatively straightforward. If you are creating a series of web services that need to support a known data set (say, a database full of existing production data) and you want to test a significant portion of that dataset (but not necessarily under load), then you’re bulk testing.

We created a process that would:

1. take production data
2. scrub it to remove any protected information
3. generate request XMLs based for each record
4. submit those requests to a service
5. check the response for exceptions
6. persist the response
7. check the persistence status for exceptions
8. log any issues
9. summarize results

It’s a fairly simple process - only a couple hundred lines of code all said and done. We created tens of thousands of test cases (if you view each request as a test). We found a lot of defects with this testing and also ended up providing the development team with a prioritized list (based on number of records failed per defect) for them to work from. It was one of the best examples of one-time automation I’ve ever done: low cost to create the automation, high value return, and mothball the scripting assets when complete.

After my talk, Ken gave a second experience report on service-oriented virtualization. Ken’s customer was unable to run performance tests on a regular basis before they implemented virtualization. That is, testing was an “event” where tens of teams had to get together to make a load test happen, and the high cost meant that this only happened a few times a year. After they implemented the virtualization performance tests were run daily. Ken figures the client got a 100x multiple in the number of tests they could run.

Virtualization at the message level gave them greater ability to experiment with issues such as “what if this key service slows down significantly”? Or try different data scenarios, such as what if the lookup returns 900 records instead of 10 records? These are things they couldn’t do in their typical load test environment. They were also able to mock new services quickly, even before the code had been implemented. This let performance testing start earlier in the SDLC. It also let people figure out misunderstandings between teams earlier in the projects they were working on because testing was going on with the mock assumptions in place.

Given the about of performance testing I’ve been doing on services, I thought that one was the coolest ER. It gave me a lot to think about.

The next workshop is Friday, September 26 on the topic of “Testers who write code.” If you have an ER or would just like to attend, drop me a line.

Here is the 2008 lineup for IWST. This year, workshops are held on a Friday and run all day with regular breaks for networking and refreshments. There is no fee to attend any workshop. Email me at Mike@MichaelDKelly.com if you would like to attend.

SOA Testing
Date: Friday, April 25
Time: 8:00 AM to 5:00 PM
Description: In this workshop we will focus on testing in a Service Oriented Architecture. We will be looking for experience reports covering test planning and estimating; experience reports that talk about functional, performance, security, regression, and interoperability testing; and experience reports and demos that show off tools designed specifically for the SOA environment.

Testers who write code
Date: Friday, September 26
Time: 8:00 AM to 5:00 PM
Description: In this workshop we will focus on those aspects of testing where we write code. There will be a strong focus on performance, automation, and security testing. A lot of discussion around tools, languages, frameworks, and design patterns. We will be looking for experience reports from people willing to share project stories, examples, and we would be willing to entertain proposals for hands-on coding activities during the workshop.

About IWST:
The Indianapolis Workshops on Software Testing (IWST) are an ongoing series of no-cost workshops for experienced software testers and related professionals. IWST strives to build skills in software testing and allows people who are interested in topics in software testing to network with their peers. The emphasis is on mutual learning, sharing hands-on experiences and solving practical problems. In these meetings, experienced working practitioners discuss pertinent state-of-the-art topics. IWST has been organized by a small team of local practitioners in the field of software testing.

Let the downloads begin! Bandwidth problems finally solved (I hope). I’ve launched a new website for the Indianapolis Workshops on Software Testing. You can find it here: www.IWST2008.com

If you would like to be involved in 2008, drop either myself or one of the other organizers an email.

This weekend we held the final IWST workshop for 2007. The topic of the five hour workshop was ‘Scripting for Testers and Test Automation with Open Source Tools.’ What a great workshop! We packed in seven experience reports (with open season questioning) and we talked a bit about IWST for 2008. Attendees for the workshop included:

• Andrew Andrada
• Charlie Audritsh
• Michael Goempel
• Michael Kelly
• John McConda
• Chad Molenda
• Cathy Nguyen
• Charles Penn
• Mike Slattery
• Gary Warren
• Chris Wingate
• Jeff Woodard

We opened the workshop with an experience report from Charles Penn on his experience learning Watir. Charles covered some of the resources he used to learn the tool and he shared the details of his experience trying to get popups to work. Charles currently uses Watir for some of his regression testing. He has a suite that he runs daily over lunch. The code Charles used for his popups can be found here: Pop Up Clicker and a Test Logging Example

Following Charles, Jeff Woodard shared an experience where he used AutoIT to automate the import and export of test cases with IBM Rational ManaulTest. Jeff worked at a client that required the use of Rational ManaulTest. He felt the tool restricted his productivity. So he created some simple scripts that allowed him to work in his tool of choice for test case design and documentation, then all he had to do was run the scripts to “check in” the tests. With about 2 days of scripting, he estimates he shaved over a week from his workload doing administrative testing tasks. He managed over 700 manual test cases using AutoIT. For those who might be interested in his solution, the AutoIT code can be found here: export and import

Next, Cathy Nguyen shared how she currently uses Selenium for her test automation. The application Cathy tests is very Ajax intensive, and Selenium was one of the tools that she could get to easily work with the application. She uses the FireFox recorder, then takes those tests and converts them to C# for integration and management in Visual Studio Team System. With a bit of help she was able to create an ‘MS Unit Test’ converter, allowing her to integrate her Selenium tests with the developer tests already in use. Results can be tracked in the team SharePoint site and the scripts can be stored in source control. One of her next steps is to get the scripts running as part of the continuous integration process. The code that Cathy used to convert the tests to MS unit tests can be found here: tbd

After Cathy, Chris Wingate and Chad Molenda shared a joint experience of using Watir for smoke testing multiple projects in multiple environments, with several builds a day. The testing load on the smoke test scripts required them to migrate the smoke tests from IBM Rational Robot to Watir (faster execution, parallel execution, and easier code maintenance). They experienced similar popup issues, but also had some threading issues and dealing with multiple concurrent browsers on the same machine. They are working on creating a new method that allows you to manage dialogs based on the parent window instead of the user32.dll. I believe they plan on sharing that code back with the Watir community when they are complete; so look for that code in a future Watir release.

John McConda then presented an experience of using RadView WebLOAD on a client project. WebLOAD has an open source model that allows for some features in the open source version with a commercial version for clients who need more features/users. John said that he was able to get it to work with some Ajax and did a quick demo against the Mobius Labs website.

Following John, Charlie Audritsh presented some scripting tips for performance testers. First he showed a trick he uses to script usage model paths using case statement with a random number generator. This simple technique allowed him to reduce the number of scripts required to implement his workload model, and also allowed him to implement fractional percentages for user groups/activities within the model. After that, Charlie showed a store procedure he wrote to help with performance test logging. He uses it for debugging and for detailed logging when he needs it (turns it off when he doesn’t). An example of both techniques Charlie shared can be found here: dec_iwst_audritsh.wri

Finally, Mike Slattery shared a number of tools he has tried for his testing, including Jacobie, webunitproj, jwebunit, htmlunit, httpunit, and Selenium. He found he had some common problems he had to work through, regardless of the tool. Those included popups, difficulty in having the same tests do double duty as functional and performance tests, difficulty in testing stylistic aspects of the application, and general test code fragility. Mike did say something about the ability to build your own recorders in Selenium, something I wasn’t aware of. I know I need to check that out.

On the topic of future IWST workshops, I’m sure we will do something for 2008. I don’t think they will be every month, but we’ll see what we can do. I plan on starting some discussions on the IWST mailing list and we’ll see what the community has energy for. I will also get all this info (code and such - when I get it) up on the IWST site at some point. I think I need to upgrade and redesign a bit in the coming months.

Thanks to all who participated in the 2007 Indianapolis Workshops on Software Testing. The full list of attendees for 2007 include:

• Andrew Andrada
• Charlie Audritsh
• Dave Christiansen
• Howard Clark
• Lisa Etherington
• Michael Goempel
• Jason Horn
• Karen Johnson
• Michael Kelly
• Steve Lannon
• Kelvin Lim
• Baher Malek
• John McConda
• Scott McDevitt
• Rick Moffatt
• Chad Molenda
• John Montano
• Cathy Nguyen
• Charles Penn
• Kenn Petty
• Vishal Pujary
• Mike Slattery
• Michael Vance
• David Warren
• Gary Warren
• Chris Wingate
• Jeff Woodard

I’ve recently been helping plan the facilitation for a workshop. As I’ve been writing up notes around the details, I thought I might share some of the questions I’m asking the organizers.

Who will be attending?
As you think about your attendees ask yourself what they will contribute to the workshop. Not everyone needs to be an expert. Sometimes novices to a topic add energy to a room. They ask good questions. They take good notes (and don’t underestimate the value of a good note-taker). You’ll need some people who contribute content, some people who contribute questions, and some people who contribute different amounts of energy at different points in the workshop.

Where will this workshop be held?
As you think about your location, think about price (especially if you’re paying out of pocket), but also about:
• How accessible it is: If people are flying in, can they get a cab there, are directions difficult, or is there road construction they will need to deal with?
• What options do you have to arrange the room: Some rooms only have large rectangle conference tables. Those make it hard to create a u-shape, a circle, or classroom style seating.
• Can they accommodate a projection screen? Will you even need one (you won’t if they have clear and open wall space)?
• Will they have wireless internet access? And is that good or bad for the type of workshop you want to run? (Sometimes you may not want wireless; it can be a distraction.)

What equipment will you need?
I now have a small “workshop kit” I’ve built. It’s a couple of Kinko’s boxes filled with the following:
• a projector
• flipcharts and markers (technically, the flipcharts don’t fit in the boxes, but they sit right next to them in my closet)
• note paper
• colored index cards
• post-it notes
• pens
• name tags
• stickers
• power chords / extension chords
• a small clock
• chimes
• collapsible easels
• paper towels, paper plates, forks and knives
• rubber bands and paper clips
• aspirin/Advil
• extra business cards

There may be more in there, but that’s what I can remember. Think about what else you’ll need based on any special activities you plan on running.

What will your schedule look like?
Try to plan out your timeline at a high level. Will people eat breakfast, lunch, or diner while at the workshop? When will that take place? Will time be needed to setup or teardown? Will you need time to transition between activities? Are there one off events that need to be accommodated in the schedule somewhere (like a group photo)? When will people get restroom or networking breaks and for how long?

What will people eat and drink?
The question is not if there will be food and drink, but where it will come from. If your workshop is over an hour (and I’m assuming it is if you’re calling it a workshop), you at least need pitchers or bottles of water. If it’s longer, you may need coffee, soda, juice, tea, etc…. At some point, you’ll need to start thinking of snacks. Sitting and remaining engaged with your brain is hard work. It takes energy to focus and participate. Make sure people have calories to feed that energy if it’s needed. If you plan on providing breakfast or lunch, make sure you plan in advance where it’s coming from and how it will be paid for. Catering can be a difficult topic with some venues, and you don’t want to surprise your attendees with a food bill if they weren’t expecting one. If they will need cash, let them know in advance of the workshop.

What’s your housekeeping list look like?
As you plan the first few minutes of the workshop, you’re going to need to run down a list of “housekeeping” items. Things like how often breaks will be taken, where the restrooms are located, appropriate use of cell phones, where and when will food come into play, coordinating evening activities, and an overview of the format and content of the workshop. It can be helpful to script some of that out in advance so you don’t forget anything.

Those first few minutes are where you tell your audience how put together you are and whether or not you’re going to be able to take care of their needs throughout the workshop. If they feel like you’re forgetting about their basic needs, they won’t respect your timelines as much. In a poorly run workshop, you’ll see three or four people take a restroom break five minutes before one is “scheduled”. That’s because they have little confidence the break will happen on time.

What information do you need from your audience to be effective?
At many of the workshops I attend, there’s a check-in and check-out at the start and end of each day. That’s done for a couple of reasons. One reason is so everyone knows who everyone else is. Another reason is for the workshop coordinators to better understand the needs of the attendees. It’s at these checkpoints that the coordinators get feedback around expectations and likes and dislikes. At check-in, make sure you have questions that help capture what people are hoping to get out of the workshop. Write those answers down. At check-out, make sure you get feedback around what’s working and what’s not. Write that down as well. During breaks and at the end of the day, go back and review that information to make sure you’re fulfilling the needs of the attendees. If you have more then one person coordinating the workshop, do it as a group.