Mike Kelly’s blog

Have you ever seen one of these?

I bet you have. It’s the staple of large software projects everywhere. It’s the estimation spreadsheet; the silver bullet of project planning. Enter the factors and the functionality, guesstimate the size, and the total effort for the project is generated in a very nice clean column. Yea right…

Now perhaps I’m not being fair. The spreadsheet above was developed by David Tonne and I while working on an article titled “Managing the Test Project.” At the time, I was a tester working under Dave on a project Dave was managing. I have a lot of respect for Dave; he delivered two large and very difficult projects while I worked with him. We consider our article to be a success because we both learned a bit about what motivated the other and how we both plan our work

From Dave’s point of view, most test managers fail to engage actively in the estimating process. They then complain about lack of resources or condensed test windows at the end of the project. While he’s trying to deliver a product to the customer, we’re complaining about why life’s so hard and nobody listens to us. That may not be how you and I work, but many testers do work that way.

To be more specific, my problem is not with using spreadsheets for estimation. I use spreadsheets when I estimate. My problem is with how I typically see them used:

- Many times they oversimplify a complex problem
- They don’t typically account for the rapidly changing focus of a test team
- They don’t take into account the skills of the project team (“Just plug in any resource…”)
- They don’t account for what we don’t know (which is often a lot)

Formulas and auto-calculating columns gloss over the messy details of smart people struggling with difficult problems. Instead, I prefer to get the people doing the work involved and to get into the details of each task (that I’m aware of).

For an example of how I might approach a problem, let’s look at estimating testing for the Windows Calculator (we used the Calculator in Windows XP Pro for this example). I started by opening the Help file and pretending that was the list of requirements (or stories) I was given for estimation purposes.

This might actually make a fairly interesting set of tests:

- The Calculator overview could equate to some sort of usability testing.
- You have the functional tests for these items:
o To perform a simple calculation
o To perform a scientific calculation
o To perform a statistical calculation
- You have tests for the integrity of temporal data with these items:
o Transfer numbers between Standard and Scientific view
o To work with numbers stored in memory
- Here’s some bounds checking:
o Understanding Extended Precision
o Performing calculations with large numbers
- Compliance to a specification occurs with the Help Viewer

And that just gets us started! We could identify all sorts of tests based on this list alone.

Once you have your ideas listed, move them around according to size, complexity, risk, type of testing, etc.—find relationships. Draw deeply from your past experience. Even better, get your team involved and find a whiteboard. Based on some of the testing we identified above, we might change our idea list to look like the following figure.

Within each category of tests, we might imagine that work effort will look similar, because each category we identified is a type of testing. If it makes sense, we assign a qualifier—Simple, Average, Complex, Small, Medium, Large, etc.—something that can be used as a multiplier and at the same time show the relationship of the items (see the following figure).

Notice we didn’t identify a relationship for our data integrity tests. That may be because we just can’t readily see that relationship yet. That’s okay. This is the difference between using a spreadsheet to help your estimation and relying on a spreadsheet for your estimation. As long as we can justify our estimate in some other way, we don’t need to get all of our numbers using the same method. In fact, many times it makes more sense to use several methods for estimation.

From here, drawing on our experience from testing the Calculator application from Windows 3.1 to 2000, we might determine that Average functional testing for any given feature takes one person 40 hours. Better yet, if we still have the testers who tested earlier version, we can have each of them enter their own estimates (much better then me guessing how long it will take). We might also know from experience that usability testing and compliance checking both average around 20 hours. If we say that our estimate is half of the average for simple types of tests, and double for more complex tests, we can come up with the numbers in the following figure.

The next step is to estimate some of the testing that may be new or more difficult to understand. For example, we may not have had 32-digit precision in past versions, so we’re unsure what that means for our testing. Now things get more detailed.

You may want to add some tasks to better identify what this testing will entail. Will we need new tools or new methods? Do we have anyone on the team who can do this testing? Perhaps it will take a couple of hours to figure out some of this stuff, so we add a planning task. In addition, if we need tools (perhaps to see memory values while we’re testing), we have to get those installed and configured. We should give ourselves some time for that process. Therefore, we have two subtasks so far, as shown in the following figure.

This process continues until you have estimates for each step, up to and after the actual execution of the testing for this task. As you go through this process, think about other unique artifacts and efforts going on at the same time; think about test setup time, status reporting, management overhead, and so on. Depending on how the project is managed, these issues may all need to be recognized as a necessary part of the cost structure for the project.

In our estimation above, we’ve tried not to oversimplify (in fact we may have made the simple complex) and we tried to take into account the skills of the test team (have the testers actually estimate their work based on their own experience). What we haven’t done yet is allow for rapidly changing focus or accounting for what we don’t know.

Here’s how I do that: Don’t put it in the spreadsheet.

Don’t add a contingency (unless you call it contingency – I don’t mean you can’t plan for the unexpected). Don’t say “We don’t know 10% of our stuff.” Make it clear up front that you consider software development to be an intellectual effort where things change over time and that you plan on making continuous adjustments and plan on providing feedback in an ongoing basis for the duration of the project.

By not pretending to account for the things that no one can account for, your managers can better understand your real needs and will know to engage you in continuous communication as the project unfolds over time.

In my current role, I deal a lot with vendors who provide testing resources and services. Let me start by saying I think most vendors (at least the ones I interact with), provide a good product. I am generally happy with both of the large vendors I work with, and with several of the smaller vendors. For some background, most of my requests are for products and services around automation and performance testing.

However, given my role I get to participate in a number of meetings where vendors try to sell me on completely outsourcing a function (like performance testing) or on bringing in consultants to help me “develop the right implementation model and frameworks” (like with automation). I’ve pulled together some observations around how these discussions normally go, and I have some feedback for vendors who hold these conversations with people in my role.

The magical letters ROI don’t instantly sell me your product
I’m a fan of understanding return on investment for automation and performance. I think I understand how to do that for my current context, and for a couple of past contexts I’ve worked in. My current boss, whom I respect more then anyone I’ve worked for in the past, holds me accountable for understanding our return on investment for automation and performance. I try to have regular discussions around return on investment with the application teams we support.

Having said that, every time I hear “let’s take a look at the ROI,” or “it will increase your ROI,” or “all we need to do is use the ROI calculator” some little part of me shrivels up and dies. It drives me insane. I refuse to believe that for products as complex and involved as automation and performance testing services (where you need to understand infrastructure, application architecture, business and use cases, deployment models, culture, risk tolerance, and the other aspects of the design, development, and testing taking place for a project) that you can so easily capture the ROI. If it were that easy you wouldn’t be talking to me about it.

In all of the conversations I’ve had, the term ROI comes up frequently. I can’t bring myself to challenge it, because I don’t want that to become the focus of the conversations (not really productive – regardless of how happy it would make me). What I’m looking to you for is for solutions to problems, not for you to help me sell my role or organization upwards to my management. I don’t need your help to build the business case for the role I’ve been given. If I do, I hope my manager either fires me or gets me the training I need to be effective in my role.

“Additional value add” doesn’t actually tell me anything about the value you add
A close second to ROI in terms of frequency of use is the term “additional value add.” I don’t even know what that means. Well, I know what it means from a marketing perspective – much like the term “best practice.” I get it. I’m not completely dense. But I don’t know what it means to me in the context it’s used in the discussions.

How do they even know what would be valuable to me? In one conversation I listened to a vendor for 20 minutes discuss an approach to automation that I told them I didn’t think added much value. Then they proceeded to discuss that approach and the “value added” solutions they had created to allow that work to be done faster and cheaper. If I don’t think it’s valuable, doing it faster and cheaper still doesn’t add value. There might be a divide by zero error in there somewhere.

It’s a superfluous phrase (or non-value add – if you will). I’m tempted to ask, “What services or products do you provide that don’t add value? I just want to know so I can be sure I’m not paying for any of those.” I hope all your services add value. If they don’t, don’t offer them. When you use terms like “additional value add,” in my mind you become the used car salesman of your industry.

Opening a “dialog” doesn’t solve my problems
On a final note, the last thing I want to do is “open a dialog” about my problems. If I put a specific problem in front of you, I’m interested in specific solutions. Put the person in front of me who can help me understand what we need to do, and what you can do to help. If you’re not that person, I’ll find another vendor who is. I don’t have time to dialog, I have problems.

For the most part, I’m generally skeptic that you know more about my problems then I do; so you have little credibility to begin with. Telling me you want to drag out high-level conversations rather then talk to me about specific problems is not going to get you more of my time. The best vendors I’ve seen ask, “What is your biggest problem that we are best positioned to help you solve?” Then they either deliver a solution or help you deliver a solution. If the delivery was successful, they then ask, “How can we best help you solve other problems?” That’s when the dialog begins. They now have credibility and are worth me giving them my valuable time.

Deliver something before you talk to me about what you can deliver. Delivery is hard to do. If I see you do it once, I’m more inclined to think you can do it again. Once I think you can deliver, I’ll talk with you as much as you want me to.

I’ve recently been helping plan the facilitation for a workshop. As I’ve been writing up notes around the details, I thought I might share some of the questions I’m asking the organizers.

Who will be attending?
As you think about your attendees ask yourself what they will contribute to the workshop. Not everyone needs to be an expert. Sometimes novices to a topic add energy to a room. They ask good questions. They take good notes (and don’t underestimate the value of a good note-taker). You’ll need some people who contribute content, some people who contribute questions, and some people who contribute different amounts of energy at different points in the workshop.

Where will this workshop be held?
As you think about your location, think about price (especially if you’re paying out of pocket), but also about:
• How accessible it is: If people are flying in, can they get a cab there, are directions difficult, or is there road construction they will need to deal with?
• What options do you have to arrange the room: Some rooms only have large rectangle conference tables. Those make it hard to create a u-shape, a circle, or classroom style seating.
• Can they accommodate a projection screen? Will you even need one (you won’t if they have clear and open wall space)?
• Will they have wireless internet access? And is that good or bad for the type of workshop you want to run? (Sometimes you may not want wireless; it can be a distraction.)

What equipment will you need?
I now have a small “workshop kit” I’ve built. It’s a couple of Kinko’s boxes filled with the following:
• a projector
• flipcharts and markers (technically, the flipcharts don’t fit in the boxes, but they sit right next to them in my closet)
• note paper
• colored index cards
• post-it notes
• pens
• name tags
• stickers
• power chords / extension chords
• a small clock
• chimes
• collapsible easels
• paper towels, paper plates, forks and knives
• rubber bands and paper clips
• aspirin/Advil
• extra business cards

There may be more in there, but that’s what I can remember. Think about what else you’ll need based on any special activities you plan on running.

What will your schedule look like?
Try to plan out your timeline at a high level. Will people eat breakfast, lunch, or diner while at the workshop? When will that take place? Will time be needed to setup or teardown? Will you need time to transition between activities? Are there one off events that need to be accommodated in the schedule somewhere (like a group photo)? When will people get restroom or networking breaks and for how long?

What will people eat and drink?
The question is not if there will be food and drink, but where it will come from. If your workshop is over an hour (and I’m assuming it is if you’re calling it a workshop), you at least need pitchers or bottles of water. If it’s longer, you may need coffee, soda, juice, tea, etc…. At some point, you’ll need to start thinking of snacks. Sitting and remaining engaged with your brain is hard work. It takes energy to focus and participate. Make sure people have calories to feed that energy if it’s needed. If you plan on providing breakfast or lunch, make sure you plan in advance where it’s coming from and how it will be paid for. Catering can be a difficult topic with some venues, and you don’t want to surprise your attendees with a food bill if they weren’t expecting one. If they will need cash, let them know in advance of the workshop.

What’s your housekeeping list look like?
As you plan the first few minutes of the workshop, you’re going to need to run down a list of “housekeeping” items. Things like how often breaks will be taken, where the restrooms are located, appropriate use of cell phones, where and when will food come into play, coordinating evening activities, and an overview of the format and content of the workshop. It can be helpful to script some of that out in advance so you don’t forget anything.

Those first few minutes are where you tell your audience how put together you are and whether or not you’re going to be able to take care of their needs throughout the workshop. If they feel like you’re forgetting about their basic needs, they won’t respect your timelines as much. In a poorly run workshop, you’ll see three or four people take a restroom break five minutes before one is “scheduled”. That’s because they have little confidence the break will happen on time.

What information do you need from your audience to be effective?
At many of the workshops I attend, there’s a check-in and check-out at the start and end of each day. That’s done for a couple of reasons. One reason is so everyone knows who everyone else is. Another reason is for the workshop coordinators to better understand the needs of the attendees. It’s at these checkpoints that the coordinators get feedback around expectations and likes and dislikes. At check-in, make sure you have questions that help capture what people are hoping to get out of the workshop. Write those answers down. At check-out, make sure you get feedback around what’s working and what’s not. Write that down as well. During breaks and at the end of the day, go back and review that information to make sure you’re fulfilling the needs of the attendees. If you have more then one person coordinating the workshop, do it as a group.

Gordon Merritt has started I’mAScripter; a new social networking site for those who automate and script. I’ve added the site to my blogroll and I’ve signed up as a member. I’ve also created a category on my blog called I’m A Scripter (I know, I thought of that all by myself) that will get syndicated there.

I hope to start posting some code out there and I’ll try to make a habit of spending some time in the forums (most likely places to find me will be Ruby, RFT, and Robot). If you have questions related to those tools (and my personal inbox indicates that people do) post them there.

It’s been a while since Windows has blue-screened on me. Probably a couple of years actually. The other day I got a blue screen for the first time on my Vaio. When the restart was complete, I was greeted with this priceless gem…

I love that Windows protects me from Microsoft processes…