Old Masters and Young Geniuses
A while ago, I read David Galenson's "Old Masters and Young Geniuses: The Two Life Cycles of Artistic Creativity." This book was appealing to me because I use to be an art student and I'm always looking for books on creativity, but as I read it I started to see patterns in it that lead see the Old Masters and Young Geniuses metaphor as something useful for software testers. If you often find books that have nothing to do with software testing, useful in helping you form your opinions about software testing, then I think you'll enjoy this book immensely.

Disclaimer, I only read the book once, so some of my summary statements might be a little off, but for the most part, I think I do it justice. I might have miss-read something here or there. Don't be too upset if I got something wrong.


In the book, Galenson draws the distinction between the experimental and conceptual artist. The experimental artist is the artist who works in a very controlled fashion; almost scripted. They make variations to their process and practice over time, but always in a calculated way. Always trying to perfect some small aspect of their craft. They might do the same work, or series of works, over and over again. Think of Monet as an archetype of the experimental artist. Galenson tries to illustrate that the experimental artist's best work is done late in their career. They become an "Old Master."


The conceptual artist is the artist who works more from intuition then from process. They break from custom. They do something never done before of take some established philosophy and turn it on it's head. The might do a particular work or even style only once, and move on to the next phase of their career. Think of Picasso as an archetype of the conceptual artist. Galenson tries to illustrate that the conceptual artist's best work is done early in their career. They start as a "Young Genius."


Interesting dynamics emerge in the differences between the way the two groups work:


"The distinction between experimental and conceptual artists can be sharpened by considering their procedures in making paintings. For this purpose, we can divide the process into three stages: planning – all the artist does before beginning a particular painting; working – all the artist does while in the process of putting paint on the canvas; and stopping – the decision to cease working."


Galenson explains that for the experimental, all three stages are controlled. Extensive upfront planning takes place, work is tightly controlled (often delegated), and the decision to stop is predicated on completion of a goal. For the conceptual, there is little to no planning; that stage is very short. The "signing" artist does most work, and the process for the work may vary day to day. The decision to stop is heuristic.


As I read the book, I found myself inserting scripted for experimental and exploratory for conceptual. Think of a continuum like the scripted vs. exploratory continuum (you can see an example on slide 15 of Jon Bach's Breaking Down Exploratory Testing Skill talk) many of us use to illustrate the continuum for testing. In a similar continuum for artists, you might have those that tend to favor small controlled change, with learning over time, and those that favor large rapid change, with breakthrough results. It's a stretch, and it doesn't fit perfectly throughout the book, but it was useful for me. This book changed the way I view the dynamic between the two polarities.


"Recognizing the differences between the experimental and conceptual approaches provides the basis for systematic predictions concerning the relationship between age and artistic innovation. The long periods of trial and error often required for important experimental innovations means that they will tend to occur late in an artists career. Because conceptual innovations are made more quickly, it might be thought that they should be equally likely to occur at any age. Yet the achievement of radical conceptual innovations depends on the ability to perceive and appreciate the value of extreme deviations from existing conventions and traditional methods, and this ability will tend to decline with experience, as habits of thought become more firmly established. The most important conceptual innovations should therefore tend to occur early in an artist's career."


With this prism, if I view an early book on software testing, or an exemplar from the factory school, I might choose to see someone who is trying to perfect a method of their craft. With small controlled changes, they might make that method a little bit better each time. If I view an article on exploratory testing (because there aren't many books dedicated to the topic), or an exemplar from the context drive school, I might choose to see someone who is trying to revolutionize the field. Someone with the "ability to perceive and appreciate the value of extreme deviations from existing conventions and traditional methods."


This is important, because for me it creates a place where I can appreciate the innovations one might want to make within a school that isn't the one I choose. I sometime struggle with that; this helps. Don't get me wrong, I think there are exploratory testers who are old masters (James Bach?) and scripted testers who are young geniuses (Kent Beck?) within their particular school. Like I said, it's a model...


After I read that last large quote in the book ("Recognizing the differences between..."), a couple of questions occurred to me. I wrote them down in my moleskin and wanted to explore them here.


First, I've heard Cem Kaner talk about our field's problem of not learning from history. To paraphrase what I think the problem is, software testers (computer scientists in general?) don't invest much time in learning the history of their craft. Does this continuum relate in some way to that history problem?


I think it's possible that, of those that are "aware" – I suspect some testers don't want to be young geniuses or old masters, most would rather be conceptual, and thus don't think they need to understand history to make a meaningful contribution. Thus, like many artists, they may make (and probably often do) an "original" contribution that was original 20 years ago. In addition, if most old masters really are old (in terms of how long they have been practicing their craft), and if a solid understanding includes understanding past history, and many testers leave testing for career changes, project management, BA roles, programming, or some other development role, can we even develop old masters?


Second, if I look at someone whom I think might fit the young genius mold, Jon Bach, what are the implications for how I can learn from the behavior I see? (I choose Jon as my example because of the work he did early in his career with session based and open book, two ideas that I think are conceptual innovations in our field. Jon, I hope you don't mind.)


If I look at examples of young genius in testing (or old masters), just as I could in art, what could I learn personally to influence how I become conceptual (or experimental). What are the implications for me (or for anyone who wants to innovate – either drastically or a little bit over time)?


Finally, in the book, Galenson provides some models for measuring the success of a work of art. I won't go into them here, but imagine if we had a way to measure the success of a testing idea. It wouldn't necessarily tell you it if was good, just like you can't look at the popularity of a painting and say that makes it good, but it tells you it was potentially influential is some way. How would, or could we, measure the influence of testing ideas?


Most of those questions are rhetorical. I'm not really expecting anyone to answer. The book just got me thinking. It has some great stuff on schools and movements, analytic techniques, developing intuition, community, and a ton of stuff on the exploratory process. Recommended read.

SOA Testing

In April we held the first IWST workshop for 2008. The topic of the all day workshop was ‘SOA Testing.’ We were able to get through five experience reports (with open season questioning) and we looked at some specific examples in iTKO LISA . Thanks to Mobuis Labs for hosting us and providing breakfast. Attendees of the workshop were:




  • Ken Ahrens

  • Michael Goempel

  • Ingrid Grant

  • Michael Kelly

  • John McConda

  • Matt Osgatharp

  • Eddie Robinson

  • Jeff White

  • Jeffrey Woodard

  • Christina Zaza



The first ER came from Ken Ahrens and in it, Ken related an experience where he participated in an SOA evaluation project for a government agency. Ken gave a great overview of the technologies involved, and was kind enough to set the stage for some of the beginners in the room by defining all the acronyms (SOAP, JMS, MQ, ESB, etc...). He also talked a bit about some of the different models for SOA (pub/sub, sync/async, etc...). In the project, they compared different SOA technologies and products to determine which was the best fit for the client. For example, they tested different ESBs and ESB configurations to determine which would perform best in the client's context.

During the presentation, Ken spent some time talking about the current state of SOA and some of the common challenges he's seen testing in SOA environments. I particularly found his discussion of some of the common themes and "Three C's" interesting. Dave Christiansen and I gave a webinar on some of the challenges of testing SOA a few months ago and if I could go back, I would frame some of the challenges using the "Three C's" instead of the way we did it. It's a useful model for talking about how SOA testing is different from some more traditional manual testing contexts.

After Ken's talk, Tina Zaza presented her experience doing component level testing for a financial services application. Tina and her team did manual testing of several services during her first project as a tester. She used Excel to manage test coverage and traceability. Tina struggled with some of the short-term thinking of the contract staff she worked with; which lead her to rework and lost time. She found that inter-team communication was initially a big challenge (getting the BAs, Devs, and Testers to all talk). They have since tried to solve that challenge by getting the developers to review the test case XML before the initial test cycle, asking "Is this the right XML?" They also were challenged with managing the volume of XML, encountering challenges with naming, tracking versions, showing relationships, and backup/recovery.

Tina's talk got praise from almost everyone during checkout. It resonated with me because many of the challenges Tina faced on the project are universal challenges and aren't specific to SOA. It highlighted what common testing problems SOA fixes, creates, and doesn't change at all. Tina is also very candid, which made it easy to ask questions.

After Tina's talk, Eddie Robinson presented an experience report on testing for a big-three automaker centered on software for supplier management. Eddie presented a tool be built in Excel to convert a largely manual process to an automated execution and test management framework. In his project, there was a nightly FTP from the client. A VBS program pulled those files and another program persisted those changes into a staging database. Eddie's spreadsheet then executed a large number of queries (each query representing a test case) and checked those actual results against expected results. A summary tab in the spreadsheet rolled up the results and presented a summary of execution to date. It also had some columns to help track issues logged in UAT as well as functional testing.

After Eddie's experience report I talked about a project where Jason Horn and I did what we affectionately call "bulk testing" with some simple Ruby scripts. The concept is relatively straightforward. If you are creating a series of web services that need to support a known data set (say, a database full of existing production data) and you want to test a significant portion of that dataset (but not necessarily under load), then you're bulk testing.


We created a process that would:


  1. take production data

  2. scrub it to remove any protected information

  3. generate request XMLs based for each record

  4. submit those requests to a service

  5. check the response for exceptions

  6. persist the response

  7. check the persistence status for exceptions

  8. log any issues

  9. summarize results



It's a fairly simple process - only a couple hundred lines of code all said and done. We created tens of thousands of test cases (if you view each request as a test). We found a lot of defects with this testing and also ended up providing the development team with a prioritized list (based on number of records failed per defect) for them to work from. It was one of the best examples of one-time automation I've ever done: low cost to create the automation, high value return, and mothball the scripting assets when complete.

After my talk, Ken gave a second experience report on service-oriented virtualization. Ken's customer was unable to run performance tests on a regular basis before they implemented virtualization. That is, testing was an "event" where tens of teams had to get together to make a load test happen, and the high cost meant that this only happened a few times a year. After they implemented the virtualization performance tests were run daily. Ken figures the client got a 100x multiple in the number of tests they could run.

Virtualization at the message level gave them greater ability to experiment with issues such as "what if this key service slows down significantly"? Or try different data scenarios, such as what if the lookup returns 900 records instead of 10 records? These are things they couldn't do in their typical load test environment.

They were also able to mock new services quickly, even before the code had been implemented. This let performance testing start earlier in the SDLC. It also let people figure out misunderstandings between teams earlier in the projects they were working on because testing was going on with the mock assumptions in place. Given the amount of performance testing I've been doing on services, I thought that one was the coolest ER. It gave me a lot to think about.

The next workshop is Friday, September 26 on the topic of "Testers who write code." If you have an ER or would just like to attend, drop me a line .