Many of us are comfortable testing from a use case. It's familiar. Most of us are trained and experienced in thinking about a problem starting from an actor, who takes action, and ends up somewhere. It's the classic "Given... When... Then..." way of thinking about a problem. You start from point zero, then advance to the destination. Lately I've found it helpful to instead start from the destination, and then work backwards from there to think of all the different ways I could end up there.

For example, let's say I'm on a search results page. Here are all the ways I might have gotten there:

• I go to the search field on the homepage, enter search criteria, I land on the search results page


• I enter a URL someone sent me which takes me to the search results page


• I click on a link with a hard-coded search request that takes me to a search results page


• I click the back button on my browser which takes me to a search results page


• etc...

This technique is sometimes helpful when the piece of functionality I'm testing has preconceived notions about what defaults it can assume based on how the programmer thinks I got there. They were likely thinking of the one or two use cases they had available - not the 8 or 9 other ways you can get there.

For example, most features on search results page (like sorting, filtering, etc...) assume you've entered search criteria. In some cases, that's just not true. Sometimes you can get to a results page without entering anything. If your search criteria isn't pre-populated in the search field for you, in those cases you can get some really odd behaviors. This can become even more interesting when you're dealing with features that perform calculations, or where the issues resulting from the false assumptions aren't immediately visible.

The next time you run across a feature that looks like it might be making some assumptions based on state, work backwards from that point to think of all the other ways you might be able to get there. Do all of those paths lead to the necessary conditions for the defaults to be populated? If not, what possible problems might that cause?

Frankly, I picked up the idea of this technique after reviewing source code validating some input. I remember, programmers even called it "cheating". But since then, I successfully tried it on different applications without having an access to the source code - so, obviously, there are some patterns.

The idea

User data traverse through a few validation layers. What is valid at one layer, might be invalid for another. Data also get transformed and fed into other modules which may or may not have comprehensive validation of internal input.

The idiom

 "A fly in the ointment" means a small defect that spoils something valuable or is a source of annoyance while being even in a tiny proportion.

The heuristic

Creating combinations of valid and invalid data sometimes allows passing through, or triggers a program to transform data into something causing problems internally.

A few examples in today's tip.

Some "restricted" characters

• Backslash (\). This character is used to "escape" other system characters, and to create system commands as well.


• Less than (<), Greater than (>), Ampersand (&). These characters have a primary meaning as tags in mark-up languages.


• Space character.


• Asterisk (*) is used as a wildcard in queries and regular expressions.

 

Some combinations to try

• Valid inputs wrapped up by tag characters. Examples: "<123>", "</123>"


• Restricted characters "escaped" with a backslash. Examples: "\&", "\\"


• System commands created with a backslash. Examples: "\d", "\t"


• Asterisk alone or in combination with a valid input. Examples: "*", "Toronto*"


• Space characters before, after, or around delimiters. Examples: " 123", "1. 23"

 

Armed with the examples provided above I went on the hunt and picked a couple of publicly open web-sites belonging to large organizations...

You can see results below.

 

This morning I saw mint data for the first time. I love playing with software like this - this is cool stuff. Within a couple of searches (less then five) I found a couple of different issues I'd call bugs. All of them were found using a technique I call testing for "ubiquitous" features.

A ubiquitous feature is one that exists "everywhere." You don't question if it's actually a feature, you just assume it is. An example is using quotes in a search field. After using search engines for the last ten years, you just have some basic assumptions around how search works.

So this morning on mint data, I tested with quotes in my search. Once I performed that search, the interface completely froze up. I had to reload the site to get and new search to work. It was only later that I noticed that this feature is even illustrated in the mint data example search text.

Other search features caused similar issues. When you think about the application you're testing, it's sometimes useful to understand how users will map their existing expectations onto your application (which they likely don't understand yet) and how that will drive what they will assume it will do.

Today's tip comes from Brett Leonard.

In the Beginning... Mass Hysteria and Confusion...
When faced with a new testing challenge, finding a place to start designing your tests can be difficult. It is natural to suffer from stimulus overload and feel overwhelmed. Using a minimalist approach to data setup can accelerate the development and execution of complex scenarios and quickly provide a means to report on the status of the application. In this article I will describe a testing challenge I recently faced and how I used a simple end-to-end test scenario, or “Golden Scenario”, to rapidly test and track the development of the feature under test.

A Challenge with No Easy Answer
Recently, I needed to design tests for a complex feature that had been in development for a few months but had not been tested from an end users point of view. I needed to quickly understand the feature and design a method to test it. The first thing I did was schedule a meeting with our developers, architect, and product manager. Our meeting was productive but did not give me enough information to begin drafting my test scenarios - much of our conversation revolved around the data architecture challenges that were still ongoing.

I wanted to find a way to cut through the complexity and define initial scenarios. Once the initial scenarios were defined and communicated, I would have a mechanism to report on the application status by tracking the progress of our test execution.

Studying Economics Finally Pays Off
The next day, while contemplating screenshots of the feature, my schooling in economics came to mind. Economists use simple models to understand complex systems. For example, in my international economics class, we constructed a model using a country that produced one good - a ”one good country” and examined the supply and demand curve in this simple economy. Once we understood the “one good country” economy we added complexity by introducing another “one good country”. Later, we increased the number of goods and created much more complicated models. This led me to the idea of using a similar approach to develop my first end to end scenario that I deemed, the “Scenario of One”.

The Golden Scenario Technique is Born
The Golden Scenario Technique (ours was named “The Scenario of One”) uses a minimalist approach to setting up test data. To implement it you simply define a scenario that requires the absolute minimum amount of setup necessary to interact with your targeted feature. In my case, I’m testing a project management application, which starts with “projects” as a container. A “project” can have multiple “packages” and “packages” can have multiple “participants”.

Our First Scenario Defined and Communicated
My first scenario was to create a project with one package and one participant. The act of setting up the data for this scenario exercises a great deal of code that would provide us with valuable information about the state of the application. By defining and adding this scenario to our testing dashboard we finally had a clear direction and method to communicate test results.

Excited with our path forward, I quickly wrote up a mission in One Note© (my favorite test documentation tool), and presented it to my team. That morning in the daily development status meeting (or daily Scrum meeting), I described my idea and declared that our mission for the day would be to complete the setup of the “Scenario of One” or document the roadblocks that prevented it.

Mass Hysteria and Confusion Evaporates
With the mutual understanding of this limited scope I broke down the complexity to a manageable, understandable, and reportable form which I can now use to guide the development and execution of additional test scenarios. I had blazed a path forward and felt as though I had finally taken control of our testing effort.

To document our results in real-time, I drew a chart on our dashboard with the various steps of the process in a straight line. As we executed our tests, I would use color-coding to indicate how far our testing had progressed. At first, there was a lot of red but eventually (after two painful weeks) we were able to report that we made it through all variations of the scenario.

Jump Start Your Testing with Golden Scenarios
The next time you feel overwhelmed with a testing situation, try asking yourself “what is the minimum amount of set up that I can do to start testing this application?”, make it your first scenario, and begin testing. It will jump start your test execution and give you valuable information about the state of your application. You will be well on your way towards providing the valuable information testing is supposed to provide the stakeholders of your application.

Image courtesy: theinspirationroom.com

 

Sometime ago I wrote about a quick way to shutdown a hanged application.
But, in fact, shutting down a working application could help you exposing certain types of resource management defects. This is how ends become means.

Sample tests, suggested below, were tried on different Windows Desktop applications, sometimes helping to reveal serious issues.

Local Tests

• Start an application. Begin working process then shutdown the application in the middle of it. Start the application. See if it reports than the previous session wasn't properly finished (and does it suggest something about it, like "start in safe mode").


• Start an application. Begin working process then shutdown the application in the middle of it. Start the application. See if it can access any previously opened files, and how is it handling and reporting the issues.


• Change settings in an application, apply changes. Shutdown the application. If the application saves settings only at exit point, all changes will be lost.


• Change settings in an application, apply changes and shutdown the application simultaneously. Start the application. See if it can access configuration files, does it report that they might be corrupted, does it suggest repairing them? (you can also try corrupting the files manually).


• If an application allows running 2 copies: open 2, then shutdown one of them. Observe behavior of the remaining one.

Server Tests

• Start a client. Shutdown a client. Try to login as the same user: a) immediately b) after a timeout given.


• Start a client. Shutdown a client. Repeat this process (you may want to automate that) while monitoring how Server side of an application allocates and releases (does it?) RAM and hard-drive resources.


• Start a client. Shutdown a client. Repeat this process (you may want to automate that) for a while - you may see errors like "max number of connections exceeded", etc.

Let us know results of your exploration.