How do you decide if a bug is trivial?
I was recently involved with testing greats Matthew Heusser, Jonathan Kohl, Jonathan Bach, Paul Carvalho, Danny Faught, and Michael Bolton in an email discussion on credibility in bug reporting. In passing, I mentioned that I like to save up my "trivial" defects and submit them in batches (for various reasons).
This sparked the question, "That's an interesting take. How do you decide if a bug is trivial?"
The following is my answer...
I was then treated to a clinic on heuristics by Michael Bolton. My statements are in normal text. Michael's comments are in bold.
--------
Well, I'm not sure how I do that. I've never thought about it.
You apply heuristics; it's just that you're not particularly conscious of them.
0) On anything other than the strongest possible conviction that it's trivial, I assume that it isn't.
I guess I might do something like this:
1) Is it like a past bug? (I'm thinking they might already know how to fix it.)
That's the "consistency with history" heuristic, which is often very powerful.
2) Can I envision a fix for this that would only take me an hour? (I'm thinking if I can fix it, anyone can.)
That's the I Am But A Simple Tester heuristic. That heuristic gets weaker and weaker as you develop your programming skills and your imagination.
3) Is this something I would care about as a user of the product? (I'mthinking if I don't care - and I typically care about everything - then this is really low risk and probably trivial in terms of priority.)
You can strengthen this by imagining at least three customers, each with different personas and agendas, who might be annoyed by this bug, each for different reasons. If you can do it, it's probably not trivial.
4) Is this something that I would be embarrassed if I didn't report it? (I'm thinking, if Michael Bolton was looking over my shoulder and saw the bug, would he ask my why I didn't log that obviously critical defect yet.)
Not only that, but while being incredibly smug about the bug I was seeing, I'd be oblivious to a different one pointed out by James; I'd forget to report a different one one that Jon provided a beautiful description for; I'd waste time writing too much about it, when I could have included the annotated screenshot that Paul provided; I'd be unable to write a quick script that would exercise the other 9999 entries in the table, as Danny could; and even if it were something really cool and instructive, I'd neglect to write about it in my blog with a compelling story about how both the developer and I learned something, as Jonathan Kohl would.
But thanks for the nod, even if you were joking. :)
5) Is this something I would by embarrassed if I did report it? (I'm thinking will the developer laugh at me for submitting such an obviously nit-picky bug.)
In this case, I include an appropriate disclaimer; "I recognize that this might seem nit-picky; here's why I think it's something more than that." (And then I try to apply Weinberg's Rule of At Least Three; if I can come up with at least three reasons why it's not nit-picking, it probably isn't.)
6) Is this something that's probably already fixed and I just need a new build to find it? (I'm thinking the developer already caught this, I just need to see the new code to confirm it. If not, I'll log it then. Note: perceived priority plays a roll here.)
I'm sure there are more, but that's what I can think of now...
Note that thinking about more of them in collaboration with other people is a great and revealing exercise.
This sparked the question, "That's an interesting take. How do you decide if a bug is trivial?"
The following is my answer...
Well, I'm not sure how I do that. I've never thought about it.
I guess I might do something like this:
- Is it like a past bug? (I'm thinking they might already know how to fix it.)
- Can I envision a fix for this that would only take me an hour? (I'm thinking if I can fix it, anyone can.)
- Is this something I would care about as a user of the product? (I'm thinking if I don't care - and I typically care about everything - then this is really low risk and probably trivial in terms of priority.)
- Is this something that I would be embarrassed if I didn't report it? (I'm thinking, if Michael Bolton was looking over my shoulder and saw the bug, would he ask my why I didn't log that obviously critical defect yet.)
- Is this something I would by embarrassed if I did report it? (I'm thinking will the developer laugh at me for submitting such an obviously nit-picky bug.)
- Is this something that's probably already fixed and I just need a new build to find it? (I'm thinking the developer already caught this, I just need to see the new code to confirm it. If not, I'll log it then. Note: perceived priority plays a roll here.)
I'm sure there are more, but that's what I can think of now...
I was then treated to a clinic on heuristics by Michael Bolton. My statements are in normal text. Michael's comments are in bold.
--------
Well, I'm not sure how I do that. I've never thought about it.
You apply heuristics; it's just that you're not particularly conscious of them.
0) On anything other than the strongest possible conviction that it's trivial, I assume that it isn't.
I guess I might do something like this:
1) Is it like a past bug? (I'm thinking they might already know how to fix it.)
That's the "consistency with history" heuristic, which is often very powerful.
2) Can I envision a fix for this that would only take me an hour? (I'm thinking if I can fix it, anyone can.)
That's the I Am But A Simple Tester heuristic. That heuristic gets weaker and weaker as you develop your programming skills and your imagination.
3) Is this something I would care about as a user of the product? (I'mthinking if I don't care - and I typically care about everything - then this is really low risk and probably trivial in terms of priority.)
You can strengthen this by imagining at least three customers, each with different personas and agendas, who might be annoyed by this bug, each for different reasons. If you can do it, it's probably not trivial.
4) Is this something that I would be embarrassed if I didn't report it? (I'm thinking, if Michael Bolton was looking over my shoulder and saw the bug, would he ask my why I didn't log that obviously critical defect yet.)
Not only that, but while being incredibly smug about the bug I was seeing, I'd be oblivious to a different one pointed out by James; I'd forget to report a different one one that Jon provided a beautiful description for; I'd waste time writing too much about it, when I could have included the annotated screenshot that Paul provided; I'd be unable to write a quick script that would exercise the other 9999 entries in the table, as Danny could; and even if it were something really cool and instructive, I'd neglect to write about it in my blog with a compelling story about how both the developer and I learned something, as Jonathan Kohl would.
But thanks for the nod, even if you were joking. :)
5) Is this something I would by embarrassed if I did report it? (I'm thinking will the developer laugh at me for submitting such an obviously nit-picky bug.)
In this case, I include an appropriate disclaimer; "I recognize that this might seem nit-picky; here's why I think it's something more than that." (And then I try to apply Weinberg's Rule of At Least Three; if I can come up with at least three reasons why it's not nit-picking, it probably isn't.)
6) Is this something that's probably already fixed and I just need a new build to find it? (I'm thinking the developer already caught this, I just need to see the new code to confirm it. If not, I'll log it then. Note: perceived priority plays a roll here.)
I'm sure there are more, but that's what I can think of now...
Note that thinking about more of them in collaboration with other people is a great and revealing exercise.