A Positive Way to Think About Bugs

Bugs are the things of nightmares. There are the ones that appear unannounced in production systems, at a critical time, and are spotted by the most vocal hater of your company. There are the intermittent ones that you spend hours trying to track down, and the small, irritating ones that you know will never be prioritised. There are the ones you’ve never seen but suspect are there, lurking. Waiting.

Over the years I’ve had an additional bug nightmare. One where someone expects me to spend a significant amount of my time capturing bug report metrics. They want to know how long bugs take to fix, how long we took to find the bug, or how good a developer is based on the number of bug reports we have.

When you take all this negative bug energy and throw in Jira. Or Mantis. Or BugZilla… it’s easy to understand why we all hate bugs.

But there is a positive side to bugs.

It’s too common to hear of bugs surprising teams and nothing changing. Some bugs are unexpected, and it doesn’t always make sense to try to stop a similar things happening again, but those are the exceptions. It is much more common for similar bugs to appear in different parts of the systems, these are strong indicators that things are not quite right. Observing and sharing these patterns can be a great trigger to motivate teams to change.

Every wondered how you can convince people to give you time to work on tech debt, or process improvement, or make progress on that cool infrastructure project? Well bugs can be very persuasive…

When a release showstopper bug, or production issue is found use it to its full potential. Understand the root cause, try to establish exactly why the bug appeared when it did. Was the issue poor communication, lacking test environments, or something else? Don’t make this a blame game about individuals but use it to find pieces of your process or technology that could be improved. In my experience bugs are almost never a result of poor code, they are much more likely to indicate code complexity, lack of context, poor test data, or simple lack of experience.

Once you have your information tell your story. Keep telling it. You’re not trying to force people to change things, you want to simply highlight the issues that are occurring as a result of things being the way they are. Once you find your interested audience work with them to make improvements.

Step by step you’ll find things improve.

What’s the cost of shipping bugs?

A tiny question to reveal huge insight.

At Songkick we use Continuous Deployment for many of our releases. This approach requires a risk-based approach for every change. Are we tweaking the copy, or re-designing the sign up system? Are we changing a core system, or an add-on feature? The answers will determine how much risk we’re willing to take. And subsequently decide whether this feature can be automatically deployed by our Jenkins pipeline or not. The most useful question we ask ourselves before writing any code is “What’s the cost of shipping bugs?”.

If the answer is “low”, perhaps because this is an experiment, easily fixed, invisible to users etc then we know we can move faster. Developers can be more autonomous. Maybe they don’t need a code review. Maybe the testers don’t need to spend so long testing things before we release. Maybe we don’t need to update our test suites.

If however the answer is “high”, perhaps because we’ve broken something like this in the past, or it’s going to be hard to fix, or highly damaging, or we’re all about to take a week off to visit New York. Then we know that we need to be more cautious. Testers need to be more involved. We need to consider releasing this with feature flippers, or using a dead canary release. We’ll make sure the release takes place at a time when there are people available to monitor the release, and get involved if needed.

It’s a tiny question that takes just a minute to ask but this tiny question can shape our entire development and release approach.

How do you estimate the cost of shipping bugs? 

Do testers really prevent bugs?

The question of whether testers prevent bugs or find them is a fairly regular tester conversation.

When you find a bug in the software you have clearly found a bug. It makes no difference whether you were actively testing the software, trying to use the software, or simply clicking around randomly, by performing some actions and recognising that a problem occurred you were testing.

Many testers do much more than testing the actual software product and many want to somehow define what these additional things are. Actions like reviewing requirements, planning releases, training users can all fall to testers. So at what point does testing really start?

Richard Bradshaw argues that finding a bug in the requirements is still just finding a bug. You might have found it earlier but you hadn’t prevented it. A mistake in requirements can still be considered a bug. Your knowledge of the system, the domain, or maybe just the user came together to allow you to recognise that there was a problem.

James Bach defines testing as “evaluating the product by learning about it through exploration and experimentation”. You are gathering information about some thing with the intention of using it. There must be a thing, whether a software system, a document, or maybe just an idea about which you can gather data, You must have some intention of using the information to make judgements about the system. If I read the requirements and never stop to consider if there might be gaps, conflicts, or problems with the proposed solution am I testing? Possibly, although I am very unlikely to form any reasonable impressions without trying to use the information I have. When reading through requirements you are most likely to find problems relating to how the system is intended to work if you really imagine what the system is going to do. You have applied your own knowledge to the information and found an issue. In other words you were testing.

So by considering testing to be something that can happen at any time from idea conception right through to general usage in the wild we are saying that testers find bugs rather than prevent them. The bug in the requirements document might not yet exist in the software but it hasn’t been prevented from occurring – it simply occurred in the requirements document rather than the code. The cost of fix a bug at the stage is probably cheaper than if we found it in code but there is still a cost to fixing it.

So, what is a bug anyway?
For me a bug is anything unintentionally undesirable within the system. Testers might uncover things that are undesirable about the system, these things might look like bugs but if we agreed to build the system with some shortcoming on a feature, or not to support a particular browser, these things would not be considered bugs. Suddenly discovering that the system doesn’t scale, or the mobile experience is poor would be bugs. We didn’t plan to make it this way so it is unintentionally undesirable and therefore a bug.

Can bugs be prevented?
I talked about my thoughts on preventing reoccurring bugs in my podcast with Rosie for the Dojo. It’s my experience that systems, and teams tend to re-create the same type of bugs in their work. Maybe there is a particular complexity to your system that trips developers up. Maybe there is some complexity within your domain that leads to repeated re-creation of the same type of bug. Or maybe the experiences of your developers mean they are fantastic with certain things and not so great at others. Whatever the reason I find there is great potential to actually prevent bugs from ever being created by pointing out the re-occurring issues that you see.

Songkick has a special case that we trip over time and time again. Festivals are different from concerts. Both are music events but a festival might not have a venue. It might take place over one day or several. A festival might be announced before any of the line-up are revealed. All of these things can catch us out if we just think about our most common case, the concert.

Rather than waiting to see the issue in requirements, designs, or worse, in the code, I have tried to keep it at the front of everyone’s minds. To me this is bug prevention rather than testing. My techniques involve communication and training rather than creative thinking. This role of sharing knowledge, keeping known areas of risk or complexity visible and talked about does not have to fall to testers. Anyone with some understanding of the system could perform this role. However testers hold valuable knowledge about the system as a whole. They see these bugs reappearing throughout the system (or they’re motivated enough to read through the bug tracking system). Their experience of testing allows them to build a model of the system with interesting features highlighted. Most testers re-use this model to quickly jump to the interesting parts of a system. With some imagination this same model can also be used as a tool for training others.

Richard Bradshaw makes a good case that we don’t prevent car accidents by not driving our car. A prevented bug is impossible to measure because it never existed. Rather than treat this as something that is impossible, and therefore not worth the time or energy I consider the likeliness of the bug appearing to be something I can influence. By taking care to share my knowledge with the whole team I have seen the number of bugs in our products drop.

Whether this is preventing bugs or reducing the chance of them appearing seems like a question of semantics. Call it whatever you like but recognise that it is an important service that testers are well placed to provide.

Share your model of the system. Help developers, and designers understand why you make a bee-line for particular aspects of the system. Help them think about the oft-forgotten areas of a system or domain when planning new features. When I perform this role I’m not hunting for the issue, I’m trying to reduce the likelihood of seeing it in existence. I know what the issue is. My goal now is to make sure everyone else in the development team also knows about this issue.

I believe that testers really can prevent bugs by doing things outside of testing. By taking steps to help the whole development team avoid re-creating the same issues testers can free themselves up to have more time for the really interesting testing.