Why that production bug might be the best thing that ever happened to your testing

Don’t you just hate bugs that get found in production? You’ve been testing for days (maybe even weeks) and yet as soon as it hits production you see that bug. Then you have the hassle of regression testing and patch releases.

Maybe it wasn’t even you who found the bug. A mocking tweet or an angry email, sometimes a very confused user who can’t understand why you didn’t test that this thing actually works!?!

We all know that you’ll never achieve perfection and yet we always try. Each release is relentlessly tested until the deadline is upon us, each time you hope that just maybe you’ve got all the big bugs. Maybe you even did, but maybe a bug found in production will make you a better tester.

So how do bugs get into production?

You didn’t think to test that area of the system: Why not? Maybe you didn’t realise that it had changed. Maybe you didn’t even know that this part of the system existed. Why? Did you miss something in the requirements? Did the developer make changes that you didn’t know about? Maybe you trusted automated tests to check this for you. Whatever the reason, it didn’t work so think about ways you can avoid this happening again. Maybe checking commit logs or improving regression suites would help or maybe you could improve your communication with developers.

You didn’t know how to test this area of the system: Hopefully this is an extension of not knowing this area of the system exists. Ignorance is not a good reason to miss bugs. If this bug was reported by a user then you need to re-visit your test planning to understand why you missed it. If it was discovered by someone technical then this is a great chance to learn. Find out how they discovered the bug. What techniques or tools are they using? If they work within your company then some pair testing might uncover more bugs whilst allowing you to pick up new testing knowledge.

You weren’t able to test this part of the system: Maybe you’re lucky but I have never worked anywhere which had an exact replica of the production available for testing. Any difference between the two opens up the chance of bugs making it through to production. Different sets of data, different release processes and different setups will undermine your testing. Use production bugs as evidence for why you need to improve your test environment. If you really can’t create a realistic test environment then consider how you could safely test in production.

You didn’t know you could test this part of the system: Related to the above. If you’re working on a test environment with limitations then make sure you, and everyone else who is testing knows exactly what should and shouldn’t be expected to work. Nothing is more frustrating that realising that thing that wasn’t working is broken rather than just not working because the test environment doesn’t support it. In my experience this is most likely to happen around 3rd party integrations.

As testers we aim to find all bugs before the code is released to production. Deep down we all know this is impossible. Use Production bugs as a way to improve your testing and you should see a significant reduction in big production bugs (but no guarantees you won’t still have bugs in production – sorry!).

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s