Do testers really prevent bugs?

The question of whether testers prevent bugs or find them is a fairly regular tester conversation.

When you find a bug in the software you have clearly found a bug. It makes no difference whether you were actively testing the software, trying to use the software, or simply clicking around randomly, by performing some actions and recognising that a problem occurred you were testing.

Many testers do much more than testing the actual software product and many want to somehow define what these additional things are. Actions like reviewing requirements, planning releases, training users can all fall to testers. So at what point does testing really start?

Richard Bradshaw argues that finding a bug in the requirements is still just finding a bug. You might have found it earlier but you hadn’t prevented it. A mistake in requirements can still be considered a bug. Your knowledge of the system, the domain, or maybe just the user came together to allow you to recognise that there was a problem.

James Bach defines testing as “evaluating the product by learning about it through exploration and experimentation”. You are gathering information about some thing with the intention of using it. There must be a thing, whether a software system, a document, or maybe just an idea about which you can gather data, You must have some intention of using the information to make judgements about the system. If I read the requirements and never stop to consider if there might be gaps, conflicts, or problems with the proposed solution am I testing? Possibly, although I am very unlikely to form any reasonable impressions without trying to use the information I have. When reading through requirements you are most likely to find problems relating to how the system is intended to work if you really imagine what the system is going to do. You have applied your own knowledge to the information and found an issue. In other words you were testing.

So by considering testing to be something that can happen at any time from idea conception right through to general usage in the wild we are saying that testers find bugs rather than prevent them. The bug in the requirements document might not yet exist in the software but it hasn’t been prevented from occurring – it simply occurred in the requirements document rather than the code. The cost of fix a bug at the stage is probably cheaper than if we found it in code but there is still a cost to fixing it.

So, what is a bug anyway?
For me a bug is anything unintentionally undesirable within the system. Testers might uncover things that are undesirable about the system, these things might look like bugs but if we agreed to build the system with some shortcoming on a feature, or not to support a particular browser, these things would not be considered bugs. Suddenly discovering that the system doesn’t scale, or the mobile experience is poor would be bugs. We didn’t plan to make it this way so it is unintentionally undesirable and therefore a bug.

Can bugs be prevented?
I talked about my thoughts on preventing reoccurring bugs in my podcast with Rosie for the Dojo. It’s my experience that systems, and teams tend to re-create the same type of bugs in their work. Maybe there is a particular complexity to your system that trips developers up. Maybe there is some complexity within your domain that leads to repeated re-creation of the same type of bug. Or maybe the experiences of your developers mean they are fantastic with certain things and not so great at others. Whatever the reason I find there is great potential to actually prevent bugs from ever being created by pointing out the re-occurring issues that you see.

Songkick has a special case that we trip over time and time again. Festivals are different from concerts. Both are music events but a festival might not have a venue. It might take place over one day or several. A festival might be announced before any of the line-up are revealed. All of these things can catch us out if we just think about our most common case, the concert.

Rather than waiting to see the issue in requirements, designs, or worse, in the code, I have tried to keep it at the front of everyone’s minds. To me this is bug prevention rather than testing. My techniques involve communication and training rather than creative thinking. This role of sharing knowledge, keeping known areas of risk or complexity visible and talked about does not have to fall to testers. Anyone with some understanding of the system could perform this role. However testers hold valuable knowledge about the system as a whole. They see these bugs reappearing throughout the system (or they’re motivated enough to read through the bug tracking system). Their experience of testing allows them to build a model of the system with interesting features highlighted. Most testers re-use this model to quickly jump to the interesting parts of a system. With some imagination this same model can also be used as a tool for training others.

Richard Bradshaw makes a good case that we don’t prevent car accidents by not driving our car. A prevented bug is impossible to measure because it never existed. Rather than treat this as something that is impossible, and therefore not worth the time or energy I consider the likeliness of the bug appearing to be something I can influence. By taking care to share my knowledge with the whole team I have seen the number of bugs in our products drop.

Whether this is preventing bugs or reducing the chance of them appearing seems like a question of semantics. Call it whatever you like but recognise that it is an important service that testers are well placed to provide.

Share your model of the system. Help developers, and designers understand why you make a bee-line for particular aspects of the system. Help them think about the oft-forgotten areas of a system or domain when planning new features. When I perform this role I’m not hunting for the issue, I’m trying to reduce the likelihood of seeing it in existence. I know what the issue is. My goal now is to make sure everyone else in the development team also knows about this issue.

I believe that testers really can prevent bugs by doing things outside of testing. By taking steps to help the whole development team avoid re-creating the same issues testers can free themselves up to have more time for the really interesting testing.


Linda Rising – Organisational Change

I don’t need to convince people. They should just see how good this idea is” – Linda Rising

I’m sure anyone who has ever worked in a team has experienced this thought pattern at some point in their life.

Linda Rising’s fantastic keynote, “Myths And Patterns Of Organizational Change” at this year’s Pipeline Conference has some brilliant ideas for dealing with change. If you haven’t already seen it, I highly recommend making the time to do so.

Linda Rising – Myths And Patterns Of Organizational Change – PIPELINE Conference 2015 from Software Engineering Practice on Vimeo.

Should that link open a new window?

If you test software that allows users to navigate using hyperlinks you need to think about new windows. It sounds simple enough, links send the user off to a different place, but context should determine whether the link opens in a new window, or tab, or re-uses the existing one.

During an E-commerce checkout you expect the “Buy now” button, or link, to open in the window or tab you’re using. The same thing with “Sign Up” flows, they are part of the main journey and so should be re-using the main window. It would be pretty strange to end up with multiple tabs each displaying part of a “Sign Up” journey.

What about “FAQ” or “Terms of Service” pages? Well it depends. If you click one of these links in the main site footer then it makes sense for the window to be re-used. If however I click a “Terms of Service” link from a checkout or payment page then I really hope it opens in a new window or tab to avoid interrupting my purchase.

Resuming journeys
Have you ever tried to purchase something from a website only to be sent off into a “Sign Up” flow? If you were lucky you completed the “Sign Up” and were gracefully returned to the page you were originally on. Sadly this isn’t always the case. As a user it is hugely frustrating when websites make you repeat actions just because they wanted you to do something else first. Make this easy. Return users to the page they were originally on.

Indicating behaviour
Links to external websites or services should open new windows as the default behaviour. In addition, external links, and mailtos, should be labeled with the standard icon to indicate that they will take you away from the current site. Have a look around a service such as Spotify to see these icons in action.

Interaction between tabs
More complex tab thinking leads to considering the interaction between tabs. I often search for items on Amazon and open several tabs to compare different items. After I’ve compared the items I might add one or two to my basket. I expect my session to exist across all the tabs. That means that I am logged in on all the tabs, and my basket is gathering items from all the tabs. Checking out on any one of the tabs should show me a basket containing all of my selected items.

Following these guidelines will help make your website intuitive to use. However the question of context must still be answered. What happens on your website when you open a new tab and log in? Are both tabs now showing the correct state? Is that correct for your service? How about if you log out? Or purchase something? Think about the standard behaviour but always do what is right in your context.

Thoughts from TestBash 2015

I’ve just returned from the fourth TestBash conference. Each year it grows, getting better and better every time. A packed out conference (10 speakers!) and many social events make it easy to catch up with old friends, and make new ones too.

I kicked things off with the social event on Thursday evening. Sadly I hadn’t been able to attend the workshops during the day but there were plenty of people singing their praises over drinks. I had a whirlwind of catching up, beers, and a little bit of 99secondtalk prep with my fellow Weekend Tester facilitators Neil Studd and Dan Billing.

After just a few hours sleep, something of a tradition at testing conferences, it was run time! Once again there was a fantastic pre-conference run along the seafront. I love starting the day with a run, this year was an especially beautiful morning and we had a good turnout despite the early start. One of my favourite parts of the run is having 10 minutes or so to just chat with fellow testers completely uninterrupted. We run and we chat. Then we all dash off to try and get ready in time for Leancoffeebacon.

The conference itself had a fantastic line-up. Michael Bolton gave a predictably solid talk on language. A great reminder to actually say what you mean. Ian McCowatt was up next with a great talk on Bug Detection. He gave me the push I needed to pick up Harry Collin’s “Tacit and Explicit” book. I was also reminded of the importance of re-reading books. It’s so easy to get caught up in the endless of book list that sometimes I forget how much you can get from a book on the second, or even third reading.

There were great talks from Martin Hynie, Matt Heusser, and Stephen Janaway. There is still plenty of digesting of the ideas but it was fascinating to hear Martin’s experiences of the job title ‘Tester” actually limiting tester’s ability to get involved in projects. Stephen Janaway had some really interesting ideas in his talk “Why I Lost My Job As a Test Manager and What I Learnt As a Result”, the coaching menu was particularly interesting to me. I can see something like that being very useful on my team.

Vern Richards, and Richard Bradshaw both gave thought-provoking talks. Richard’s story of moving into automation only to find that he had “automated too many things” was really good. So many teams have the goal to automate everything. it was interesting to hear what happens if you actually succeed in doing so.

Sally Goble and Jon Hare-Winton demonstrated that it is possible to do a good double act. Maaret’s talk ” Quality doesn’t belong with the tester!” was a really resonating experience report. Being the only tester on a team is challenging and Maaret shared lots of ways that she tackled it. I really liked that she had talked to her team of developers about how they wanted to define testing. So often it seems testers want to name everything and tell developers how it should be done but developers do testing too, it’s just different.

Karen Johnson wrapped up the day (well apart form the 99second talks!) with a really engaging talk on asking questions. There were so many great ideas in this talk, and a number of interesting book references too.

All in all I have watched so many brilliant talks from engaging, interesting people. I have a list of new books to read and lots of thinking to do. I’ve come away from TestBash having seen so many friends. I’ve got a list of names of my new friends in my pocket and I feel inspired to get stuck in to some testing!

Thanks, Rosie and all of the TestBash speakers and organisers. It was an absolute blast. See you next year!

What works for me might not work for you. Reverse. Repeat.

Earlier this week I posted my thoughts about DevOps and Continuous Delivery. TheSunshine tester responded on Twitter with a strong defense for Agile

I love that methodologies are able to create such passionate responses. Even amongst the truly committed I doubt I would find any passion for the Waterfall method. Yet the Agile, DevOps, and CD communities seem genuinely excited about working together in this way.

Software Development has greatly improved in the last few years but I still think we have a long way to go before it is considered reliable. Last year reported that the majority of ERP projects are over time and over budget. If Agile is truly the silver bullet then all teams would be adopting it, and finding success with it. Unfortunately that doesn’t appear to be the case.

No matter how successful Agile, or any practice might prove to be we should still be seeking new and better ways of working. Humans are unique and our involement makes every project equally unique. The team who has success with agile is unlikely to have the same team make-up, or collective experience as one who doesn’t transition, or who fails to succeed with agile. Our interpretation of what a process involves, and our previous experiences all serve to influence what the end process will actually look like.

We need to have discussions about DevOps because it has arisen from an apparent divide between developers and Ops people. Whether this divide is caused by agile, or a failing of the implementation is irrelevant, the fact is that many teams are finding things as they stand are not good enough.

At Songkick we’ve followed Lean practices for many years. We have small cross-functional teams, daily stand-up meetings and weekly retrospectives. We measure things and try to remain flexible and responsive. Despite this we were having trouble releasing working software. Implementing Continuous Deployment was an enhancement to our agile practices. It went beyond giving us a framework for working together and gave us the steps we needed to make frequent releases too.

One of the criticisms I often hear is ‘it wouldn’t work here’. Hopefully these reservations come from understanding that there are no best practices, but whatever your reason for resisting copy-book change you are right.

What has worked for me, in my team, on my project will never work in exactly this way with your team.

Simply put, best practices do not exist.

Continuous Delivery is the current trendy approach and that is a concern. You should be trying to fix a problem not copy a textbook. If you cannot identify the benefit of any practice, agile ones included, then you shouldn’t be using it. One of the most valuable skills we learned from adopting Continuous Delivery was the ability to review our process and identify exactly what was working and what wasn’t. We continue to review things, making changes as our needs, or the team, changes.

Talk to other teams, listen to their experiences, read widely but use their processes as your guide. The most useful thing you can learn from other teams is how they implemented change. What were the triggers or warning signs? How did they know it was working? How did they choose this change?

These are things that you can apply to your own situation. These are the things that are likely to actually help you deliver good software.

DevOps with Testers

Last week I spoke at QCon London about Songkick’s experience of moving to Continuous Delivery. You can download the slides, or watch the longer version of this talk on the LondonCD Meetup’s Vimeo channel.

My talk was part of the DevOps and Continuous Delivery track hosted by Eoin Woods. QCon videos of all the talks should be available soon.

One particularly interesting talk was by Steve Thair on ‘DevOps and the need for speed‘. Previously I thought DevOps was about developers and Ops people collaborating to enable faster software delivery. Steve’s definition went much further and ended up being closer to how I think about Continuous Delivery.

Steve defined DevOps as a model that “encompasses a product-centric view across the entire product life-cycle (from inception to retirement) and recognises the value in close collaboration, experimentation and rapid feedback.” He emphasised that the business and testers should be engaging in the collaboration just as much as developers and ops people.

An interesting distinction between Steve’s DevOps definition and Dave Farley’s Continuous Delivery definition focused on the definition of ‘Done’. Continuous Delivery is defined as being about “making businesses more experimental through the early and continuous release of valuable software”. It appears to be complete once the release is out in production. DevOps adds an important final step of evaluating how much value the release is actually adding once it’s in the hands of users.

I love Steve’s definition of DevOps but unfortunately think this term is sadly restricting. I doubt business people or testers will feel included in something called ‘DevOps’. Even more confusingly I believe there are both devs and ops people who do believe ‘DevOps’ only covers devs and ops.

Even re-defining Continuous Delivery to extend beyond the release doesn’t really help the situation. We’re trying to create a process that keeps the software in a state where by it can be released whenever the business wants it to be. Obviously there are some questions around why the business would suddenly decide to release with little notice and whether we should support that or not. Leaving this aside I think Continuous Delivery is a positive way of working to remove unexpected delays to releases.Any process that encourages collaboration and stress-free releases should be given strong consideration in my book.

A name is always just a name but in the case of improving processes and making sure all roles are covered I think it becomes pretty important. We need to hear everyone’s voice as we attempt to fix software development.The concepts being discussed are important and will affect us all. I personally feel that neither the ‘DevOps’ or ‘Continuous Delivery’ terms are quite right for describing an attempt to engage with all stakeholders and release timely, and functioning software to end-users. Regardless of this I believe everyone involved in software design, development and releases should be involved in deciding the best way to fix the software industry.

What do you think? Are you a Continuous Delivery or DevOps fan?

Did you scroll to the bottom of the page?

It’s easy to get caught up in the one thing you’re testing right now. This narrow view of the world (or system) makes it easy to miss obvious problems.

One technique I use is to always scroll to the bottom of the page. It doesn’t matter if my test calls for me to click a link in the page’s top nav, first I scroll down, then I click my link.

It takes just seconds to scroll to the bottom of the page. By taking just a little time to broaden your view of the world you might find you see some surprising things.

A Sole Tester

Many years ago I led a fantastic team of testers. Together we were able to plan and execute testing in a rapidly moving environment. Our skills and experiences were varied and complimentary. Most usefully we had skilled people who could cover our absences.

More recently I’ve been grappling with the unique challenges that come from working as the sole tester in a development team. The additional pressure to plan and execute testing efficiently. The unmovable fact that your skills are the only tester skills available. Whilst the learning opportunities are huge you’re likely to find yourself relying heavily on the (fantastic) testing community to support you through the testing challenges.

You can read more about the challenges and opportunities of being a sole tester in my Testing Circus article.

A look back at 2014

2014 was the year that I resolved to do less stuff. I have a terrible time turning things down. Empty calendars seem to fill me with fear, and I simply can’t help getting involved in things I believe in. Unsurprisingly I ended up busier than ever.

I attended, or spoke at some fantastic conferences – TestBash, Let’s Test, Agile Cambridge, BBC Develop, and EuroSTAR. At each event I met, or re-acquainted with fantastic testers from around the world. My conference highlight of year was being involved in organising Pipeline. It took months of hard work but it was hugely satisfying to see the conference unfold on the day.

A few difficult months of writing and editing finally saw my contribution included in Build Quality In. If you’re interested in Continuous Delivery this book contains some fantastic accounts of how different teams have approached it. Steve and Matthew did a great job of selecting contributors and editing the submissions.

After many years of deferring I finally completed the Black Box Software Testing Foundation certificate, and attended James Bach’s RST course in Brighton. One day of Security Testing training with Bill Matthews completed my training for the year. My greatest takeaway from all of these courses was a realisation that I do have the time, and the desire, to learn. Hopefully I’ll be building on this in 2015.

Unexpectedly I found myself co-facilitating Weekend Testing Europe. Meeting, and having the opportunity to learn with the Weekend Testing attendees has been a pleasure. Alongside Neil and Dan I’m excited about the sessions we have (almost) planned for 2015.

Podcasts seemed to be a theme of my year. Steve and Dan were kind enough to have me on Testing in the Pub twice and I wrapped up my year with a great chat with Rosie for The Minitry of Testing Dojo.

Throughout 2014 I have thoroughly enjoyed all the interactions with so many testers – thank you to all of you.

Resolutions for 2015? Well I think I might have another go at being just a little less busy…

Quick start guide to… Weekend Testing

I’ve recently teamed up with Neil Studd to re-launch the European chapter of Weekend Testing (WTEU). Both of us were inspired by Alessandra Moreira‘s excellent talk at Let’s Test conference in May. Weekend Testing, originally set up by Ajay Balamurugadas aims to facilitate peer-to-peer learning through monthly Skype testing sessions. I’m a little embarrassed to admit that before taking steps to resurrect WTEU I had never actually taken part in a Weekend Testing session, somehow I just always seemed to be too busy…

As a soon-to-be facilitator I stated joining testing sessions with the other Weekend Testing chapters in the vain hope that some of their experience would rub off on me. I discovered that not only is Weekend Testing great fun, It’s a fantastic place to meet like-minded testers from all over the world, and most of all, a great place to learn.

So what’s Weekend Testing all about?
Originally started by Ajay as a place for peer-to-peer test practice. There are now four chapters (India, America, Australia/New Zealand, and Europe). You are welcome to join in any session, the locations drive the session time rather than the entry criteria. I recently joined a fantastic session with the Australia/New Zealand chapter but it did start at 5am…!

How does it work?
Each session takes place over Skype (typing, rather than talking). Skype, Tweet, or email the chapter to let them know you want to join the session and be on Skype 20 minutes before the start-time to be added to the session. Each session has a theme or topic so keep an eye on for full details.

What can I expect from a session?
Expect to learn. Each session lasts for 2 hours and is roughly divided into four sections – Intros, an introduction to the session topic, some time to carry out the mission, a post-session discussion on what you learnt. Sometimes you’ll work in pairs, or small groups, sometimes you’ll be working alone and then re-grouping to share your findings. Each session is unique.

Who attends sessions?
Everyone! Each session has a mix of people from all kinds of backgrounds, most are testers but we welcome anyone who wants to learn about testing. You are not required to bring anything along other than your curiosity.

How do I know when session will take place?
Chapter’s generally try to hold their session on a regular slot but each session is announced on as well as over email and Twitter.

What if I can’t make the whole session?
That’s OK. Weekend Testing is very informal so feel free to come and go as you need to. Obviously if you can stay for the whole session you will get a lot more out of it but everyone understands that sometimes life has a way of interfering.

If you’re still not sure it’s for you then take a look through the write-ups of past sessions on or drop me an email at