Learning to Test

What is Quality?

A few months ago I asked people to complete a short survey on quality. Thanks to all who took part.

Quality is often defined using Jerry Weinberg’s definition of “Value to some person who matters”. Although testers tend to use this definition frequently, and comfortably, I’ve found it to have some limitations. The word ‘value” is almost as fuzzy as “quality” and who exactly are those people who matter? It seems that even with a widely accepted definition it’s still hard to translate quality into a tangible concept for developers or customers.

I decided to try turn a definition of quality into a feeling that people could actually relate to, and hopefully use to better their own work. To do this I asked people to rate the quality of LinkedIn. I know many of you will be laughing at the idea that LinkedIn might represent quality but it turns out to be a very divisive topic. People who want to manage a professional network generally rate LinkedIn’s quality highly. Testing professionals usually don’t.

Although heavily leaning towards the “no”, we can see a decent number of people do think LinkedIn is a quality service.


If a service offers value then many people are able to ignore, or maybe don’t even notice shortcomings. I asked people to rate the quality of LinkedIn using just a yes, or a no option. As many respondents pointed out quality is definitely on a scale. What’s interesting to me is that the people who wanted to give a “yes, but no” response tended to say that LinkedIn offers value in managing a professional network (so it is a quality service), but that it it unintuitive to use (so it isn’t a quality service).

The frequency of “intuitiveness” as a measure of quality is of particular interest to the tester side of me. How often do we get caught up testing something against the requirements and ignore the unspecified side of the system? We spend hours hunting down weird and wonderful edge cases, maybe too often choosing to spend our time logging bugs rather than helping to improve the intuitiveness of a system.

We frequently discuss whether testers should learn to code or not. Maybe now is the time to start talking about whether testers should be learning about UX design. At Songkick we see huge benefit in including all team members in testing and quality activities. UI designers are experts at creating usable designs. UX designers understand how users interact with the system. Developers understand how code complexities can cause unexpected behaviour. Ops Engineers know how the system actually runs in production. Product Owners know exactly what they want the system to do. Including all of them in testing discussions brings a wealth of knowledge and skills that, when combined with a testing expert, can have a huge impact on quality.

Quality isn’t the (sole) responsibility of the test team. Everyone involved in developing software should have a good understanding of who the intended users are, their goals, and at least a working understanding of the methods we can use to meet expectations. Facilitate discussions to identify the “people who matter” for your particular system. Help connect team members to users. Even a single session on the customer support desk can have a huge impact on helping people see the real-life impact of a poor design choice. Even just making sure you have a solid approach to managing code maintainability will impact quality as least as much as prioritising features correctly.

Treat quality as a team responsibility. Work together to assist each other in building systems that people actually want to use.

Do testers really prevent bugs?

The question of whether testers prevent bugs or find them is a fairly regular tester conversation.

When you find a bug in the software you have clearly found a bug. It makes no difference whether you were actively testing the software, trying to use the software, or simply clicking around randomly, by performing some actions and recognising that a problem occurred you were testing.

Many testers do much more than testing the actual software product and many want to somehow define what these additional things are. Actions like reviewing requirements, planning releases, training users can all fall to testers. So at what point does testing really start?

Richard Bradshaw argues that finding a bug in the requirements is still just finding a bug. You might have found it earlier but you hadn’t prevented it. A mistake in requirements can still be considered a bug. Your knowledge of the system, the domain, or maybe just the user came together to allow you to recognise that there was a problem.

James Bach defines testing as “evaluating the product by learning about it through exploration and experimentation”. You are gathering information about some thing with the intention of using it. There must be a thing, whether a software system, a document, or maybe just an idea about which you can gather data, You must have some intention of using the information to make judgements about the system. If I read the requirements and never stop to consider if there might be gaps, conflicts, or problems with the proposed solution am I testing? Possibly, although I am very unlikely to form any reasonable impressions without trying to use the information I have. When reading through requirements you are most likely to find problems relating to how the system is intended to work if you really imagine what the system is going to do. You have applied your own knowledge to the information and found an issue. In other words you were testing.

So by considering testing to be something that can happen at any time from idea conception right through to general usage in the wild we are saying that testers find bugs rather than prevent them. The bug in the requirements document might not yet exist in the software but it hasn’t been prevented from occurring – it simply occurred in the requirements document rather than the code. The cost of fix a bug at the stage is probably cheaper than if we found it in code but there is still a cost to fixing it.

So, what is a bug anyway?
For me a bug is anything unintentionally undesirable within the system. Testers might uncover things that are undesirable about the system, these things might look like bugs but if we agreed to build the system with some shortcoming on a feature, or not to support a particular browser, these things would not be considered bugs. Suddenly discovering that the system doesn’t scale, or the mobile experience is poor would be bugs. We didn’t plan to make it this way so it is unintentionally undesirable and therefore a bug.

Can bugs be prevented?
I talked about my thoughts on preventing reoccurring bugs in my podcast with Rosie for the Dojo. It’s my experience that systems, and teams tend to re-create the same type of bugs in their work. Maybe there is a particular complexity to your system that trips developers up. Maybe there is some complexity within your domain that leads to repeated re-creation of the same type of bug. Or maybe the experiences of your developers mean they are fantastic with certain things and not so great at others. Whatever the reason I find there is great potential to actually prevent bugs from ever being created by pointing out the re-occurring issues that you see.

Songkick has a special case that we trip over time and time again. Festivals are different from concerts. Both are music events but a festival might not have a venue. It might take place over one day or several. A festival might be announced before any of the line-up are revealed. All of these things can catch us out if we just think about our most common case, the concert.

Rather than waiting to see the issue in requirements, designs, or worse, in the code, I have tried to keep it at the front of everyone’s minds. To me this is bug prevention rather than testing. My techniques involve communication and training rather than creative thinking. This role of sharing knowledge, keeping known areas of risk or complexity visible and talked about does not have to fall to testers. Anyone with some understanding of the system could perform this role. However testers hold valuable knowledge about the system as a whole. They see these bugs reappearing throughout the system (or they’re motivated enough to read through the bug tracking system). Their experience of testing allows them to build a model of the system with interesting features highlighted. Most testers re-use this model to quickly jump to the interesting parts of a system. With some imagination this same model can also be used as a tool for training others.

Richard Bradshaw makes a good case that we don’t prevent car accidents by not driving our car. A prevented bug is impossible to measure because it never existed. Rather than treat this as something that is impossible, and therefore not worth the time or energy I consider the likeliness of the bug appearing to be something I can influence. By taking care to share my knowledge with the whole team I have seen the number of bugs in our products drop.

Whether this is preventing bugs or reducing the chance of them appearing seems like a question of semantics. Call it whatever you like but recognise that it is an important service that testers are well placed to provide.

Share your model of the system. Help developers, and designers understand why you make a bee-line for particular aspects of the system. Help them think about the oft-forgotten areas of a system or domain when planning new features. When I perform this role I’m not hunting for the issue, I’m trying to reduce the likelihood of seeing it in existence. I know what the issue is. My goal now is to make sure everyone else in the development team also knows about this issue.

I believe that testers really can prevent bugs by doing things outside of testing. By taking steps to help the whole development team avoid re-creating the same issues testers can free themselves up to have more time for the really interesting testing.

Quick start guide to… Weekend Testing

I’ve recently teamed up with Neil Studd to re-launch the European chapter of Weekend Testing (WTEU). Both of us were inspired by Alessandra Moreira‘s excellent talk at Let’s Test conference in May. Weekend Testing, originally set up by Ajay Balamurugadas aims to facilitate peer-to-peer learning through monthly Skype testing sessions. I’m a little embarrassed to admit that before taking steps to resurrect WTEU I had never actually taken part in a Weekend Testing session, somehow I just always seemed to be too busy…

As a soon-to-be facilitator I stated joining testing sessions with the other Weekend Testing chapters in the vain hope that some of their experience would rub off on me. I discovered that not only is Weekend Testing great fun, It’s a fantastic place to meet like-minded testers from all over the world, and most of all, a great place to learn.

So what’s Weekend Testing all about?
Originally started by Ajay as a place for peer-to-peer test practice. There are now four chapters (India, America, Australia/New Zealand, and Europe). You are welcome to join in any session, the locations drive the session time rather than the entry criteria. I recently joined a fantastic session with the Australia/New Zealand chapter but it did start at 5am…!

How does it work?
Each session takes place over Skype (typing, rather than talking). Skype, Tweet, or email the chapter to let them know you want to join the session and be on Skype 20 minutes before the start-time to be added to the session. Each session has a theme or topic so keep an eye on http://weekendtesting.com/ for full details.

What can I expect from a session?
Expect to learn. Each session lasts for 2 hours and is roughly divided into four sections – Intros, an introduction to the session topic, some time to carry out the mission, a post-session discussion on what you learnt. Sometimes you’ll work in pairs, or small groups, sometimes you’ll be working alone and then re-grouping to share your findings. Each session is unique.

Who attends sessions?
Everyone! Each session has a mix of people from all kinds of backgrounds, most are testers but we welcome anyone who wants to learn about testing. You are not required to bring anything along other than your curiosity.

How do I know when session will take place?
Chapter’s generally try to hold their session on a regular slot but each session is announced on http://weekendtesting.com/ as well as over email and Twitter.

What if I can’t make the whole session?
That’s OK. Weekend Testing is very informal so feel free to come and go as you need to. Obviously if you can stay for the whole session you will get a lot more out of it but everyone understands that sometimes life has a way of interfering.

If you’re still not sure it’s for you then take a look through the write-ups of past sessions on http://weekendtesting.com/ or drop me an email at testingthemind@gmail.com

Why Every Tester Should Have a Blog

Testing is a journey. Each testing session forces you to ask new questions, review assumptions and hopefully learn something along the way. Like all journeys there will be problems, missed bugs or unexpected delays that force you to adapt your approach.

Sometimes it’s the difficult times that actually teach you how to become a better person. I am far more likely to recall the time I failed in the diplomacy stakes or remember every little detail about the critical bug that somehow made it to production. Digging in to the reason for these failures and finding something positive to take away can make mistakes a worthwhile pastime. Recording these lessons will help you apply them next time.

So much of testing relies on the tester having strong communication skills. Being able to write clearly and concisely is half the challenge when reporting bugs or reporting on testing. Writing a personal blog is the perfect place to practice your writing, as a useful aside it also helps you explore your own ideas about testing and record all those great, and not so great ideas that you have.

Many potential bloggers are deterred by the amount of time that they think a blog will require. Obviously the most successful blogs are updated frequently and I certainly find it easier to complete a post if I maintain a reasonably regular writing schedule. However it is your personal blog, no one will die if you don’t post for 6 months. Write when you have time and write when you have something to explore and enjoy the journey.

Do you have blog? If not, why not?

Choosing to be a Tester

Stephen Blower recently published this excellent post about proving your worth as a tester; If you haven’t read it then you should. As well as being struck by the excellent points he was making I was surprised to find that I am more unique that I realised – I actually did choose to go in to Software testing.

I was lucky and discovered that I enjoyed programming during my A-Levels. I went off to university to study Software Engineering. We did lots of programming but testing was never mentioned. Luckily I’m not such a great programmer so without realising it I was doing lots of debugging just to get my assignments to work.

One of the best aspects of my degree was that it was a Sandwich Course that required me to spend a year working in the IT industry. The European job market has a much better grasp of under-grads spending time in industry so I ended up  working in Germany. Unfortunately it was within an IT department that outsourced all their development work, they really didn’t really know what to do with a budding developer so I got dumped with the test contractor. Amazing! I had no idea that you could be paid for nit-picking someone else’s software.

After several months of testing I returned to university and used my final year to find out as much as I could about testing. As well as reading books and convincing Mercury to give me free access to their tools (Thanks, Mercury!) I decided to carry out a questionnaire on testing (ok so there were extra marks if you did this but still). By another very lucky chance one of the people I emailed about this survey ran a Testing Consultancy. As well as answering my survey he offered me the chance to do some work experience once I had completed my degree. When I contacted him 6 months later to take up this offer he lined me up for an interview and then offered me a permanent job as a graduate test consultant. I have never looked back.

As well as proving to Stephen that some people actually do choose to become testers (albeit with a lot of fortuitous circumstances) I thought it would be useful to highlight just how important that offer for some work experience turned out to be. If I had been rubbish the company could have easily fobbed me off with a week or two of safe project work. As it was they took a risk and launched my career.

The greatest challenge is identifying people who desperately want to break into testing. Hopefully Twitter and blogs make it easier for the interested tester to actually find out and follow testing but if you spot one then do whatever you can to help them. If you know of universities that teach aspects of testing contact them, maybe you could give a talk on testing or spread the word about tester meet up in the area.

Lets spread the word and open a few doors to the testing industry.