Should developers own acceptance tests?

A couple of weeks ago I watched a talk from Dave Farley that said developers should own acceptance tests. It’s a great talk, you should watch it if you haven’t already. Afterwards I realised that I was going to have to write this blog post to explain why I thought the talk was brilliant, but misleading.

First, let’s define acceptance tests. Dave explains them as “trying to assert that the code does what the users want the code to do”. For me they are the tests that we perform (with, or without tools), to decide whether to accept a feature/system or not.

Dave’s talk addresses an important topic. Namely that testers shouldn’t be the owners of these tests. I totally agree with this statement. In all the projects I’ve worked on, I have never seen a test suite solely owned by testers being a useful thing. Of course the testers usually design brilliant tests, and these tests often uncover serious issues before any code gets released but they are still a terrible idea for two reasons.

Firstly, we have an entire suite of tests that exists and runs without any developer involvement. Unless you work in a team that really doesn’t think about testing in any way before code get handed over to testers then this is going to cause test duplication. If your developers really don’t think about testing then you probably have some serious silo issues that need addressing. Go and sort those out before you read the rest of this post. But seriously, developers think about testing all the time. Allowing acceptance testing to be separated from development is going to guarantee duplication in tests. It’s also likely to guarantee some gaps in testing as the approaches of developers and testers are not joined up.

Secondly, testers are brilliant at designing tests. We should expect them to be able to turn up some issues, many of them fundamental and serious. Why wouldn’t we want to do that much earlier in the process? Acceptance tests are usually run against a release candidate. Often a release candidate contains one, or even several weeks of work. Waiting that long to run these tests and then turning up a problem is going to make for some expensive re-work.

In Dave’s talk he argued that developers should be the owners of acceptance tests. To me this sounds like a bad idea. Acceptance tests should be a measure of how well the feature meets the requirements. Developers are probably least qualified, or at least biased enough, to be unable to make this decision. That’s not to say that testers should be the owner either. We still want to avoid creating any kind of “wall” for code to be thrown over.

So who should own the tests? Maybe the team would be a better owner. If acceptance tests are truly going to be a measure of how well the feature meets requirements then it seems to me that a business/product owner needs to decide what gets tested. So we ask the business owner to define the journeys that need testing. Testers know a lot about designing good tests so the Testers should be helping to turn those journeys in to good test scenarios with suitable test data. Developers know how to write robust code so the Developers should be writing and maintaining the tests.

The end result is a well designed, and implemented test suite that actually tests something the product owners wants to have tested.

This collaborate test design brings together the strengths, and input, of the entire team. We give testers the opportunity to uncover issues, but the collaborative nature of the test design means it’s likely to happen far earlier that it would if testers own the tests. Maybe even at the test design stage instead of a week later during release testing. Developers can write the tests and avoid many of the “tester written test” issues. The test scripts get treated as any other code, written and maintained by developers. Finally the real strength of this approach is being able to involve the business owner in the discussions. Hearing about the things they’re worried about right at the beginning can have a massive impact on the design and implementation of the feature.

In fairness I think Dave was actually arguing for exactly this approach. Developers should absolutely be at the heart of acceptance tests but I don’t think we should use the word ‘own’. Teams own things, not individuals. The right group of people collaborate to achieve the best result.

What’s the cost of shipping bugs?

A tiny question to reveal huge insight.

At Songkick we use Continuous Deployment for many of our releases. This approach requires a risk-based approach for every change. Are we tweaking the copy, or re-designing the sign up system? Are we changing a core system, or an add-on feature? The answers will determine how much risk we’re willing to take. And subsequently decide whether this feature can be automatically deployed by our Jenkins pipeline or not. The most useful question we ask ourselves before writing any code is “What’s the cost of shipping bugs?”.

If the answer is “low”, perhaps because this is an experiment, easily fixed, invisible to users etc then we know we can move faster. Developers can be more autonomous. Maybe they don’t need a code review. Maybe the testers don’t need to spend so long testing things before we release. Maybe we don’t need to update our test suites.

If however the answer is “high”, perhaps because we’ve broken something like this in the past, or it’s going to be hard to fix, or highly damaging, or we’re all about to take a week off to visit New York. Then we know that we need to be more cautious. Testers need to be more involved. We need to consider releasing this with feature flippers, or using a dead canary release. We’ll make sure the release takes place at a time when there are people available to monitor the release, and get involved if needed.

It’s a tiny question that takes just a minute to ask but this tiny question can shape our entire development and release approach.

How do you estimate the cost of shipping bugs? 

Get outside your comfort zone

The testing community is awesome. There are so many friendly faces. So many people reading things, discussing things, watching things, and developing their ideas. As the communities grow stronger the people you spend most of your time with are likely to be similar to yourself. Maybe you all belong to a similar school of thought. Maybe you’ve worked on projects together before, or attended training courses or conferences together.

It’s great.

Or is it?

The problem is these people are likely to be very similar to you. They share your ideals, and your ideas. You start to think that you’re in the majority whereas is many cases you’re not.

When was the last time you read something you disagreed with? Or attended a conference that wasn’t solely about your craft? Now I’m not saying that you have to go out there and engage everyone in debate. You’re not looking to convert these people, or even to change your own opinions. Broadening your view might simply give you something to measure your ideas against.

If you’re an agile advocate do you really understand why not everyone is into it? Do you know why testers are often excluded from projects? Have you asked a developer why they haven’t attended a test conference? Have you asked yourself why you haven’t attended a design conference?

We all work in development and yet we all hold these independent, and often incompatible views. Look up and see the world that your work fits in to. It might just make you a better tester.


Are you an Independent Tester?

March, the time of TestBash Brighton. As always it was a pleasure to return to Brighton and catch up with some of the smartest, friendliest, and most inspirational testers.

The final talk of the day was from Nicola Owen. Her story was about moving from being a tester in a large organisation to being the only tester in a company. She talked of the challenges and benefits that came from being a sole tester.

Nicola’s talk made me ponder the role of a tester. Often we struggle to gain recognition in teams and companies. Testers are frequently the ones who get forgotten about when teams are thanked, or team lunches are organised. Is that really because we’re so forgettable? Does the sheer number of developers make testers invisible?

I believe that testers can, and should, be seen as a beacon of expertise throughout the company. Testers have so much knowledge about the product, the users, and the project risks. Every tester should know exactly when the project deadline is, who the customer is, and what the project goals are. Hopefully they also know the technology stack being used, the experience-level of the developers, and have a deep understanding of every step of the development and release process.

Knowledge is power. Testers don’t have to be technical. The ability to write code doesn’t have to be a measure of how good a tester you are. If you work with good developers then it probably doesn’t matter if you know how to configure a web server, or submit a binary to Google. What does matter is being able to initiate, and contribute to important conversations.

Talking about a problem out loud triggers your brain to think about things in a different way. For this reason many developers use the “Rubber duck debugging” technique to find issues in their code. Talking things through, even with just a rubber duck, can make you realise that you’re missing something, or spot an obvious problem in the design. If a rubber duck can bring this much value to a developer just imagine what a creative, and knowledgeable tester can bring to the conversation.

Whether you work in a small team or a large team you should take responsibility for your own role. A bigger team doesn’t mean you get to take less responsibility.

Behave as you would if you were the only tester around. Ask questions, and make notes, connect the information people give you and turn it into knowledge. If you come across a casual group conversation then get involved. Kitchens are a great place to spend time. Tell people your ideas and ask for their input. Not only will you learn some interesting, and possibly useful things, but you’ll also meet some new people. Don’t under-estimate the power of being well-connected within a company.

Your own experience is worth more than a thousand books. Reflect, search, and understand how your actions impact the team. At the same time read widely, watch talks, and engage with people outside of your organisation. Compare your experiences with others. What can you learn?

Always have an opinion about everything – even if you don’t always share it. Learning to question things, to spot the areas that could have been better will help you become better. Do this with your own work and with others’. When you read an article, or a book, question what it is telling you. How does your own experience differ?

Independent testers are resilient and self-supporting. They have the knowledge and skills to be able to excel as a sole tester in a company but they also have the knowledge and skills to make a larger test team a powerhouse. So don’t look around you and use your team size as a measure of how good a tester you need to be. Look around you and see the opportunities that are open to you. Now grab them.

The TestBash videos are now available over at The Dojo.

An efficient tester

Are you efficient? Do you work in a well-organised and competent way? I think most of us aspire to be effortlessly organised, to be in control of our lives. Some people raise children, run businesses, and still manage to find time to file their papers. The rest of us go around in a haze of hastily bought birthday cards, missed appointments, and mental to-do lists.

Unfortunately efficiency isn’t something that turns up in your life to sort things out like a benevolent fairy godmother. Efficient people have systems and techniques to help them organise their life.

To-do lists are a great way to keep track of tasks but if yours looks anything like mine they they can get overwhelming. Spending time at the beginning of each day choosing the most important tasks, the things that really, really, must happen today, can help. Once you’ve chosen the ‘must do’ tasks you simply hide everything else away for another day.

Batching tasks can also be a great way of saving time. If you have three bank transfers to make then doing them all at the same time will save you two online account log ins. Bulk buying birthday cards will save you multiple trips to the card shop. Grouping meetings together keeps useful chunks of the day available for hands on work.

Tools like Evernote, Feedly, and Instapaper help save important, or interesting things. I find the volume of interesting articles shared on Twitter to be overwhelming. Setting up an IFTTT channel to save starred items to my Evernote account has been a huge help. Now instead of trying to read or categorise links as they come up I save them away and deal with them all in one go.

Automation can be another way to remove waste. A simple automated script to create test users, or set the system under test into a desired state can save considerable time and effort. However, as with all automation, there are creation and maintenance costs associated with getting the script running, and keeping it running as the system develops.

Recently the fantastic Danny Dainton reminded me how important it is to automate small tasks as well as big ones. Removing a couple of mouse clicks a day might not seem like much, but over time those seconds will add up to big time savings.

One of my favourite tiny tasks to automate is editing URLs. When I’m testing I frequently switch between test environments and production ones. It only takes a second or two to highlight and change the necessary part of the URL but over an entire day these few seconds start to add up. A simple JavaScript bookmarklet takes this tiny task and makes it a simple button click.

As I go through my working week I look for monotonous, or repetitive tasks. Are there ways to batch these tasks? Can I use Boomerang to schedule emails in advance? Maybe I can find, or create, a tool to do this task for me. Each time I save time, I’m creating an opportunity to do something better, to become a more efficient tester.

How do you make yourself more efficient? Do you have a favourite tool or script to help you?

Tester connections

In 2011 I had coffee with someone who would go on to make a massive difference to my life. That person was Rosie Sherry. We were having coffee before one of the Cambridge Tester meetups that Rosie ran. It was one of those meetups where James Bach just happens to be speaking. One of those meetups that seem to be special to the Ministry of Testing family.

Over the years Rosie has personally introduced me to amazing testers. She’s organised events and created online communities that have led to me knowing so many more great people. Many people comment on how strong the UK testing community is. That strength comes from having someone like Rosie cajoling, encouraging, and quietly connecting us together.

The TestBash conference has turned into an event where you can hear great talks, but also a place where you can catch up with or meet some of the world’s best testers. It was at TestBash that I met my Weekend Testing co-facilitators Dan Billing and Neil Studd. A year later I got to know my awesome co-tester Kim Knüp.

Over the years I’ve been encouraged to write articles for The Testing Planet, film things for The Dojo, contribute to The Software Testing Club, speak at TestBash, and host Masterclasses.

I don’t think I’m the only one who has appreciated, and benefited from all the hard work that Rosie puts in to create an amazing testing community. Maybe today is the day that you return the favour by buying a TestBash ticket, signing up to the Dojo or even contributing some of your own ideas.