Continuous Delivery

What works for me might not work for you. Reverse. Repeat.

Earlier this week I posted my thoughts about DevOps and Continuous Delivery. TheSunshine tester responded on Twitter with a strong defense for Agile

I love that methodologies are able to create such passionate responses. Even amongst the truly committed I doubt I would find any passion for the Waterfall method. Yet the Agile, DevOps, and CD communities seem genuinely excited about working together in this way.

Software Development has greatly improved in the last few years but I still think we have a long way to go before it is considered reliable. Last year reported that the majority of ERP projects are over time and over budget. If Agile is truly the silver bullet then all teams would be adopting it, and finding success with it. Unfortunately that doesn’t appear to be the case.

No matter how successful Agile, or any practice might prove to be we should still be seeking new and better ways of working. Humans are unique and our involement makes every project equally unique. The team who has success with agile is unlikely to have the same team make-up, or collective experience as one who doesn’t transition, or who fails to succeed with agile. Our interpretation of what a process involves, and our previous experiences all serve to influence what the end process will actually look like.

We need to have discussions about DevOps because it has arisen from an apparent divide between developers and Ops people. Whether this divide is caused by agile, or a failing of the implementation is irrelevant, the fact is that many teams are finding things as they stand are not good enough.

At Songkick we’ve followed Lean practices for many years. We have small cross-functional teams, daily stand-up meetings and weekly retrospectives. We measure things and try to remain flexible and responsive. Despite this we were having trouble releasing working software. Implementing Continuous Deployment was an enhancement to our agile practices. It went beyond giving us a framework for working together and gave us the steps we needed to make frequent releases too.

One of the criticisms I often hear is ‘it wouldn’t work here’. Hopefully these reservations come from understanding that there are no best practices, but whatever your reason for resisting copy-book change you are right.

What has worked for me, in my team, on my project will never work in exactly this way with your team.

Simply put, best practices do not exist.

Continuous Delivery is the current trendy approach and that is a concern. You should be trying to fix a problem not copy a textbook. If you cannot identify the benefit of any practice, agile ones included, then you shouldn’t be using it. One of the most valuable skills we learned from adopting Continuous Delivery was the ability to review our process and identify exactly what was working and what wasn’t. We continue to review things, making changes as our needs, or the team, changes.

Talk to other teams, listen to their experiences, read widely but use their processes as your guide. The most useful thing you can learn from other teams is how they implemented change. What were the triggers or warning signs? How did they know it was working? How did they choose this change?

These are things that you can apply to your own situation. These are the things that are likely to actually help you deliver good software.

DevOps with Testers

Last week I spoke at QCon London about Songkick’s experience of moving to Continuous Delivery. You can download the slides, or watch the longer version of this talk on the LondonCD Meetup’s Vimeo channel.

My talk was part of the DevOps and Continuous Delivery track hosted by Eoin Woods. QCon videos of all the talks should be available soon.

One particularly interesting talk was by Steve Thair on ‘DevOps and the need for speed‘. Previously I thought DevOps was about developers and Ops people collaborating to enable faster software delivery. Steve’s definition went much further and ended up being closer to how I think about Continuous Delivery.

Steve defined DevOps as a model that “encompasses a product-centric view across the entire product life-cycle (from inception to retirement) and recognises the value in close collaboration, experimentation and rapid feedback.” He emphasised that the business and testers should be engaging in the collaboration just as much as developers and ops people.

An interesting distinction between Steve’s DevOps definition and Dave Farley’s Continuous Delivery definition focused on the definition of ‘Done’. Continuous Delivery is defined as being about “making businesses more experimental through the early and continuous release of valuable software”. It appears to be complete once the release is out in production. DevOps adds an important final step of evaluating how much value the release is actually adding once it’s in the hands of users.

I love Steve’s definition of DevOps but unfortunately think this term is sadly restricting. I doubt business people or testers will feel included in something called ‘DevOps’. Even more confusingly I believe there are both devs and ops people who do believe ‘DevOps’ only covers devs and ops.

Even re-defining Continuous Delivery to extend beyond the release doesn’t really help the situation. We’re trying to create a process that keeps the software in a state where by it can be released whenever the business wants it to be. Obviously there are some questions around why the business would suddenly decide to release with little notice and whether we should support that or not. Leaving this aside I think Continuous Delivery is a positive way of working to remove unexpected delays to releases.Any process that encourages collaboration and stress-free releases should be given strong consideration in my book.

A name is always just a name but in the case of improving processes and making sure all roles are covered I think it becomes pretty important. We need to hear everyone’s voice as we attempt to fix software development.The concepts being discussed are important and will affect us all. I personally feel that neither the ‘DevOps’ or ‘Continuous Delivery’ terms are quite right for describing an attempt to engage with all stakeholders and release timely, and functioning software to end-users. Regardless of this I believe everyone involved in software design, development and releases should be involved in deciding the best way to fix the software industry.

What do you think? Are you a Continuous Delivery or DevOps fan?

What is the Business Value of Continuous Delivery?

Michael Bolton sparked a fascinating discussion on Twitter by asking what exactly is the business value of Continuous Delivery? I’ve attempted to capture the conversation on Storify but I also want to explore the topic a little further.

Michael quite rightly pointed out that beyond CD being technically possible and cool, it is difficult to see the business value of the approach. I agree with him that there are practically no systems that could ever need to release new features every few minutes. In fact I can’t think of a single system which would require such frequent updates.

Despite this I believe that Continuous Delivery can provide value to the business.

Let’s step back for a moment and consider what ‘Business value’ actually means. It obviously means ‘providing something of value to the business’. Usually this ‘something’ will be a new feature or maybe a bug fix. Being able to release frequently could certainly get either one of those out to the business, but CD is only one possible solution to this problem. Even more important that releasing frequently is releasing reliably.

Almost every business wants to know exactly when a feature will be useable by real users. Releases require code to have been written and tested. Deploy scripts need to be reliable. Code needs to be packaged for release. Release documents, notes, maybe even marketing materials need to be produced. It seems logical to think that with so many parts to a release, and most likely the need for multiple people or even teams to be involved, you should release as infrequently as possible to reduce impact. In reality the more infrequently you perform an action the more likely you are to make a mistake.

The more frequently you practice a series if actions the more natural they feel.

Painful systems requiring workarounds or insider knowledge are often accepted if the user only has to feel the pain once in a blue moon. Bringing the pain forwards is really the only way to motivate people to fix these issues.

Simon Stewart responded to Michael Bolton’s initial Tweet questioning the business value of CD with the benefit of cleaner code. To be successful at CD you will need to learn how to be disciplined about writing and maintaining code. Developers need to be adding automated checks, and builds will need to be manged. My experience of adopting CD showed that having a goal that the developers were excited to achieve could be be a great motivator to get people fixing test environments, taking responsibility for quality, and maintaining automated checks. Obviously there are others ways that you could achieve these benefits but it would be foolish to dismiss the impact that CD can have on a development team.

Maybe the value of CD only starts to appear when you have multiple development teams contributing to a single code base. With a single developer and a single codebase the thought of releasing multiple times a day points to poorly written and tested code being endlessly thrust upon the user. If you consider a CD approach for multiple small development teams working on a shared codebase then you can see that if you do 3 releases in a day you are not necessarily saying that each developer released in that day. In fact at Songkick our release problems increased as the development team grew. Each developer was spending around 1-3 days on building and testing new features but this alongside regular bug fixing and technical improvements meant we were exceeding our release capabilities.

We could see that creating a process that allowed developers to commit code freely and receive fast feedback would be a huge improvement. Once we had made the necessarily cultural and technical changes to our process, attitude, and environments, we realised that with just a few additional changes we could ship our code directly to production without increasing the risk beyond our comfort zone. Taking this final step was hugely motivating for the development team; we wanted to create an environment where everyone has access to the information they need, can make decisions based on their expertise, and has access to support when they want it. At Songkick we did this by giving all developers the freedom to ship code whenever they were ready.

Unfortunately it seems rare for a team to sit down and define their release process, instead they rely on some process which evolves over time to handle new tools or respond to problems. Business value can only be achieved if you have code that actually makes it to the hands of your users. Being able to release the code to production in minutes doesn’t remove the need to define, design, build, and test changes but it can help you to develop a reliable release process.

I consider CD to be a technical solution to the problem of release delays. As with all releases there is likely to be some impact to the user, consider what this means and then plan accordingly.

Look at the risks of every release and identify actions to monitor or mitigate them. CD gives you a hugely flexible release process, you have the means to release quickly and frequently, but you also have the choice over whether you use it.

Build Quality In

Build Quality In

I’m excited to announce that Songkick‘s experience of moving to Continuous Deployment will be included in the forthcoming book “Build Quality In” edited by Steve Smith and Matthew Skelton.

The book, available on Leanpub aims to gather together experiences, lessons learnt, and the highs and lows of building quality in. I’ll be contributing my experience of moving to continuous deployment and in particular taking a look at the impact the change had on testing and quality at Songkick.

Testing In The Pub – Part one of my interview about Continuous Delivery

Part one of my interview with Testing in the Pub is now live. You can download it from

Here are a few links for further reading (can also be found in my comment on the Testing in the Pub website):
– The place to start –
– Etsy’s developer blog and in this post in particular –
– There are some interesting webinars available on
– Videos from the Pipeline conference –

In addition I personally enjoy the following blogs:

And soon there will be a great book of experience reports all about CD. OK so I might have a report included but It’s still going to be a great book 🙂

Please comment with links to other blogs or resources that you find interesting or useful.

Testing in a Continuous Delivery World

The video of my recent talk at the LondonCD Meetup group is now live. The talk was kindly filmed by BBC Future Media.

May 2014 – Amy Phillips – Testing in a Continuous Delivery World from Software Engineering Practice on Vimeo.

Slides are also available at

Choosing Continuous Delivery

Continuous Delivery (CD) is looking like it’s the new Agile. Teams hear about the wonderful things that happen with you implement CD and rush to get on-board. In reality CD is the same as any process, there are pros and cons to adopting the approach. Not every team could. or even should want to change their culture and process to allow CD to work. However if you have a team, including business people, who are willing to take the steps required then CD can have a huge impact on team morale.
Process change is hard. You are asking a group of people to change the way they do things. Not all of these people will agree that you’re moving to a better approach and so at least some resistance is normal. CD requires everyone on the team to trust each other. You need to work together to agree on a process that allows you to move fast and deploy code to production far more frequently that you are used to. You need to understand and agree on risk levels and technical changes. How will you test the builds and who gets to make the decision as to what gets deployed? Most important of all you need to agree on what will happen when things go wrong.
Despite the obvious difficulties of making these changes I still believe CD is a positive process change to make. Full team conversations around acceptable risk are always going to be a useful thing. Actually sitting down and defining a process which can then be completely managed by a tool such as Jenkins will make life easier for everyone. Most significant of all are the benefits of creating a culture where developers are trusted to release code as and when they need to.
If you would like to hear more about how Songkick moved to CD then join me for the EuroSTAR Webinar Wednesday 11th December 2pm GMT