Help, there are too many web browsers to test

Over the past 4 or 5 years the web has become hugely advanced, and more and more users are accessing the web on mobile devices, all of this has led to new, more complex, web browsers and seemingly endless browser and operating system (OS) combinations. If you test a website or mobile site then you’ll need to consider where to focus your test effort, unless you have a very large budget (or a very simple site) then it is simply impossible to cover every combination in-depth so what should you do?

There are two things to consider when approaching browser/OS testing – the first is usage, what are your users using? Ideally you’ll have access to some user stats (analytics) to tell you which browsers and OSs your users are using. If not, then you’ll need to do some research, at the time of writing Internet Explorer (IE) 8 is the most widely used browser with IE9 in hot pursuit, but Safari (particular on mobile devices) and Chrome are both popular. Make sure you consider who your audience is, teenagers will be using different systems to tech savvy pensioners, which means that depending on your audience you either care deeply about Opera or you don’t care a cent, maybe most of your users are on Macs or maybe they’re on Windows,. Don’t forget to consider mobile devices, games consoles and TVs in your research and keep an eye on browser upgrade dates and OS launch dates to help you predict which combinations are relevant to you.

The second thing to consider is risk, based on your experience which browser/OS combinations will cause the most issues? This is very much an experience based metric, consider where you have previously found bugs and see if the risk applies here. If you’re new to the project then look through the bug tracking system to see what bugs have been previously fixed and consider if they could occur again, or maybe the architecture is similar to a previous project giving you an idea of how much you need to worry about cookies, cache and security settings. Some knowledge of how the website and app has been built will give you a huge help here so if you’re not sure then it could be time to have a chat with a developer.

So once you have your data you can start to put together a prioritised list of the browser and OS combinations that need testing. It is a very good idea to run through all your high priority test cases on all browser and OS combinations if possible, then you can run your medium and low priority tests on a reduced list of combinations, that way, if you run out of time you’ll at least have tested the site for the largest segment of your audience and for the riskiest combination of browsers.

A key web testing rule is to remember that Javascript and Ajax will be handled by the web browser, which makes them likely to functionally differ between browsers but anything handled by the web server should behave in the same way on all browsers although you still need to test for styling issues. So if the new feature is a server calculated pricing component you only need to fully test the calculating logic once but you’ll need to test the price input and output fields on each browser/OS combination that is important to you.


So, in summary you should not expect to test every browser and operating system combination but with a little effort you can work out which ones are important or high risk for your project and then focus the majority of your test effort there.


One comment

  1. It’s a very good view point; testing websites appears to be an ever increasing area of the development process. It makes me wonder if smaller/medium sized agencies should recognise this more and begin to think about creating new teams of testers, specifically designed to help keep the quality assurance that was once so easily acheivable. As a UX developer myself; the amount of testing and debugging time can take longer than the development itself. A personal opinion of mine; but I do get a sinking feeling sometimes when I realise that I have to test a relatively large website, cross browser and cross platform, all by myself just because I was the UX developer on the project. Leaves me to question if there has a come a time when a UX developers job has become to big for one person to handle. At no fault to developer when i say this; just a reflection of the how the changes in user behaviour can affect job responsibilities.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s