Over the past 4 or 5 years the web has become hugely advanced, and more and more users are accessing the web on mobile devices, all of this has led to new, more complex, web browsers and seemingly endless browser and operating system (OS) combinations. If you test a website or mobile site then you’ll need to consider where to focus your test effort, unless you have a very large budget (or a very simple site) then it is simply impossible to cover every combination in-depth so what should you do?
There are two things to consider when approaching browser/OS testing – the first is usage, what are your users using? Ideally you’ll have access to some user stats (analytics) to tell you which browsers and OSs your users are using. If not, then you’ll need to do some research, at the time of writing Internet Explorer (IE) 8 is the most widely used browser with IE9 in hot pursuit, but Safari (particular on mobile devices) and Chrome are both popular. Make sure you consider who your audience is, teenagers will be using different systems to tech savvy pensioners, which means that depending on your audience you either care deeply about Opera or you don’t care a cent, maybe most of your users are on Macs or maybe they’re on Windows,. Don’t forget to consider mobile devices, games consoles and TVs in your research and keep an eye on browser upgrade dates and OS launch dates to help you predict which combinations are relevant to you.
The second thing to consider is risk, based on your experience which browser/OS combinations will cause the most issues? This is very much an experience based metric, consider where you have previously found bugs and see if the risk applies here. If you’re new to the project then look through the bug tracking system to see what bugs have been previously fixed and consider if they could occur again, or maybe the architecture is similar to a previous project giving you an idea of how much you need to worry about cookies, cache and security settings. Some knowledge of how the website and app has been built will give you a huge help here so if you’re not sure then it could be time to have a chat with a developer.
So once you have your data you can start to put together a prioritised list of the browser and OS combinations that need testing. It is a very good idea to run through all your high priority test cases on all browser and OS combinations if possible, then you can run your medium and low priority tests on a reduced list of combinations, that way, if you run out of time you’ll at least have tested the site for the largest segment of your audience and for the riskiest combination of browsers.
So, in summary you should not expect to test every browser and operating system combination but with a little effort you can work out which ones are important or high risk for your project and then focus the majority of your test effort there.