The growing number of platforms and devices to be supported across applications has made compatibility testing very crucial now, more than ever before. The multitude of desktop and mobile operating systems and browsers, while on one hand offer a great set of options for the end user, make it challenging to ensure application and device compatibility within the given time and cost constraints.
Evolution of Compatibility Testing:
If we look at the evolution of compatibility testing, one would have started off setting up multiple physical machines each with the required system configuration and you would remember seeing compatibility testing labs filled with a bunch of machines. Those were the days when the set of browsers and operating systems were limited to a handful that could easily be tracked and tested on. Following this timeframe, were the days of virtualization software, mostly from commercial players such as Microsoft, VM Ware, CITRIX, Oracle enabling testers to test on multiple operating systems from one single machine. This still called for the tester’s time to setup and configure the machine and the virtualization software. Ongoing upgrades to the virtual copies and images, associated clean-up effort after each test pass, made this a time consuming process although it was an improvement over the earlier one on maintaining multiple physical machines. Around the same time as cloud computing hit the market, mobile computing also started booming. The combination has in some sense created a whole arena of opportunities, challenges and solutions for the testing community. Infrastructure as a Service (IaaS) has made on-demand machine and software availability so feasible saving testers’ a significant amount of setup time. From a cost angle as well, one pays for actual usage helping reduce any redundancy, wastage in obsolete machines etc. making the process both time and cost effective.
Current Compatibility Testing Scenario and Solutions:
When it comes to compatibility testing, you are mainly looking to verify your application’s functionality, UI and rendering accuracy and consistency across platforms. In the current day world, cloud computing and associated virtualization solutions on the cloud have eased the compatibility testing process significantly for the testers. A plethora of commercial and open source automated solutions are available. It is important to pick the right solution based on your requirements and constraints including – cost, time, support and customization options, parallel execution feasibility. Here is a neat consolidated list on cross browser compatibility testing including automated open source solutions available on the cloud.
Typically, when you are in the process of defining what your support matrix needs to be, you can rely on a couple of data points:
a. Look for market usage statistics such as http://www.netmarketshare.com/operating-system-market-share.aspx?qprid=8 to decide what the latest usage patterns are and optimize your test effort accordingly. This is especially useful for v1 products where you do not have a user base to start with
b. If you are an existing player, I would suggest you decide your test matrix, based on support that is planned for the current release, keeping in mind past usage patterns and problem areas. Google analytics kind of tools help you determine such data points that can play an important role in your decision making. Here is an article that I liked on this topic, that talks about setting up Google Analytics and helping you see usage patterns along with an array of cross browser testing tools that are available in the market today.
Whether you are looking at compatibility testing from a multi browser, multi device, multi platform support, each of them have their own challenges, yet are inevitable if you want your product/service to succeed in the market place. The good thing is, alongside these challenges arise new opportunities and solutions as discussed above. Clearly understanding your needs upfront, along with analysis of available solutions to hone in on what works the best for your product is an important step in finalizing a robust test strategy.