One the same note, I happened to read recently about a couple of new trends: one around gesture based technology and one around the semantic web, which sounded interesting and promising, hence this brief blog. To give a very high level view into what these are:

 
Gesture based technology – revolves around techniques and ways to capture the user’s gestures and incorporate them into relevant actions

 
Semantic Web – revolves around smartly using the user data that is available in abundance to draw inferences and relationships, to make the computing environment carry out intelligent tasks for the user rather than merely following a set of instructions
 
Experts are betting on both these as some of the core trends in the computing technology over the next decade or two, which you will see are largely user-centric. And if things such as semantic web, take off well, the possibilities are endless. It can come in handy in almost every possible discipline in a variety of ways.
 
While all of this is great and we are excited to see them unfold, such complex evolutions pose their own challenges to the testing world. Let’s look at them one by one:
 
Gesture based technology – Feeding realistic user inputs into a test is always a key in maximizing the value of a test in finding valid bugs. We’ve had various sources of user input over the last decade including the traditional keyboard/mouse combination, from a console, from various touch interfaces etc. Test automation has evolved from the initial stages of a record and play framework to accommodate such changes. Now, with gesture based technology it only gets all the more challenging. How will a test recognize the user’s gestures and if it were to be designed the same way the programmatic interface has been designed to understand gestures, how will the tester ensure the validity of such gestures in ensuring the test data is accurate? The answer is not completely known as of today, but there will probably have to be some extra manual verification done at the automation testing phase to ensure the automated test has interpreted the gestures correctly. Some external factors such as multiple gestures, very quick, very slow gestures may also have to be tried to incorporate boundary conditions. Probably some amount of record and play will seep back into this futuristic test automation strategy

 
Semantic Web – The data available to process here is abundant. The tester has to smartly accommodate various data sources, and also create realistic test data is ensuring the program correctly interprets the various sources of information. In addition to test automation to verify the semantic protocols, some amount of manual intervention is going to be absolutely necessary to bring in the human element in the verification process. This may sound ironic, because semantic web is all about the program smartly assisting the user in carrying out his/her tasks without having to strictly implement what is has been told to do, whereas for verifying it, my take is that the human element cannot be completely done away with. Several new tools including parsers to read basic semantic resource description frameworks, browser plug-ins, validators etc. will enter the market to aid in semantic web testing. These in combination with enhancements to test automation frameworks and the core manual testing will make it possible for the test teams to test for this important evolution
 
At this point, as you see, some of the above points may not still be black and white. The important thing here is to be aware of these trends, keep track of them and invest in R&D in your testing labs upfront, around these areas, rather than having to play “catch-up” down the line.

 
1. Visual Impairments
2. Mobility Impairments
3. Auditory Impairments
4. Cognitive Impairments    

 
Like any other area of product development, special attention needs to be given to accommodate product usage for these categories of users too, by specifically incorporating their needs in the product’s architecture, implementation and quality assurance. Identifying lack of support in these areas later in the game will make it very difficult to fix them, leading to a possible situation of not supporting a large set of users for your product. So, what can you as a tester do proactively, to ensure comprehensive accessibility support for your product from the early stages?

   
1. Take time to understand the Accessibility Guidelines and Standards set by governmental agencies and consortiums including W3C’s Web Content Accessibility Guidelines 1.0 and 2.0, Section 508    

2. Understand Accessibility from a Usage standpoint and discuss your inputs with your product team upfront: If your team has done usability tests in the past leveraging real end users or is open to allowing you interact with real time users with accessibility issues, grab the opportunity. Interact with such users, observe them play around with the product and carefully make note of the kind of issues they face from a UI, functionality, usability angles. If you have a usability expert on the team work with him/her to analyze your observations. These findings go a long way in helping design the right product. Even if you do not have a product to demo as yet, talk to such users to understand their pain points and what they would like to see in a product such as yours

    
3. Proactively think of any collaborations you can work out with organizations that support people with accessibility issues to elicit ongoing valuable feedback from such users

   
4. Detailed Manual Accessibility Testing: Some content simply cannot be tested using automated accessibility validators/tools. As an example, an image of a Tiger could have its alt text set to ‘Mouse’ which is clearly inappropriate. There is no automated tool yet that could recognize the contents of an image and determine whether the alt text was correct. Ensure you chalk out a clear test plan with areas that you want to cover manually to cover the accessibility guidelines extensively  

 
5. Automated Accessibility Testing: There are several tools that scan through the source code as well as analyze the application UI to report core accessibility issues. Such findings greatly supplement the manual test efforts in reaching out to all corners of the code, that may sometimes be very difficult in manual code reviews  

  
6. Use of the VPAT checklist – The Voluntary Product Accessibility Template (VPAT) is a great resource for the entire product development team, especially the test team. Developed in 2009 and owned by the Information Technology Industry Council (ITI), it details out the requirements for Section 508 to accommodate for accessibility in the product under development. This checklist is almost like a readymade set of items to verify, that is available for the tester. He/she should proactively ensure this list is discussed upfront with the business, design and development teams so they are all on the same page on accommodating them in the product. This VPAT when included in your accessibility test efforts, is almost like a certification for your product’s compliance with Section 508  

  
Think about the points listed above. Some of them are pure science on specifics that need to be accommodated, some of them are pure art in terms of working with end users to elicit usability feedback, and some are a combination of art and science with your hands on accessibility testing efforts. Arrive at the right balance in your overall accessibility test efforts and collaborate with your product development team and end users in creating a product that is accessible by one and all!    

One of our senior testers, Himani, recently presented on this topic at the QAI STC conference in Bangalore and of the 25+ papers chosen for the “Best of Best” Paper awards, her’s closed in as the second runner up, which not only emphasized the depth of her content but the importance the industry is attributing to this overall topic. For more information on her presentation or any of the points mentioned above, please write to us at rajini.padmanaban@qainfotech.net

• 1. Voluminous content to be handled during digitization
• 2. Lack of time to focus on testing towards end of release, when product functionality testing takes front seat
• 3. Complex specifications to test
• 4. Compatibility to be verified across a wide range of platforms and devices

In this blog I am going to specifically talk about doing Content Digitization QA the smart way trying to handle as many issues as possible upfront before content hits the application’s UI. Explained in very simple terms, the content digitization process can be represented as below:

To tackle the challenges outlined above the “Content extracted into XML” is a very important stage to get a lot of testing done. Issues caught here and resolved before content hits the application UI saves a lot of time and money in the overall testing process. There are going to be some tests though which can be done only after the final ingestion happens (such as the overall fit and finish of the content on a specific device). But several tests such as content format and meta data checks and associated parameters can be done at the pre-ingestion stage. At both the pre-ingestion and post-ingestion stages, smart ways need to be adopted to identify potential areas of automation to save time and handle voluminous XML files without missing any checks.

We at QA InfoTech have built our own scripts along with an application that consumes those scripts to offer a neat UI to take on pre-ingestion testing. This has been customized with keywords and checks specific to the publishing and educational domains which are where a lot of digitization happens. This application and scripts can easily be customized for pre-ingestion testing for any domain, with minimal effort. We’ve built this on PhP making it simple to use and reporting issues down to the line number at which they occur to ease the debugging process. We’ve used these scripts for several of our publishing clients and the expertise that we’ve built help us guarantee a turnaround time of 24 hours for such pre-ingestion testing from the time, we have a new file to test (regardless of how voluminous the XML is).
We have our test automation framework built on open source technologies with built in virtualization, pluggable utilities for continuous integration and reporting, to address test automation at the post ingestion / application UI level. For more details on this framework, listen to our webinar hosted here. Seen below is a pictorial representation of our overall Content Digitization QA process. We soon plan to do a webinar explaining our content digitization automation solution in detail. Stay tuned for more updates. In the mean time, if you have any questions/need more information, please reach out to me at rajini.padmanaban@qainfotech.net

]]> http://www.qainfotech.com/blog/2011/11/content-digitization-qa-%e2%80%93-do-it-the-smart-way-2/feed/ 0 http://www.qainfotech.com/blog/2011/11/testing-your-test-code/?utm_source=rss&utm_medium=rss&utm_campaign=testing-your-test-code http://www.qainfotech.com/blog/2011/11/testing-your-test-code/#comments Thu, 10 Nov 2011 02:01:11 +0000 Rajini Padmanaban, Director of Engagement, Global Testing Services http://www.qainfotech.com/blog/?p=712 Continue reading →]]> In the world of software testing and quality assurance, we all know the value Test Automation brings in, to improve test coverage, overall product quality and the tester’s productivity. But all of this value flows in and the ROI is reaped only when the automation code is robust and reliable and can produce consistent results to catch product bugs. The term “product bugs” is very important here. If the automation code does not catch bugs (if this is truly because the product has reached a steady state and is largely bug free, then it is acceptable) or shows more false negatives (due to test code issues rather than product issues), there will be a lot of wasted effort including:

• Test Automation Effort – code design, implementation and maintenance resulting in wasted time, cost and human resources
• Triage time – involving the product team to look into the invalid bugs reported resulting in expended time, cost and human resources and more importantly the reputation of the test team
• Resource Usage – Machine, other infrastructure and software usage for automation execution.

One of the traditional solutions adopted to solve this problem and make the test code reliable and robust is to “test” the test code itself, as if it were product code. While this sounds great in theory, are all teams really practicing it? Also, it is all the more important to carefully chose a smart strategy and decide “How much to test” your test code because your goal is very different in this test effort compared to product testing goals. Any extra test effort here is again going to lead to a lot of wasted resources and is not going to add any value. Choosing what areas to test, how to test, and agreeing upon your goals of testing the test code are vital. Here are some best practices on “Testing your test code” which will help you make these important decisions.

1. Static tests around code reviews are very important. Testing your code to ensure coding standards are adopted is very important to ensure automation maintenance effort is minimal down the road. In such reviews check for modular test design, including things like reusable methods, decoupled test data to ensure code is clean and easy to work with.

2. Your test code’s functionality testing is what is most important – specifically focus on whether your code is testing what is intended. It would be good to pick a few core tests and run them both manually and through automation to ensure consistent results are produced

a. Focus more on unit testing your code. Leverage simple unit testing tools such as NUnit, JUnit etc. coupled with manual testing efforts using simple checklists to achieve your goal without making this process a complicated one

b. Testing the code that you write to verify builds (Build Verification Tests) in detail is very important because this is an area where you will use your test code very frequently. If build breaks happen due to test code issues it is not only a lot of wasted overhead but also adversely impacts the visibility of the test team within the overall product team

3. Before you actively start using your test automation code and share it with the product team, carry on a few trial runs where you execute the code on a few builds (if required use even local builds) to ensure the code yields consistent results (e.g. consistent passes and fails between builds) and closely evaluate the kind of failures returned to ensure they are indeed product bugs

4. Schedule your code to run in sequence a few times (e.g. 10-15 times) over a day in an unattended mode to ensure there are no built in dependencies which affect the test runs. E.g. access issues, any versioning issues, any software / hardware dependencies etc. Also, while a few initial trial runs could be on local environments, get your test code deployed and run against the actual test environment (at least on a trial basis) as soon as possible as this is what will reveal any other deployment and cross module dependencies

5. Code coverage – Running your test code with code coverage tracking on, is a very good test to determine the coverage achieved. This is a good objective evaluation of your test automation’s ROI also helping you to find out areas where you can beef up your automation and areas of product code that need to be removed

6. Don’t focus as much on areas such as performance, security, UI, usability, globalization, compatibility etc. while testing your automation code. In fact, you can almost eliminate a few of these, such as globalization out of the mix to purely focus on the functional aspects of your test code

7. Beware of any tool/framework limitations to ensure they don’t impact your test results including any support clauses to help build your automation on a sustainable platform

8. Test for your test code’s reporting capabilities – This is one area where you can focus on UI and usability aspects since your test code results are often shared with various levels in the management all the way from even the executive management down to the grass root level. Test for your test code’s accuracy, level of detail, archiving capabilities and usability aspects when you test the reporting module’s code

9. Set clear exit criteria – Similar to product testing, you will need to set clear exit criteria to sign off on your test automation code and move it to production (where you start using the code for actual product testing). Exit criteria need to be very stringent (I would even say 100% reliability at all times) in this case to ensure test automation code is reliable and can be leveraged to test the product. If specific scenarios/tests are delaying your process, de-activate them, and move them to a queue to be analyzed later. I say this because although the test code needs to be reliable, the team should also understand that testing the test code is only a supplemental activity and that the larger goal is really to leverage this and focus efforts on actual product testing. So, the test team has to lay down its priorities right in expending the required amount of time to test and strengthen the test code while still focusing on the quality assurance of the final end user product

Try these out and do share any other tips/inputs you have around “Testing” your test code. I would love to hear what you have to say!

Mukesh’s posts are available at: http://www.ctoedge.com/author/287
 

My posts are available at: http://www.ctoedge.com/author/274

Walk through the test strategy with the rest of the product team– call in a meeting with key stakeholders from the entire product team and explain what the testing strategy would be. This helps them understand the overall test methodology, test environment, key resources and timelines and their role in the overall test life cycle. The test manager/director needs to ensure this meeting happens early on in the product life cycle as an in person or an online meeting to ensure the product team’s buy in. Such meetings need to percolate at various levels in the product team to promote lateral discussions and reviews and need to be managed by the head of the testing efforts

Work with the product management team in finalizing user scenarios, incorporating field feedback and finalizing core metrics and data such as product performance numbers. The test team is the one that best understands the test environments and test data that will be used to simulate and test user scenarios. So, it is important for them to work with the business team in explaining additional product and feature design opportunities, test constraints, competitive analysis etc. to help the business team finalize realistic yet cutting edge product functionality

Work with the development team to highlight core scenarios that test would be watching out for. Educating developers on such tests to be run, will help them additionally watch out for those areas upfront, reducing chances of defects. Test should also explore the possibility of building a set of unit tests/build verification tests, which developers can use every time they check in new code. This will help catch any basic build breaks even before the test team looks at the build thus promoting a very efficient dev-test handoff

Deployment Team – If a deployment team exists, the test team should work on providing them a set of build verification as well as regression tests that they need to run every time a build is deployed. This will empower them to catch basic issues sooner and work with the dev team in resolving them while the test team can focus its efforts on more important areas such as test automation, performance testing, security testing etc. and more system level testing to best use their skills and competencies. For teams to take on such additional responsibilities though, the test team should do its part to make adoption as seamless as possible. They should look at providing the right test infrastructure, providing automated test suites (that have been well tested), required test data etc. to reduce any unwanted overhead for the other teams

Operations Team – Quite often the test team is pulled in very quickly as soon as a customer reports an issue from the field. While it is important for the test team to get involved in such cases, there are several cases, where the issues may not be real ones or they may be known issues that the team has decided to live with at this point of time. If the test team can work with the Operations team upfront, giving them an overview of existing known issues, workarounds, quick fixes, high level tests to run in a debugging process etc. they can empower and motivate the operations team to handle a lot of the customer issues themselves. This saves time for everyone involved and can possibly even provide a faster turnaround to the customer. A knowledge base/guide can be prepared and built upon as an evolving document to handle ongoing issues over releases

Effective defect management practices to all –  Defect management is an area where test can additionally add value by reducing the product team’s overhead. When multiple teams are working on a tight release, having a robust defect management process that is not very cumbersome and that everyone understands not only helps with everyone’s productivity but also with the overall product’s quality. If defects are not tracked and managed well, there is a very high chance of defect slippage, partial fixes, ongoing regressions all of which greatly impact product quality and burden the overall development costs. Specific to each project’s requirements, the test team can create a set of defect management best practices, own it and educate the entire team about it. This could include several things such as:

• What to include in a bug so as to increase the probability of reproducing it at the first shot
• How to resolve bugs
• Defect management timelines and SLAs
• How to handle bugs across platforms, locales, markets
• Special tools to use for capturing bug repro steps

Scalable infrastructure, accessible environments, use of ALM –When multiple teams work on a project and are especially globally distributed, the challenge often is around how to use an environment that is accessible to all, how to share artifacts etc. that everyone needs. A lot of time is often wasted on resolving such access issues. In the best interest of the entire team, the test team can take ownership of maintaining test beds not just for their use, but that can be leveraged by the entire team. Scalable infrastructure has been a breeze lately with advent of technologies such as cloud and virtualization. Test team would command a lot of respect if it can keep itself abreast of such latest technologies which will help improve the overall team’s productivity. Even if the rest of the product team has not implemented an ALM (Application Life Cycle Management tool), the test team should certainly look into doing so, given the number of artifacts it uses and needs to share with the product team, as discussed above. If the test team can take the lead in implementing such solutions, it certainly creates a lot of efficiencies and also motivates the other teams to follow.    

Conduct periodic bug bashes – every quarter or so, enabling the entire product team to take the time to test the product and provide feedback. This will also help the test team uncover any issues it may have missed and further clarify its understanding of the product based on reported issues. Such cross group exercises also help in team bonding across disciplines.

Clearly, the right levels of communication are very important to make all of the above happen at the right time and with the right entities. When such communication and cross group collaboration happens along areas mentioned above, the test team would have clearly made its mark in improving quality not just in its own areas, but truly across the board, helping create a great product for the end users.

Firstly, let’s look at what a TCoE is. A TCoE is a team model where multiple teams come together to work in a “shared” mode to share knowledge, best practices and resources. It is a model which retains the core specialization of individual teams yet offers room for cross team collaboration.
The ideal operating model to be implemented for a TCoE will be one that leverages the strengths of the TCoE setup and at the same time that does not add too much overhead and bureaucracy to the overall setup. It will be setup as a model that promotes end to end collaboration, yielding:

• Team load balancing
• Cross product knowledge
• Tools and process sharing
• Faster team ramp up
• Access to specialists

Individual silos with core teams will be setup for specific projects so as to have focused test effort to meet the quality needs of each of the projects. These individual teams will be managed by lead level resources who will manage daily project deliverables. Virtual teams will be setup with specialists to handle specialized testing needs such as performance, security, globalization, accessibility/usability across the TCoE projects. Given that these testing needs do not span throughout the Software Testing Life Cycle (STLC) and these resources are expensive they will work as a virtual team to address the specialized testing needs of all the projects in the TCoE umbrella, rather than just one project. All these leads, their respective teams and the virtual specialized teams will report to a test manager, who will manage the overarching TCoE. He/She will ensure:

• Projects share common best practices
• Teams understand the bigger picture and quality requirements at large so as to drive value upstream
• Teams are empowered with the required common training
• Buffer resources are maintained across projects to take care of flexible and dynamic project needs

Such a test manager will work closely with the stakeholders from the product team and conduct quarterly reviews reflecting the work done under the TCoE umbrella helping everyone on the product team understand and be on the same page about the project and product status and health, from a QA standpoint. This review will not be just a retrospective/backward looking meeting, but will be proactive in focusing on the road ahead to practically implement an objective QA effort.

The test manager should encourage monthly meetings of the entire team, inclusive of all projects in the TCoE umbrella. During such meetings the following could be discussed: Project accomplishments, challenges, work arounds, presentations made by team members on specific products/new technologies. Such monthly meetings could also set aside 15 to 30 minutes for cross group bug bashes, allowing testers from other teams to test cross group products. This promotes cross product knowledge as well as helps find bugs that the core team may have missed. Such meetings bring in good team bonding which plays a major role in the TCoE’s success. Email aliases should be setup for intra project and inter project communication helping share useful information to the required set of people on time. Dedicated trainings should be conducted periodically either during monthly team meetings or outside of such sessions. Depending on the size and scale of the TCoE implementation the test manager should determine the size of the buffer team to be maintained. Such a team will be trained and ready to work on any of the TCoE projects within a very short lead time. Typically a 10% shadow/buffer team of the overall team size is a good size to have providing the flexibility of the shadow team yet not being an expensive overhead to maintain.

 
Thus, a TCoE has its own nuances for a right implementation. A test manager plays a very pivotal role in its successful implementation. As a first step the test manager should get buy in from the overall product team in educating them about the model, how it works etc. and then carefully implement the TCoE as a step by step process. Such a planned approach along with the test manager’s objective mindset open to feedback will help make the overall model a success and well received by the entire product team.

 
We, at QA InfoTech, have successfully leveraged the TCoE model for some of our strategic project implementations including one for a leading ISV in the rich media product space. Our team strength for this account at peak times has been upwards of 75 people. This included testers’ onsite at the client’s location, and in our premises, each working on different release cycles, reporting into various client managers. From our end, we had an overarching TCoE setup which accommodated the individual needs of each project but provided specialists across the board. Our common pool of buffers combined with different release cycles at the client’s end, helped us effectively load balance resources. Such cross sharing of resources also helped share knowledge across projects, build detailed product and process training material and also the entire team understand the overall mission of the test effort. Such a team setup not only helped provide great opportunities to our team but also share the economies of scale with the client in providing cost effective services at a much faster turnaround.

1. Increasing complexity of product architecture
2. Several interfaces that the product has with external systems and dependencies
3. Increased user base, leading to heavy load on the system
4. Performance goals undergoing a lot of change owing to competition in the market and demanding customers
5. Faster time to market for product release, which shrinks available time for performance testing
6. Shortage in availability of specialized testers for performance testing
7. Investment in performance testing infrastructure and tools, which are often very expensive and may become obsolete soon

 
If you see the above list, some of them are product and user specific challenges while some are test process challenges. The key to solving the product and user specific challenges is really around developing a robust performance test strategy and adequately testing the product for its performance before release. So, what we really need to be looking at is how to address the performance testing challenges; this forms the core of our discussion below.
 
Performance testing has long been in existence, however it has rightfully been given a lot of thought around specialization, domain centric testing, using the right infrastructure which mimics end user environment etc. only in the recent years. As product development technology advances with use of secured protocols, rich internet applications, service oriented architecture, web services etc. developing performance testing scripts has also become more complex. No longer will a regular “record and play” tool work across a major set of products. There are a lot of very advanced performance testing tools that are entering the market from commercial players like HP, Microsoft, IBM which are constantly being enhanced to keep pace with the changing product technology. However, taking advantage of such tools does not come for free. These are all very expensive Commercial of the Shelf (COTS) tools, which you need to very carefully evaluate before investing. These might especially make sense for a product company to invest in, but not so much for a testing services company, because the services company is often required to use a tool that its clients have aligned their product with. On the other end of the spectrum, we have some excellent open source testing tools available for performance testing lately, which give the COTS a run for their money. A lot of companies, including product companies have started looking at such tools as good alternatives from a cost and feature standpoint, to use in their performance testing efforts. Jmeter is one such tool which a lot of companies are leveraging in the recent years. Again talking of trade-offs here, Jmeter is an open source tool available free of cost, easy for a tester to ramp up on, but is not comprehensive in its feature set, unlike the COTS. It has quite a few limitations around its reporting capabilities, which is a very important feature in performance testing, especially when the tester is handling large volumes of data. Also, this is data that the management and business teams are going to be interested in viewing, so a performance testing tool has to certainly offer very rich reporting functions.
 
This is where a company can draw its balance and decide to go with using an open source tool to take advantage of what it has to offer, but also invest in building reusable intellectual property (IP) on top of it, to address the tool’s limitations. Doing so, is beneficial for all entities involved as seen below:

1. Clients (Product companies) are able to get performance testing done at a cheaper cost without compromising on quality
2. Vendors (Test Service companies) are able to offer good value add services to their clients, differentiate themselves in the services market amongst their competition and also offer interesting out-of-project opportunities for their employees to work on building such core IP
3. 3.     Very challenging and interesting opportunities and thus good career path progression avenues are available for the employees of test services companies

 
Such reusable IP and frameworks often also serve as productivity enhancers – e.g. data generation, especially user data creation is quite an overhead in a performance testing effort compared to regular functional testing. If the performance testing tool that you choose does not offer good user data generation features out of the box, this is an area worth investing some time on upfront as it will save time both in the current test cycle as well as future cycles. Automating such time consuming and monotonous jobs, also frees up time for the tester to focus on more important areas such as performance benchmarking, system analysis, competitive data analysis etc. Also, to maximize the performance testing efforts, the product company should involve the performance testing team right in the early stages of product design. This will help the performance test team work in unison with the business team to understand end user’s performance expectations from the product under development; Understanding this will help the team chalk out “measurable” performance goals for the product and help the entire product development team understand these goals on the same page. Another benefit to such early involvement is the suggestions that the performance test team can provide on the product architecture from a performance angle (e.g. it could be the number of interfaces the product has, how certain web service calls are made, threshold values that are being set, production system configuration parameters etc.) which will help reduce the number of performance bugs in the test cycle. Such interactions between the developer and tester truly help build a quality product in the limited available release time.
 
We talked about the challenges around performance testing tools and availability of time to test for performance and some potential solutions to address them. Another major challenge we broached above was the availability of the right test infrastructure on which performance testing can be done.
 
Performance testing is one area which can be quite time consuming, especially in doing tasks that aid the core testing activity itself. For e.g. setting up the test infrastructure which mimics the production deployment environment, user data generation, report generation etc. It is considered a best practice, if the product or services company can find the right and optimal solution to reduce the tester’s efforts on these supplementary tasks; this will then give the tester more time to focus on tasks that really need his attention such as deciding on the performance test strategy, scenario identification, scripting, test execution, result analysis, benchmarking and regression. That brings our discussion to how we can help the tester spend less time on setting up the test infrastructure – in the more recent years, the answer to this question is using the “Cloud”.  The cloud basically offers services to the end users on a pay per use or lease model. Depending on what the service is, a cloud could basically offer “Software as a Service (SaaS)”, “Platform as a Service (PaaS)” or “Infrastructure as a Service (IaaS)”. We are specifically herein interested in “Infrastructure as a Service”, where one connects to machines on the cloud and uses them for the performance testing effort. Doing so, offers a lot of benefits to the product or services company, which include:

1. No locking of funds in expensive hardware for performance testing – this is a very compelling benefit since performance testing often needs very high end hardware with several core CPUs, lots of RAM etc.

1. No risk of hardware becoming obsolete – Besides just the initial investment, one of the concerns of owning the hardware is that hardware specifications are changing by the day. So, investing in such expensive hardware just for performance testing will not yield the required “return on investment” for the product or service company.

1. Good control on overall test costs – since you pay for the test hardware on the cloud only on a “pay per use model” or “possibly a minimal seat cost” model, the costs associated with test infrastructure are kept low and under control. Costs are kept under control also from the standpoint that the tester’s time that is required to setup the machines is now very minimal or non-existent. Such savings in time also translates to savings in costs.

1. Improved accuracy in testing results and better interaction in the product teams – Performance testing is often an area where reproducing bugs/issues is very difficult especially when global teams are involved. This is often because the entire team does not have access to the same machine setup and setup is a huge dependency factor in getting predictable results. However, when the team uses infrastructure on the cloud, anyone on the team can access the same setup, improving the chances of reproducing a given issue.

A lot of commercial clouds are available these days at very competitive prices and feature sets for you to choose from. Some of the more popular ones are from Amazon, Google, Microsoft.
 
Before deciding to go with a cloud based solution for IaaS, a product or a services company should clearly plan for its cloud usage strategy. Some of the core things to consider here include:

i)                Clear cut test strategy:

Identify testing objectives – cost benefits, scalability, and return on investment

Identify types of Performance testing to be conducted

ii)               Infrastructure –  Identify the following:

Hardware and software requirements

Test automation tools

Number of concurrent users

Network bandwidth

Memory requirements

Storage requirements

Application usage (Frequency)

iii)             Service providers:

Must have high QoS

Must be able to provide the entire suite of end to end services – infrastructure, software licenses, setting up and dismantling the environment

Must take minimal time to set up and dismantle the infrastructure

Must have proven record of adhering to SLAs

When all of these are carefully planned and a cloud based test infrastructure setup is leveraged, it brings in a lot of time and cost savings to the service provider and product company, helping bring down the overall product development costs.
 
One of the last core challenges of performance testing we mentioned above is the lack of performance testing engineers. Performance testing is a highly specialized area and a very important area that plays a critical role in a product’s acceptance in the market. Every company has to pay close attention to how to build a center of excellence around performance testing that defines associated test processes, tools, best practices and also how to build, groom and nurture performance test experts. This includes hiring the right talent, spotting the right talent within the company that can be trained on performance testing, offering periodic workshops and sending the experts to the right trainings to further build their skills and keep them motivated, investing in the right resources (infrastructure, tools, internal projects such as building internal IP) that will empower these engineers to give their best on the job and also motivate them. All of these go a long way in giving the company a leading edge in building the right performance test team and performance test center of excellence, helping them meet the overall performance goals of the product under development.
 
Having talked of the overall process of how cloud and open source technologies can be leveraged in performance testing, I will next talk about how we at QA Infotech, an independent test services company, have implemented it at our end, through which we have been able to add value to a lot of our customers

At QA InfoTech, we have an R&D division that is focused on solving real world problems and challenges we face on the ground. These include learnings from our project execution efforts and also building solutions for futuristic technology trends that we predict. In accomplishing the above strategy for innovation and value add, performance testing is a core area of focus for our R&D division. The division is co-headed by our CTO and Director of Engineering, who guide the team with the required technical leadership, empower them with the right resources (tools, infrastructure etc.) and also ensure that both QA InfoTech and our customers benefit from the output of this R&D team.
 
To address a given problem, we typically put together a heterogeneous team comprising of a few core/ resident engineers from our R&D team and a few people from our core project team. This combination yields us best of breed solutions as our resident R&D team brings in expertise and best practices from having built past solutions and our core project team brings in specific product knowledge and how to implement the solution being built for a given client. Also, we take care in ensuring that such products being built as part of our R&D efforts go through a typical product development life cycle (mostly in an Agile model) where we have targets for working prototypes to be built every 15-20 days. We are also particular that such products undergo a rigorous test cycle to ensure a quality finished product, that we can leverage both as productivity enhancers for our employees and as value adds for our clients.
 
In the performance testing space, the following are our R&D team’s brain children:

1. Plugins for Jmeter for SNMP, reporting (using Jasper Reporting XMLs) and measuring streaming HTTP
2. Performance testing solution to leverage Jmeter and the Cloud (Amazon EC2) to ease the test infrastructure setup and to be able to run performance tests across multiple OS / Browser combinations

 
Steps taken on by QA InfoTech to:

1. invest in R&D efforts to build ongoing solutions,
2. build “just in time” solutions to address client’s product testing challenges,
3. conduct proof of concepts (PoCs) to justify the use of a certain solution etc.

 
go a long way in both:

1. establishing a strong relationship between the client and QA InfoTech and
2. helping us ensure the release of a quality product that will scale in the market to meet both the client’s business and competitive needs

Here is a screen cast of this event:

If you have questions about the Webinar or this topic please feel free to drop a mail at info[AT]QAInfoTech.net