Software Testing Services - QA InfoTech Corporate Blog

Designing Tests in the Agile world

Rajini Padmanaban, Director of Engagement, Global Testing Services — Wed, 02 May 2012 18:01:06 +0000

Agile lifecycle and methodology of developing software are
terms that have gained a lot of popularity in the last 5 years or so. Everyone
has come to accept the iterative development cycle, short release cycles, focus
on collaboration between teams with lower emphasis on documentation across
disciplines. One of the associated areas in Software Testing is how much
emphasis should be given to formal test designing and should it be done any
differently in Agile than in traditional waterfall development models. An
associated read would be one of my recent blogs on Test
Strategy

Let’s look at a few different ways in which tests can be
designed in an Agile model:

Test Driven Development (TDD) – If your
team is one that takes the route of TDD, where tests are written even before
product code is in place, the test team gets a head start into the overall
discussions, product features and is able to get robust tests in place to
enable the coding process. Greater emphasis is placed on automated tests in
such cases than manual tests.

Scenario/Story based test designing –
since detailed specifications are often times not available unlike in the
traditional development model, tests are designed to guide the team through
specific scenarios and stories rather than being a strict set of cases that
need to be executed. With the over-arching umbrella of the scenario based test,
the tester has his creative zone wide open to cover varied tests that may not
be necessarily documented. This gives him the advantage of scenario based
coverage at the same time, not spending too much time documenting every single
test case.

Continue building on your regression suite
– Typically, a large portion of the tester’s time goes into exploratory testing
in the Agile model. While exploratory testing has its own benefits, what is
important to keep in mind is the bugs found from such test efforts. Test cases
should definitely be added for valid bugs found through exploratory testing, so
that they form part of the regression test suite to be executed in future test
cycles.

Test Milestone between releases – Test case
maintenance is a very important effort, especially in development models like
Agile, where features are changing rapidly, and a lot of variables are very
dynamic. Unless care is taken to ensure the tests in place are optimized,
up-to-date and relevant, very soon they will affect the product quality adversely.
Although the team may not have time during the release to focus on test
maintenance, they need to work with the rest of the product team to build in a
day or two every quarter, to allow for test case maintenance and clean-up. Such
times can also be used to add new tests based on feedback from customers on
field, other disciplines such as marketing, business, development, design etc.

The point to keep in mind, is with evolving development
models, the ways in which tasks are done will continue to change. One has to
understand the underlying purpose of the task (in this case, designing tests)
and look for ways to accomplish that underlying purpose as opposed to sticking
with their old and rigid methods that have been used all along. There may be
other interesting ways in which you design tests in an Agile model. If so, I
would love to hear from you. Please leave a comment on this blog post or write
to me at rajini.padmanaban@qainfotech.net

Domain Knowledge in Software Testing

Rajini Padmanaban, Director of Engagement, Global Testing Services — Tue, 24 Apr 2012 23:19:26 +0000

Domain Knowledge in Software Testing – is it really needed and how do you address the dearth of experts?
Until a few years ago, testing was a horizontal discipline. Someone with core disciplinary knowledge, who understood how the Software Test Life Cycle (STLC) worked, who understood test implementation from standpoints of test strategy and planning, test case design and execution, defect management could really test any product at hand.
Then came in a timeframe, where specialization in test started gaining prominence. Someone with specialized skills in areas such as performance testing, security testing, database testing, localization testing could build a niche for themselves in these areas going deeper into each of these disciplines.
Over the last few years though, product domain knowledge is becoming more important than ever. Just the horizontal knowledge or specialized testing knowledge is not proving sufficient in adequately ensuring quality in the competitive market landscape. The vertical or domain knowledge is becoming increasingly important due to several reasons – mandated by compliances, audits and checks, need to verify domain specific content or simply put even to be able to bring in the edge in testing from the standpoint of realistic end user workflows. Diagrammatically, this is how the ecosystem is now looking like:

Some examples of popular domains include: Health care, Life sciences, Education, Banking, Financial Services and Insurance (BFSI), Transportation, Government, Banking, Retail, Media. If we were to look at examples in these domains that necessitate the vertical knowledge:
1. An understanding of the banking transactions, protocols, Sarbanes-Oxley is really going to help in the banking space
2. How Learning Management Systems (LMSs) work, the foundation of SCORM/AICC form a great knowledge base in testing for educational applications
3. Knowing how claims processing works, understanding various categories and associated workflows in insurance for life, health, fire, auto go a long way in effectively testing in the insurance domain
4. How file formats function, various media types, the kinds of testing associated with media such as install/uninstall, checksum validations help in ensuring successful media testing
With all of these said, one also has to understand the reality on ground. It is quite a challenging task to build and maintain a team rich in testing and domain knowledge. It is easy to find one or the other, but can get very tedious and expensive, to find the combination. Some of specific challenges in staffing such a team include:
1. Dearth of subject matter experts with knowledge in both testing and the domain at hand
2. Expensive to maintain such an in-house team of experts often because of the shortage of such testers
3. Ineffective to keep them in-house year round, as you actually need their expertise only in specific phases of the project
We, at QA InfoTech, being independent test experts that specialize in a handful of domains such as BFSI, Education and Publishing, Media, Retail have certainly faced this challenge too, over the last few years. Here’s how we’ve addressed or mitigated them, which will hopefully come in handy for you too, for your specific project needs.
1. Build a common test repository for domain specific test cases that largely cover user and business workflows, which can then be leverage and customized specific to individual project needs
2. Hire subject matter experts and train them on software testing. This strategy has successfully worked for us, where for a call center software testing, we actually hired real time support engineers and trained them on testing
3. Train your internal team on the domain that they need to work on, by inviting domain experts to formally impart subject matter expertise. Over time build core training material that you can use for ramping up newer engineers
4. Hire selective engineers who have both domain and test knowledge but share them across projects such that they are resident experts in the company who consult across the board, subject to projects’ NDA requirements
5. Collaborate with organizations and universities to bring in domain experts on an as needed basis which will be both cost effective and really have them cater to your demands as and when you have them
We’ve had success in each of these forms presented above in our projects. Think through your subject matter needs specific to your project and its constraints to decide the approach you want to take; obviously you can go in for one or more of these techniques to give you that right balance of domain and testing disciplinary knowledge coverage. Happy testing!

Is a test strategy really becoming an obsolete artifact?

Rajini Padmanaban, Director of Engagement, Global Testing Services — Tue, 10 Apr 2012 00:30:37 +0000

Until a few years ago, I vividly remember a separate phase in the software test life cycle that used to be dedicated to the development of a master test plan or the so called holy test strategy which served as the guiding document outlining the software testing process specific to the product under test. It was usually drafted by the test director or the test manager and was representative of the test team’s intelligence in helping them making the product sign off call. Such was the importance of this document; it also served as a reusable artifact often carried across releases, products and teams. Given the relative stability in the development environment such cross sharing and reusability were feasible.

However, over the last 5-6 years given how agile the product development environment has become de-emphasizing documentation and focusing more on dynamic planning and execution on the go, the usefulness of an artifact like test strategy has become questionable. In my experience, having worked on both the traditional and agile projects, I can certainly vouch for the importance of a test strategy and the value it provides to not just the test team but the product team at large in building a product of exceptional quality. It has vital elements such as testable points, team allocations and mapping with the rest of the product team, defect management guidelines, project risks and mitigation strategies, project exit criteria which are very important pieces that guide the test team overall.

One thing to keep in mind is that like any other evolving trend, a test strategy should also be altered to fit the changing needs without losing the underlying essence of what it has to offer to the team. One needs to understand the current scenario, the characteristics of an agile delivery model, and their project constraints in determining what needs to be retained and what can be compromised in building a strategy that is truly usable. For example, elements such as team allocations and mapping with the rest of the product team are not going to be as relevant as they used to, because this is a very dynamic piece in the current day development world. Instead this can become a living document retaining other relevant pieces such as project exit criteria, defect management guidelines and incorporate newer pieces such as team retrospective meetings, learnings specific to the current release that are key to incorporate moving forward.

In certain teams, I have seen them merge the test strategy and the test plan to create a single guiding document or a group of test plans retaining the important and relevant pieces of a test strategy but just doing away with the need to maintain a separate document. There are clear benefits of this model too, because you are incorporating the pieces you need into a document that is going to be used more often. Whatever it takes to understand and use the pieces of a strategy that will help you build and ship quality software is what really matters, at the end of the day.
In some ways, such a test strategy in an agile environment is more current, relevant and used than a traditional test strategy which was at times created with a lot of vigor and later forgotten. The current scenario provides the test management team an opportunity to revisit the strategy every few weeks, especially in a Sprint retrospective meeting, to gauge if the practices laid down really align with building a product of great quality or if any tweaks are needed.

This is where, I continue to believe that a test manager plays an important role in helping the test team adapt itself effectively to the changing product and market dynamics yet retain the original sanctity of the test effort. When such an open mindset is adopted, the team really understands strategy in its true value rather than treating it as a mere document that involves additional overhead in the creation and maintenance process.

The Future of Manual Testing

Rajini Padmanaban, Director of Engagement, Global Testing Services — Fri, 30 Mar 2012 23:41:10 +0000

Everyone in the software industry would agree that testing and quality assurance have become an imperative part of the software development life cycle in ensuring the release of a high quality product. Manual testing is an indispensable piece in this overall mix. To look at the future of manual testing, one has to first consider the history of testing to see how it has evolved.

In the 1990s, testing was just beginning to take shape as a discipline of its own. Not all products were tested before release leading to a number of quality issues from the field. Then came in a phase, when developers themselves tested the code they wrote which although was an improvement, still had a lot of issues since they were not able to do justice testing their own code. A lot of issues were missed in the process. To address these shortcomings, the field of independent testing emerged; it was initially largely manual and black box in nature, where the tester would test the product from an external user stand point. While they brought in a lot of value from an end user stand point, a complete black box focus was not sufficient to help them understand defects and issues from an end to end workflow. Their testing was very superficial and incomplete because a lot of issues often existed at an internal level, which they were not aware of. All of this led to missed issues, testers who were disconnected from the rest of the development team, testers unable to hold meaningful conversations with the developers, thus impacting the reputation of the test team. Until this time, most testing was manual with minimal automated testing which was primarily record and play focused. This was not necessarily the best automation strategy, but certainly helped reduce the manual work around repetitive tests.

Then came in a phase in early 2000’s where testers started delving more into the gray and white box areas of testing which provided a much need face lift to the discipline as a whole. They were able to add a lot of value understanding the system internals and coupling automation with manual testing to provide the required test coverage. Automation also evolved into more robust and maintainable models including framework based approaches which truly helped save time for the tester to focus on areas where their competency and attention were needed the most. Testers started enjoying the well deserved respect the profession was beginning to command along with shrinking deltas in their salaries compared to their development counterparts.

In the years that followed the industry was gradually moving into a phase where automation was the in-thing. In my opinion this was driven by a couple of core reasons. One because, most projects were beginning to adopt an agile development model, where time was a huge constraint and automation really came in handy to save test execution time. Second, the testing community was getting split in a sense, where automation testers were commanding more respect and the status of intellectually smarter than their manual testing colleagues. These together pushed automation to a prime spot and tried leaving manual testing to the more junior testers with not much experience. While some of this was inevitable because web based products started gaining prominence involving a lot of B2B communication (touching upon several areas such as web services, complex databases where manual testing was not possible or was impractical), there were still several areas where a manual tester could specialize. Such a specialized focus in manual testing was getting over-shadowed by the undue importance on test automation.

Current times are when we are beginning to witness a change again, where the industry is beginning to specialize in areas where manual testing will re-gain focus. These span both new technologies that are evolving as well as new areas of testing focus. For e.g. a lot of gesture based technology and touch technology are seeping in (some well known products in the current day include iPad, Kindle, Kinnect etc). Data input in such cases via automation is a challenge. Also, manual testing is becoming more prominent and meaningful in areas such as accessibility and usability testing where the tester needs to simulate end user actions which are not always automatable. Consider areas such as verifying that the alt text for an image has been set correctly, which is impossible to automate as of today. A couple of other relevant blogs I would like to point you to, in this context are: Keeping pace with the changes in the computing world and No one left behind.

One needs to understand that, test automation while very important and inevitable in ensuring product quality within the constraints and complex scale of operations that we work within, is only a piece of code and set of instructions. It does not have intelligence of its own and like a software program will merely follow instructions that the tester has mandated. It cannot replace the intelligence of the manual tester, get a pulse of the end user needs and wants. The years ahead are especially exciting with the specialized test opportunities that are going to be available to the manual tester. Manual testing will come back to lime-light and enjoy its own respect alongside with test automation. The test manager plays an especially important role in these critical years to nurture manual and automated testing arriving at the right balance and focus needed for both. S/He needs to work on building a set of strong test engineers in both areas, offerings ample room for career specialization and more importantly creating a balance in the demand and supply of both these areas in ensuring a product of exceptional quality in the market place.

Challenges of Multi Device Testing

Rajini Padmanaban, Director of Engagement, Global Testing Services — Fri, 16 Mar 2012 17:48:58 +0000

With the growing end user computation options, compatibility testing is becoming a very important, yet challenging area in the last few years. Personal computers, laptops, MACs, tablets, smart phones are all becoming more prevalent; especially, mobile computing options are flooding the market place. Amidst all of this changing hardware and its associated software landscape, there is an increasing need for software applications to be omnipresent in all these options. This is not an easy scenario to handle from a software testing angle though, because of the number of combinations in which the application needs to be verified, in the limited resources and time on hand. I will be talking about the testing challenges that exist today, taking a specific example of testing for Android applications, and also discussing some mitigation strategies. The Android example used is just to help with better explanation; the larger point to remember is how these discussion points apply to your specific compatibility test scenarios and how to customize some of these solutions to help serve your requirements. Also to keep in mind is the scale of testing, as Android alone brings in so many variations to the table.

Problem statement:
The scale of Android market penetration makes it difficult and impractical, if not impossible, to execute all tests across all combinations. The application that runs on an Android device could run on any platform, firmware, phone. If you were to test for all of these, you would be testing on over 100 different phones with 7 Android platform versions (1.5, 1.6, 2.1, 2.2, 2.3, 3.0 and 3.1) and approximately 2 vendor firm-wares. Thus, about 1400 combinations need to be tested, if you need to be foolproof. However, often times it is only a subset of these combinations and tests that yield maximum returns in terms of bugs found. How do you really determine the best set of combinations to test on, maximizing bugs found and minimizing risks, as much as possible?


Solution – To identify the appropriate device factors:The crucial element to restrict the list of devices is to recognize the factors that might affect the application and make sure to cover all possible combinations of such factors. Dealing with factors instead of devices will empower you to scale down the list of covered devices to a manageable subset that will provide coverage for a large portion of relevant scenarios.

Listed below are some such factors to keep in mind in determining the optimization matrix:

• Screen and its size: Smart phones or tablet applications are typically developed not for GUI (Graphical User Interface) but for NUI (Natural User Interface), which involves multiple touch gestures such as scrolling, tap to click, shaking, pinching and providing other finger-touch inputs. The main concern is the screen size of the devices, on which the application will run. One of the challenges of framing an application is how to frame a view layout so it will furnish the screen size. The screen size can differ from QVGA (240×320), WVGA (480×800), HVGA (320×480), FWVGA (480×854), in tablets (1024×600) and standard for tablets (1280×800). So one of the challenges is to run the same application on 240×320 and also on 1280×800

• Android OS versions: The android platform is rapidly evolving. The diversification from version 1.5 to version 3.1 is huge. There are lots of features that can affect the application under test

• Mobile processor: Mobile devices are very sensitive to processing power. Devices could run at single core at 600 MHz or dual core at 1200 MHz

Strategies to mitigate the risk factors:

• Edge: Select those factors that are at the edges of the scale, such as minimum and maximum screen size, OS versions at the lowest and highest end with minimum and maximum CPU power.
We can consider 2 devices when we combine influencing factors:
Lower scale: HTC Hero: Android version 1.5, HVGA (320×480) screen size and 528 MHz Qualcomm CPU
Higher scale: Galaxy Tab 10.1v: Android version 3.1, 1280×800 screen size and 1000MHz dual core

• Commonality: Keep tab on market device penetration and usage patterns to select the most popular devices. Such cases alone will give you the most bang for the buck and will help make your prioritization task easier. Sites such as http://marketshare.hitslink.com/ help determine the market penetration rates

• Simultaneous execution of test cases: Once a matrix has been finalized, as much as possible have testers run the same test across combinations simultaneously. This not only helps knock off the test case at one shot, but also helps the tester evaluate and compare application performance across variables at the same time; this often helps to get better feedback than if the tests were run at separate times, and also makes the test setup process more efficient

• Testing across combinations: Encourage testers to test across combinations, even when they are using the application in an ad-hoc fashion. Encourage cross discipline testers, who may not be directly working on your project, especially when you are testing a consumer application. The more the coverage you get through such formal and informal methods, the better are your chances of eliciting realistic end user feedback before release

With years to come, computing options are only going to increase. You will need to devise a strategy that works the best for your set of constraints and revisit it with every release to ensure it will continue to help you find relevant and the most important defects in the shortest possible time. The list above is not an exhaustive set, but an attempt to explain the gravity of the problem at hand and some of the core solutions that can immediately come in handy. Some of the core thoughts in this blog are from Chandni Sharma, one of our QA engineers, which I have expanded on.

Productively using a Tester’s Downtime

Rajini Padmanaban, Director of Engagement, Global Testing Services — Mon, 27 Feb 2012 20:52:44 +0000

With the move into a more agile development cycle in most projects, downtimes for a tester are really few and far between. That said it is always good to have a ready checklist of proactive things to take on, if such times were to arise. The list provided below is by no means an exhaustive set, rather a good set to start off with and grow into over the course of time, subject to your own project’s constraints and more importantly the amount of downtime you have

• Test case maintenance
• Regression test suite analysis and maintenance
• Compatibility matrix optimization
• Product competitive analysis
• Bug bashes across teams
• Break-out and brainstorming sessions on test, test process, technology etc. within test team
• Productivity tools and framework development
• Monitoring user forums for feedback on your product and other similar products
• Design test cases for valid ad-hoc bugs
• Voluntary code reviews if that is not one of your core existing tasks
• Research on current prevailing test and technology trends relevant to your product and domain
• Any demos or scripts or automation scripts you can prepare to benefit the rest of your team

Such a checklist when established and used at a team level encourages even the junior level tester to think big and out of box helping them understand the role they play towards the overall product quality rather than merely following instructions to test the module assigned to them. When senior management lays down such an open but wide forum to work within, it encourages transparency and creativity promoting respect for the test discipline as a whole while productively using the downtime at hand.

The Role of Test Data in Catching Bugs

Rajini Padmanaban, Director of Engagement, Global Testing Services — Wed, 15 Feb 2012 19:35:09 +0000

At our company, we encourage our employees to write blogs on test and technical topics whenever they can, to share their knowledge and experience. I am one of the reviewers of these articles, so I get an opportunity to read a diverse set of topics written by people of varying experiences. I read one on “Effective Test Data Preparation” this morning which re-kindled my passion for this space forcing me to share my additional thoughts over and above what the author had already penned.

The author, Vivek, in his write up talks about the need for effective test data, re-using test data over time ensuring they have not been corrupted and how test data should take into account: positive, negative, blank and boundary values to ensure they adequately test the application. All valid points that the author makes.

In addition to all of these, in my opinion it is very important to focus on your test data generation, depending on the type of testing undertaken. If you see, for functional testing you want a variety of test data covering all the points Vivek has mentioned above. For performance testing too, the same will hold good, but when you scale and increase your load, you would want to ensure that you maintain diversity in your data even when you deal with volumes. For security testing, focus more on the negative data sets or rather illegal ones which you would not expect your regular users to use. Scripts, SQL statements are largely used here like how a hacker would, to gain access to the internals of the application. In case of localization testing, the tester is focusing on creating real time locale specific inputs for which he might have to use translator tools. The tester will need to pay special attention around the currency, date/time formats, text length, etc. for the test data that he creates here. Prior to localization testing, when globalization and pseudo localization testing are done; the tester is focused more on creating gibberish or garbage data which is non-English to start off with. In cases of accessibility and usability testing, the tester is more focused on using tools like screen readers, narrators rather than focusing a lot on actual test data. For data base testing scenarios, you may not only need test data but also have them populated into the database, for ease of access. So, if you try to see a pattern here, you are really customizing your test data depending on the type of testing you undertake with a goal of finding as many bugs as possible. Since test data creation takes a lot of time, it is a good idea to use tools wherever possible with ofcourse due diligence done by the tester to ensure the validity of such automated test data that is created. Especially for scenarios such as user account and password creations where typically a huge volume of data is needed, a lot of freeware tools are available.

For every test pass, it would help to take care of the following checklist:
1. Understand the focus of the test pass and decide what tests are required
2. Evaluate to see what existing data can be reused
3. Ensure validity of such existing data
4. Analyze gaps for any new data that needs to be created
5. Analyze the data sources (whether data exists in XMLs, local files, data bases, reside in another application etc.) and see if any migrations are needed

Like test case maintenance effort, test data maintenance is a good practice to take on between test passes. The tester can look at any new tools in the market that can be used, any test data clean up that is needed, any new data that can be created, any user inputs from the field / bugs they have filed for which test data is very unique that you can add it to your data repertoire etc. Such pro-active tasks go a long way in saving you time and effort in your subsequent test passes and more importantly help you find realistic and good bugs. At certain times, you as a tester will also be required to share your test data with other teams in the product group including your development team to help them reproduce issues. Similarly when bug bashes are held, to save time for everyone, along with the basic instructions some test data might also come in handy for the group to use. That said, you need to draw your balance here in how much test data you share with your team, because the test data goes a long way in finding some core bugs; so you would not want to impact the team’s creativity and limit the bugs they find by sharing all of your test data.

So, chalk yourself a plan on how and what kind of data you want to generate, use and share with a goal of enhancing your product quality and containing defects as much as you can before product release. Happy testing!

Keeping pace with the changes in the computing world

Rajini Padmanaban, Director of Engagement, Global Testing Services — Thu, 19 Jan 2012 19:17:20 +0000

The computing world has come a long way and in the more recent years, a lot of buzz exists around mobile computing and the cloud. While the underlying concepts of testing and quality assurance remain the same with all of these changes in the technology and computing world, the team teams have to nevertheless empower themselves with the required skill sets, tools and frameworks to test the up and coming products adequately, leveraging the latest in technology.

One the same note, I happened to read recently about a couple of new trends: one around gesture based technology and one around the semantic web, which sounded interesting and promising, hence this brief blog. To give a very high level view into what these are:

Gesture based technology – revolves around techniques and ways to capture the user’s gestures and incorporate them into relevant actions

Semantic Web – revolves around smartly using the user data that is available in abundance to draw inferences and relationships, to make the computing environment carry out intelligent tasks for the user rather than merely following a set of instructions

Experts are betting on both these as some of the core trends in the computing technology over the next decade or two, which you will see are largely user-centric. And if things such as semantic web, take off well, the possibilities are endless. It can come in handy in almost every possible discipline in a variety of ways.

While all of this is great and we are excited to see them unfold, such complex evolutions pose their own challenges to the testing world. Let’s look at them one by one:

Gesture based technology – Feeding realistic user inputs into a test is always a key in maximizing the value of a test in finding valid bugs. We’ve had various sources of user input over the last decade including the traditional keyboard/mouse combination, from a console, from various touch interfaces etc. Test automation has evolved from the initial stages of a record and play framework to accommodate such changes. Now, with gesture based technology it only gets all the more challenging. How will a test recognize the user’s gestures and if it were to be designed the same way the programmatic interface has been designed to understand gestures, how will the tester ensure the validity of such gestures in ensuring the test data is accurate? The answer is not completely known as of today, but there will probably have to be some extra manual verification done at the automation testing phase to ensure the automated test has interpreted the gestures correctly. Some external factors such as multiple gestures, very quick, very slow gestures may also have to be tried to incorporate boundary conditions. Probably some amount of record and play will seep back into this futuristic test automation strategy

Semantic Web – The data available to process here is abundant. The tester has to smartly accommodate various data sources, and also create realistic test data is ensuring the program correctly interprets the various sources of information. In addition to test automation to verify the semantic protocols, some amount of manual intervention is going to be absolutely necessary to bring in the human element in the verification process. This may sound ironic, because semantic web is all about the program smartly assisting the user in carrying out his/her tasks without having to strictly implement what is has been told to do, whereas for verifying it, my take is that the human element cannot be completely done away with. Several new tools including parsers to read basic semantic resource description frameworks, browser plug-ins, validators etc. will enter the market to aid in semantic web testing. These in combination with enhancements to test automation frameworks and the core manual testing will make it possible for the test teams to test for this important evolution

At this point, as you see, some of the above points may not still be black and white. The important thing here is to be aware of these trends, keep track of them and invest in R&D in your testing labs upfront, around these areas, rather than having to play “catch-up” down the line.

Content Digitization QA – Do it the Smart Way

Rajini Padmanaban, Director of Engagement, Global Testing Services — Fri, 18 Nov 2011 20:30:18 +0000

I had blogged sometime back on whether or not Content QA is required. While the general consensus is that it is needed, it often gets a little challenging to tackle because of some of these core reasons:

1. Voluminous content to be handled during digitization
2. Lack of time to focus on testing towards end of release, when product functionality testing takes front seat
3. Complex specifications to test
4. Compatibility to be verified across a wide range of platforms and devices

In this blog I am going to specifically talk about doing Content Digitization QA the smart way trying to handle as many issues as possible upfront before content hits the application’s UI. Explained in very simple terms, the content digitization process can be represented as below:

To tackle the challenges outlined above the “Content extracted into XML” is a very important stage to get a lot of testing done. Issues caught here and resolved before content hits the application UI saves a lot of time and money in the overall testing process. There are going to be some tests though which can be done only after the final ingestion happens (such as the overall fit and finish of the content on a specific device). But several tests such as content format and meta data checks and associated parameters can be done at the pre-ingestion stage. At both the pre-ingestion and post-ingestion stages, smart ways need to be adopted to identify potential areas of automation to save time and handle voluminous XML files without missing any checks.

We at QA InfoTech have built our own scripts along with an application that consumes those scripts to offer a neat UI to take on pre-ingestion testing. This has been customized with keywords and checks specific to the publishing and educational domains which are where a lot of digitization happens. This application and scripts can easily be customized for pre-ingestion testing for any domain, with minimal effort. We’ve built this on PhP making it simple to use and reporting issues down to the line number at which they occur to ease the debugging process. We’ve used these scripts for several of our publishing clients and the expertise that we’ve built help us guarantee a turnaround time of 24 hours for such pre-ingestion testing from the time, we have a new file to test (regardless of how voluminous the XML is).
We have our test automation framework built on open source technologies with built in virtualization, pluggable utilities for continuous integration and reporting, to address test automation at the post ingestion / application UI level. For more details on this framework, listen to our webinar hosted here. Seen below is a pictorial representation of our overall Content Digitization QA process. We soon plan to do a webinar explaining our content digitization automation solution in detail. Stay tuned for more updates. In the mean time, if you have any questions/need more information, please reach out to me at rajini.padmanaban@qainfotech.net

Testing your Test Code

Rajini Padmanaban, Director of Engagement, Global Testing Services — Thu, 10 Nov 2011 02:01:11 +0000

In the world of software testing and quality assurance, we all know the value Test Automation brings in, to improve test coverage, overall product quality and the tester’s productivity. But all of this value flows in and the ROI is reaped only when the automation code is robust and reliable and can produce consistent results to catch product bugs. The term “product bugs” is very important here. If the automation code does not catch bugs (if this is truly because the product has reached a steady state and is largely bug free, then it is acceptable) or shows more false negatives (due to test code issues rather than product issues), there will be a lot of wasted effort including:

• Test Automation Effort – code design, implementation and maintenance resulting in wasted time, cost and human resources
• Triage time – involving the product team to look into the invalid bugs reported resulting in expended time, cost and human resources and more importantly the reputation of the test team
• Resource Usage – Machine, other infrastructure and software usage for automation execution.

One of the traditional solutions adopted to solve this problem and make the test code reliable and robust is to “test” the test code itself, as if it were product code. While this sounds great in theory, are all teams really practicing it? Also, it is all the more important to carefully chose a smart strategy and decide “How much to test” your test code because your goal is very different in this test effort compared to product testing goals. Any extra test effort here is again going to lead to a lot of wasted resources and is not going to add any value. Choosing what areas to test, how to test, and agreeing upon your goals of testing the test code are vital. Here are some best practices on “Testing your test code” which will help you make these important decisions.

1. Static tests around code reviews are very important. Testing your code to ensure coding standards are adopted is very important to ensure automation maintenance effort is minimal down the road. In such reviews check for modular test design, including things like reusable methods, decoupled test data to ensure code is clean and easy to work with.

2. Your test code’s functionality testing is what is most important – specifically focus on whether your code is testing what is intended. It would be good to pick a few core tests and run them both manually and through automation to ensure consistent results are produced

a. Focus more on unit testing your code. Leverage simple unit testing tools such as NUnit, JUnit etc. coupled with manual testing efforts using simple checklists to achieve your goal without making this process a complicated one

b. Testing the code that you write to verify builds (Build Verification Tests) in detail is very important because this is an area where you will use your test code very frequently. If build breaks happen due to test code issues it is not only a lot of wasted overhead but also adversely impacts the visibility of the test team within the overall product team

3. Before you actively start using your test automation code and share it with the product team, carry on a few trial runs where you execute the code on a few builds (if required use even local builds) to ensure the code yields consistent results (e.g. consistent passes and fails between builds) and closely evaluate the kind of failures returned to ensure they are indeed product bugs

4. Schedule your code to run in sequence a few times (e.g. 10-15 times) over a day in an unattended mode to ensure there are no built in dependencies which affect the test runs. E.g. access issues, any versioning issues, any software / hardware dependencies etc. Also, while a few initial trial runs could be on local environments, get your test code deployed and run against the actual test environment (at least on a trial basis) as soon as possible as this is what will reveal any other deployment and cross module dependencies

5. Code coverage – Running your test code with code coverage tracking on, is a very good test to determine the coverage achieved. This is a good objective evaluation of your test automation’s ROI also helping you to find out areas where you can beef up your automation and areas of product code that need to be removed

6. Don’t focus as much on areas such as performance, security, UI, usability, globalization, compatibility etc. while testing your automation code. In fact, you can almost eliminate a few of these, such as globalization out of the mix to purely focus on the functional aspects of your test code

7. Beware of any tool/framework limitations to ensure they don’t impact your test results including any support clauses to help build your automation on a sustainable platform

8. Test for your test code’s reporting capabilities – This is one area where you can focus on UI and usability aspects since your test code results are often shared with various levels in the management all the way from even the executive management down to the grass root level. Test for your test code’s accuracy, level of detail, archiving capabilities and usability aspects when you test the reporting module’s code

9. Set clear exit criteria – Similar to product testing, you will need to set clear exit criteria to sign off on your test automation code and move it to production (where you start using the code for actual product testing). Exit criteria need to be very stringent (I would even say 100% reliability at all times) in this case to ensure test automation code is reliable and can be leveraged to test the product. If specific scenarios/tests are delaying your process, de-activate them, and move them to a queue to be analyzed later. I say this because although the test code needs to be reliable, the team should also understand that testing the test code is only a supplemental activity and that the larger goal is really to leverage this and focus efforts on actual product testing. So, the test team has to lay down its priorities right in expending the required amount of time to test and strengthen the test code while still focusing on the quality assurance of the final end user product

Try these out and do share any other tips/inputs you have around “Testing” your test code. I would love to hear what you have to say!