Crowd Sourced Testing – is it really for you?

Given the global distribution of software and how internet is bringing the world together, community based testing activities have been gaining a lot of momentum in the recent years. Such activities could be forum discussions, beta testing efforts, crowd sourced testing etc. Of specific interest in this blog is to see what crowd sourced testing is and when can this model be leveraged to yield success

In simple terms, crowd sourced testing is leveraging the community at large to test a given product. This is the community that spans people from diverse cultures, geographies, languages, walks of life who test the given software, putting the software to use under very realistic scenarios which a tester in the core test team may not be able to think of, given his / her limited bounds of operation. There are specific crowd sourcing companies such as that bring together crowd sourced testers and companies needing such testing to boot strap and carry on the overall project. Given the kind of bugs such test efforts result in, the short lead time within which the test effort can yield productive results and the reasonable costs associated (often times the product company pays only for valid bugs reported), one would think the Return on Investment (ROI) is very high and be tempted to go this route. Like any other area, crowd sourced testing is not risk free. It has some inherent risks to consider and mitigate, failing which the test effort may turn out to be a very random one, affecting the overall project and product cost, timeline and quality. I’ve delved into some of the core points below on when such an effort makes sense and when it does not, to adopt to make the most of a crowd sourced testing engagement.

When Crowd Sourced Testing is likely to work well?

·There is a global user base for the product under development

oProduct under development is more consumer centric rather than enterprise centric. E.g. Gaming, Mobile, web driven consumer products/applications, where global user feedback makes sense

·The product company is committed to working with a large group of people, understanding it involves some amount of overhead in such a decentralized test effort, rather than containing everything in house

·The product requires subject matter expertise in certain specialized areas and such expertise is wide spread. E.g.content testing in specific domains, language testing etc. for which gathering in-house expertise is expensive, if at all possible

·There is a need to simulate end user scenarios in testing – e.g. performance testing for the product needs to be done at different internet bandwidths and devices available in various countries

·The product team does not have time or resources to take on full-fledged testing in-house but has a good grasp of product requirements, test coverage to achieve and an overall strategy with which it can engage a globally sourced team

When Crowd Sourced Testing is not likely to work?

·Incorrect team is assembled with the wrong choice of testers, whose backgrounds do not best fit to test the product

·When the test effort is left to the community and the product test management fails to tie their efforts into the overall test strategy

·Lack of a clear strategy on what to crowd source and what to keep in house – e.g. some secure testing areas, automation (for both re-usability and to provide opportunities to your in house testers), are better off when retained in-house rather than given to the community

·A product that requires a lot of internal communication with cross groups to understand the interfaces; such high dependencies are usually difficult to get external testers involved with

Keeping the above points inmind and taking cues from your own scenario will help you determine whether crowd sourced testing really makes sense and if so what to, when and how to leverage from the crowd sourced community.

SocialTwist Tell-a-Friend
This entry was posted in Business, Testing Practices, Testing Trends. Bookmark the permalink.

One Response to Crowd Sourced Testing – is it really for you?

  1. chandan says:

    Great stuff, a lot of information for me I would like to add how research in this field could help.
    I had a little experience in crowd sourced testing probably a quarter before I attended one and I observed There are few factors that plays important role in finding trusted crowed for crowd sourced testing say for example Research. This phase include analysis of the previous record to prepare the data or to find out the information of crowd tester. For example last time when we have conducted a CST and as a result we find out five best testers among 200+ testers these were those who provided the most valid bug. In the same way if we will get data of previously conducted few CST say 5 previously conducted CST, so we can get the data and can find out nature of group ot tester. Nature of group mean who are the tester giving their best each time for example we have got “A” ,”B”,”C” etc named tester in last conducted CST now we can summarize their nature for next CST.So this is one of the case but as we know nature of group cannot be constant, what does this mean is what if “A” ,”B”,”C” etc named tester are not able to give their input effectively every time or they started appearing in bottom in next CST so in this scenario we need to find out the background of segment which mean is in this case we’ll try to find out what is the background of these tester who are appearing best every time for example whether they are the tester who belongs to CS or IT field or they are the tester who belongs to a particular university or generally they are the tester who belongs to IT hub Bangalore or Delhi or every time folks from USA or China or Japan prove themselves best . So on the basis of research to find out nature of group and back ground of segment study, we can find out the better conclusion.
    For maintaining the confidentiality

    We could work on:
    1.Privacy Policy.
    2.Personal Information of crowd
    3.Security and limited access to database.

Leave a Reply

Your email address will not be published. Required fields are marked *


Captcha Captcha Reload

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Life @ QA InfoTech

Follow us on



Service Offerings :