For those of you who do not have much background into our history, we are a 7 year old company, who have grown in terms of numbers from less than 5 people to over 350 people in 5 testing centers of excellence. While I will not touch upon the technical expertise the organization has acquired and built over these years, the one aspect I will briefly delve into is “our bonding with our people” - what we consider our “ingredient to our receipe for success”. From inception, QAIT management has emphasized on building a strong work culture and making the organization an enjoyable and enriching place to work, for every employee. This has been implemented both:

       top-down with a very apporachable, transparent and supportive management in place and 
       bottom-up with every employee being given personal attention on what they work on, where they aspire to be in their careers and helping them build it successfully at QAIT

It is very heartening to see these practices of ours being reflected in this year’s Great Places to Work for in India, Top 100 results list, in which we have been listed at number 62. For a company of our scale, we (our management and employees) take pride in having reached this level and this is an added motivation for us to keep working towards rising up this ladder. As we announce this important news to you all, we would also appreciate if you can take a moment to read a blog written by one of our test engineers on the work culture at QAIT. This is a true grassroot level reflection of what it is like to work for and at QAIT, which we believe represents the voice of every QAITian. Please find this blog at: http://www.qainfotech.com/blog/2010/06/work-culture-at-qa-infotech/

1.      Number of defects missed in a release: This is a very important indicator of the team’s performance. That said, this is a more reactive indicator where this data can only be used for a postmortem analysis of that release and learnings implemented in the next release. However, given that this is based on bugs found post release, by various users of the product, this is a very objective data point on the team’s performance, which will greatly help improve the test coverage and team’s effectiveness in the next release. Another metric which is in a similar category, but at least gives the test management team some time to act before a product release is “bugs found in a bug bash around product release, which should have been found sooner”. This metric at least helps decide whether or not the test team is ready to sign off on a product release, giving some opportunity to be proactive before the product reaches the hands of the end users.

2.      Number & Type of defects reported by the test team vs. reported by other disciplines: Although, a test team is the one primarily chartered with testing the product and reporting defects, several other teams associated with the product such as development, marketing, program & project management, cross groups associated with the product etc. will also have an opportunity to use the product and report issues over the course of the product life cycle (PDLC). These numbers when tracked at a periodic frequency over the PDLC, is a very useful proactive metric for the test management to identify coverage or effectiveness or areas of focus loopholes, in the test effort and fix them right away.

3.      % of valid and invalid bugs: Consider not just fixed bugs in your valid count. Including bugs resolved as “external”, “postponed or won’t fix” is equally important since these are still valid bugs. Consider bugs resolved as “by design”, “not reproducible”, “duplicate” for the invalid count, since they often show that the test team has not understood the product well enough, has not done its home work well enough before filing the bug all of which just lead to a waste of time and effort for several people on the team who are looking at these bugs. A periodic analysis of this metric, in fact, I would say on a monthly basis is an excellent indicator of how the test team is performing. A more granular analysis can even be done on specific testers if you want to work with him/her to improve individual performance. This is a good proactive metric if used regularly to help improve the test team’s effort.

4.      Average time to defect finding and defect closure: Time to defect finding, shows how effective the test team has been in finding a bug soon after it was introduced. The sooner it is found, the less costly it is to fix the defect, making the overall cost of the product lower. Similarly, tracking the time the team takes to regress bugs shows how quickly the test team is responding to resolved bugs. The sooner they can resolve them, the faster will be the process to catch any regressions. Both these are proactive metrics a test manager can use to determine how effective the team is and fix any areas of lacunae.

In the above blog, I’ve discussed both proactive and reactive metrics that can be used at varied stages in the PDLC, which when tracked effectively and used for the team’s improvement as well as representation to senior management on how the test team is performing, go a long way in establishing a strong base for the test team in the overall product team’s positioning.

A consultative role does not always mean something high end. It could be as simple as sharing your suggestions with your customer contact when your task is allocated. E.g. does it make sense to run these “x” test cases at this stage in the product life cycle, is the chosen OS Browser matrix the right and optimized set to run a test pass on, is a bug’s resolution indeed the right one etc.

On the other end of the spectrum you could be hired as a vendor to provide QA consulting as a service and to define the QA strategy a company needs to adopt. No doubt, you will play a consultative role in that situation. But when you start playing such a role in your day-to-day activities as previously defined, it just begins to show your passion and commitment to the task at hand, drive to do things right,  depth in understanding of both the product and QA process and increases the customer’s overall confidence in your critical thinking, analysis and execution capabilities.

Exercise the right balance though! Your consultative role is not to question and negate the customer’s thought process, but to question it when indeed needed for the overall benefit of your project and product quality. Try practicing this on a day-to-day basis and you will see the positive zeal this brings in, into even some of your mundane tasks.

In this blog post, I would like to extend this idea to companies taking a social responsibility in contributing back to the community that they belong to. As part of its day to day operations, every company uses a plethora of software and hardware ranging all the way from servers, desktop machines, server management software, Operating systems, and productivity software, to just name the bare minimum. Obviously, all these products have been tested to meet a certain quality bar before their release. What I want to discuss in this blog is how can you as an end user of these range of products help the software/hardware vendor improve quality, even after product release, when there is no mandate on you to do so and what is the impact you create by doing such a thing?

Does this activity qualify under exploratory testing? Crowd sourcing? May be; but let’s shift our attention to the “how is this done”, “what is the impact this creates”?

Potential ways of contributing:

1. Participate in beta programs run by ISVs, IHVs

2. Discuss with your team on what is helpful, what is painful to use, when a new software, hardware is deployed in your company

3. Active participation in technical groups and forums that discuss the product’s features. E.g. Linux forums, Windows forums, MAC forums

4. If the company whose product you are using is small enough, see if you can directly reach out to them with your feedback; or even if it a very large company, see if you have some channels through which you can route feedback to the right teams

Impact you create through such contributions:

1. An opportunity for your team to think out of the box and be motivated on the job rather than just having them follow project related instructions provided to them

2. Power users of the product which help you take full advantage of the product’s feature set to improve your daily productivity

3. A more open community promoting quality and taking interest in the products you use on a day to day basis. Very often, such user feedback form the foundation of a subsequent service pack or a patch for the product

4. A status of a technology thought leader for yourself and good will of the product vendor which can sometimes even result in partnerships, new business opportunities with them

Where to draw the line?

Having talked about how to be a socially responsible technology consumer, it goes without saying this does not come for free. It does require cycles from your team to do all of this. Given their tight project deadlines, where do you draw a line of balance for these voluntary and nice-to-do tasks?

1. In the interest of time and overhead involved, you need not be very detailed in your initial communications with product vendors. Show them a few teasers and if they show interest in your feedback, you can selectively be more detailed to such companies alone

2. See if you have people on bench that can be utilized for these tasks. If resource utilization in your company is very high, you may have to compromise on these activities a little

3. Look at your past history with the company on how they have responded to your feedback and whether the product’s subsequent versions incorporated feedback you provided

As some of you may know, I am quite new to the QA InfoTech family. It is just about 3 months since I joined this company. During my ramp up cycles in this company, I was very pleasantly surprised to see some of the initiatives being taken at QAInfoTech along the lines described above, despite being a small company. I have outlined a few of them below:

1. Beta users of a system management software being released by a global premier ISV

2. Proactive efforts in testing Win 7 localized builds and working with Microsoft in sharing the feedback with detailed repro steps

3. Proactive testing of a Commercial Test Automation Software, which helped us bag a testing opportunity with them

If there are any other similar initiatives taken up by your company, from which we can all learn to make us more socially connected and help improve end product experience, I look forward to hearing from you. As always, I would love to hear any thoughts and feedback you may have on the above content. Thanks for taking the time to read!

It is often believed that only big companies, of a certain size and revenue can afford to spend on R&D and sustain it as a separate unit within the organization. While there is some truth to this, I do not completely buy into this myth. What I do agree with, is that, not every company will have the required financial and man power to run R&D as an independent unit. In my opinion every company should inculcate the R&D focus in each and every employee and integrate R&D into core engineering– the way you do would have to be based on your scale of operations. Let us look at three common scenarios, into which almost all companies would fit into, one way or another:

· Small companies (e.g. <1000 people globally, <50M USD in revenues)

· Mid-sized companies (e.g. between 1001 and 10000 people, 50-500M USD in revenues)

· Large companies (e.g. >10000 people, >500M USD in revenues)

The challenges are varied in each of the companies. In small companies, how do you financially justify R&D is the question at hand; in large companies, creating awareness for the R&D efforts, convincing core product teams to use the company’s IP and bridging the gap between the two teams is challenging. Here are some thoughts on making the overall R&D effort successful, as once there is buy in from everyone on this effort, there is no stopping in setting yourself as a thought leader in the industry; once this is done, revenues will automatically flow in, the right talent would knock on your company’s doors all of which will help you grow in your evolution lifecycle from a small to mid size to a large company.

Small companies: Premise: Do not have money or people to devote to a R&D wing. Employees themselves constitute to R&D

· Create awareness about the need, importance, value of R&D

· Encourage employees to look for IP that can be built even as they work on their day to day projects…IP that can help build reusability, enhance the functionality of open source tools they are working on

· Productively use any down time in understanding market, business, economic, technology trends and looking for potential R&D opportunities

· Have periodic forums to discuss ideas, trends, share knowledge, demonstrate any frameworks in progress etc.

· Pick one customer who will sponsor you to try your R&D efforts on their project

Mid Size companies: Premise: Have the horsepower to maintain an independent R&D wing, but will need to be flexible to multitask. While in the previous scenario, employees on projects had to take on R&D as part of their overall routine, in this scenario the tables often turn, where R&D engineers might have to be deployed on projects occasionally to ensure pilots succeed and you win the customer’s confidence about your R&D capabilities

· Pick right talent to build your R&D team – people who are promising and have shown potential

· Focus on the right strategies for R&D given the scale of your company. You may have to compromise on some long term investments in favor of ones that have a faster ROI, to justify this team’s existence

· Pick the right projects on which you need to deploy R&D engineers for hands on project execution – strike a balance here between pilot projects, customer win phase etc. that you need them to work on vs. losing their time on hands on R&D work

· Do not omit the best practices recommended in the “small companies” scenario, once you become a mid size company

Large Companies: Premise: Existence of a full blown sustained R&D team

· Reach out to various groups in the company to build a strong relationship with them. Look into possibilities of aligning yourself with the various groups by creating R&D sub groups with specialized focus

· Conduct periodic trainings, demos to create awareness of what you do and help build competencies of employees in the company

· Conduct periodic discussions with project managers and leaders so as to recommend better engineering practices, technological and productivity boosters etc. Use this opportunity to also understand their pain areas, customer needs, to look for areas to invest in

· Do not ignore your best practices from the small and mid size companies, once you hit the status of a large company

Obviously, we as a company driven by innovation, thought leadership and best practices, strive to practice what we discuss in our blogs. Rather, most of our blog’s content stems from what we’ve learnt on the ground. As a small and growing company, we are very cognizant of the R&D challenges discussed above. We do not neglect them; we rather embrace them to work towards a solution that involves our people. We have created ample awareness of the need for R&D within the company through various forums such as technical contests, Saturday open room discussions, encouraging employees to share their ideas, look for alternatives to turn project roadblocks into new R&D propositions, and most importantly showing management support in acknowledging and investing in such inputs that come in. We have some very technically involved proprietary frameworks that have evolved as a result of such investments, in areas of test automation and performance testing. These have helped us improve productivity for our clients manifold, make our solutions technically feasible and cost effective and make the work place a fun and challenging environment for our employees

Hope the above blog has thrown light that investment in R&D is good to be mandated in all companies. Just how much to invest, where to invest and how to reap the benefits are the ones to be customized. Try out some of these customizations; you will not go wrong; you will only leap forward in the right direction. Please keep me posted with your comments, feedback and questions.

Software as a Service (SaaS) has been an evolving trend we’ve been seeing over the last 2 years or so. Its increasing popularity can be attributed to several reasons such as: evolving cloud technology, need for optimization of resources and on-demand usage, global expansion of business that necessitates a platform to bring its people together,  to name a few.

In this blog, I will be talking about how testing for SaaS applications is different from testing a traditional on premise implementation or rather what more to additionally focus on, in SaaS applications. To do this, let us look at a simple definition of SaaS and derive testable elements from it.

SaaS is a software delivery model, where the software is made available to the end customers over the Internet. The software vendor hosts the software for the customer, without the customer having to build up a hardware or software environment at their end.  The customer herein, just provides software customization requirements to the vendor and follows a “pay per use model” to enable optimum use of resources. SaaS offers the customer the flexibility to scale up or down the resource usage on an “on-demand” basis. Depending on the SaaS implementation model, there are varying levels of SaaS model maturity ranging all the way from:

• Single instance - single tenant (customer)
• Customizable multi tenancy on the base source code
• Multi tenancy on the base source code without any customizations

Let us now draw the testable elements from this definition:

Take away #1 from definition: Software made available to end customers over the Internet: Software has come to enable us carry on a zillion different activities all the way from day to day shopping, banking, networking to possibly onetime events like home purchase, finding a match for yourself etc. If you are going to do all of this over the Internet, what is the first concern that comes to your mind? Privacy, confidentiality, security. Are all my transactions going to be secure especially when multiple other tenants are using the same instance of the software? Security testing, especially, web application security testing is very important for SaaS applications. Take the initiative of doing the threat model analysis, or if your development team owns it, actively participate to ensure you understand the potential threats so you can test them to ensure they are mitigated.

Take away #2 from definition: The software vendor hosts the software for the customer: Given the scale of operations involved in the SaaS model, performance is something to definitely plan for in advance and test.  This is something that the end customers have come to expect, however, this can become challenging for the vendor when multiple instances are being planned for.  Discuss the performance requirements with the business or product planning team upfront and start this testing early on in the product life cycle to conduct incremental tests. Besides just performance testing, scalability and capacity planning testing is also important in SaaS given that it works on an “on demand” model. Scalability and capacity planning does not always mean throwing in more hardware - doing so, may potentially result in wasted resources. Work with the development team to understand how scalable the product architecture is, to see if any optimizations can be done.  Start all these discussions early on, since architectural changes may be very expensive if not impossible to implement at later stages.

Take away #3 from definition: The customer herein just provides software customization requirements to the vendor and follows a “pay per use model” to enable optimum use of resources:Besides the core product functionality, two other important aspects to test in SaaS implementations are: right metering and billing and the right customer customizations.

One of the inherent benefits of the SaaS model is the pay-per-use feature. Unless this has been implemented correctly, there is no way to keep track of the customer’s accurate usage and this may affect the success of the entire application. Test for all the payment and usage algorithms, including boundary cases to ensure the implementation is correct.

Since often times the same application instance is customized and offered to meet the customer requirements, ensuring all of the customizations have been done correctly is very important. This may sound simple but any bug that is missed here, affects the customer’s reputation badly. Imagine having a logo of a competitor on your page - this could be disastrous. This is an area to be specially looked at, unlike in the traditional on-premise applications where this does not pose any risk.

SaaS applications, when implemented correctly and adequately tested for, have a lot of benefits to offer. Reap those benefits, to make your software a success by proactively planning for these above areas well in advance. Any feedback, comments, questions? - I’ll be happy to hear from you.