Using Testing Resources Most Effectively
Like everything else, software testing resources (i.e., time, people, machines) are always limited and never enough. The goal is to produce a high quality product. The challenge is how to most effectively use the available resources to achieve the goal.
One approach is to spend most of your testing resources on the code that is most often used by your customers.
The Pareto Principle for Testing
There is a general rule of thumb that your customers spend 80% of their time using only 20% of the features of your product. You should determine that 20% and test it to make it bulletproof. The trick is to identify that 20%, and then tune your tests appropriately.
Data Collection
The first step is to identify the most frequently used 20% of the product and what code that represents. Using our counting Performance Profiling tool, you can see exactly which parts of your code are executed the most.
Preparation
Use the tool to create a version of your source code that will count the number of times that each section of code is executed. Create a copy of your product as you normally would: compile and package it in an installer, for example.
Collection
Provide the instrumented version of your product to your QA group or a carefully selected set of customers. If you use your QA group for data collection, then they need representative Use Cases from your customers (and possibly some representative data). Have them execute all of the Use Cases.
If you send the product to a customer, let them use the product in their normal manner for some period of time such as a couple of days or a week. If the customer performs periodic acts such as monthly accounting, be sure to include those activities in your data collection.
Analysis
Once you have collected the data, you can analyze it with our Profile Display Tool and report. The display tool provides a graphical view of the program; the report is a textual view that is also available in XML format. You can now plainly see what parts of your code are used the most in regular use.
Verification
Now that you have this data, configure your testing to exercise the code that you have identified. How can you be sure that you are in fact testing the code identified by the Profiling Tool? You can use our Test (Code) Coverage tool. Similar to the Profiler tool, it instruments your code. You run your test suite on this code. Rather than an execution count, it provides data about whether or not a section of code has been executed by your test suite.
For more information, please contact [email protected].
