I was asked on an interview once: "How do you prove your product quality has improved"? Its such a great question. Mostly because we get so lost in the world of product development and engineering, test writing and bug validation, that we forget to make sure what we are doing is actually working. A basic underlying principle in QA, almost an unwritten understanding, is that simply by doing QA and by running QA methodologies, you are improving your product. That's true, but just because we know it, doesn't mean we don't have to prove it to management. And management needs to know what we are doing, in order to maintain and strengthen quality in the workplace. Or in other words, prove that what we are doing, actually works.
Another issue is how do we actually define "Quality Improvement". To each of the different groups in a software engineering machine, each one will have different goals. Product Managers will want to see high feature implementation, with high customer satisfaction using those features. Project Managers will want to see successful on time releases, with little to no negative repercussions. Development will likely want to see good development, mixed with solid based infrastructure able to support growing features, growing customer base, and new technologies. DevOps would also like to see a high performing product with the ability to scale out nicely. Upper Management would like to see a product that is hitting business targets, while maintaining a low bug count. Finally, the ones who really matter, customers. They want to see a good product, that both works how they want it to and is easy to use. (Those two items are not the same. You can have a product that does what they need and isn't easy to use, or you can have a product that doesn't do what you want it to, but is easy to use. The goal is to make both work).
In order to prove Quality Improvement, across all teams, metrics are the way to show it. There are many metrics you can use as part of an STR (Software Test Results) or any kind of post-sprint or post-release Report. Here are 5 metrics that will help prove that your product is improving in quality.
1. Post Production/ Customer Based Bugs Found Over Time, Per Severity
The concept of being able to track bugs that arrive from customers is not necessarily always implemented. I've seen several companies that did not know which bugs were coming from customers, and which were coming from Dev. Or they were mixed with Ops/Feature Requests and didn't differentiate. Rule #1 - tag bugs that come from customers. Make sure you are tracking them according to their severities. Then setup a graph of Bugs from customers, separated by Severity.
All bugs should decline over time, with Critical and High level bugs moving close to 0, while medium and low bugs maintain a decent low count.
2. Features Released/ Stories Completed
Presenting a graph of the amount of features you released over time, or the stories you have completed, will show how the development teams are progressing and getting work done. This will appease Product Management and Upper Management by showing directly the business side of the product taking a priority in development. In addition, add a metric to this one that shows Test Rates and Bug Rates for these stories, to impress upon the audience how QA has played a part in releasing these features with the proper quality needs.
3. Bug Tracking from Manual and Automated Tests
This is simply a metric that proves QA is filtering the garbage, and letting only the good stuff out. Over time, show the amounts of bugs that have been found by QA during testing, especially during releases. Put a special stress on bugs caught by Automation, to show that the main gates are not letting out bugs. These numbers should or can stay relatively static, or grow over time as features grow and the product gets bigger. There can also be spikes during hardenings or refactors in code. A decline in the numbers could either mean better gated check-ins, or a negative result from QA testing. Watch for these metrics, and be able to explain them and react to them. Whatever the numbers show, and whatever you can learn from them prove that work is being done and the QA Net is there to catch bugs that should not be there.
4. Story Point Completions / Sprint Result Conclusions
This one can be controversial, but I think its imperative to the QA Strategy. Show how many story points were completed for each sprint, over the last X sprints. Or, if you don't calculate story points, how many stories and tickets were completed in the last X sprints. Why? Because of two reasons. 1) it appeases the Dev needs to show work is being done, according to plan. 2) It helps the sprint plannings figure out how much time they really have for each sprint in Dev and QA work hours. The reason this proves Quality Improvement is because you will be able to identify all of the following issues:
- Which Dev teams are not planning correctly or are taking too long
- Which kinds of stories/tickets take up more time than others.
- Which kinds of stories require more testing than others
- How much a good planning really helps the Sprint Process work correctly.
A good process in your Agile Sprint, is core to getting quality improvement. If you use Agile Lifecycles, making it work right is key to Quality Management. So get your planning right, get your stories aligned, and you will see better incident management and over quality growth.
5. QA Test Coverage Overtime
Show your work! Create a chart that shows over time your increase of tests running in automation, or running in regression. Show how many tests on average are run per new feature. Show what your test execution rates and test pass rates are. Most importantly, show your test coverage. Be able to map out your test to product coverage, so you can calculate easily how many tests are being run against how many features are in the product. A high test execution rate means nothing if you have a low test coverage rate. So be sure to create that product map, and show how much of the full product is actually being tested on a regular basis. This will provide an excellent view of the quality of the product and how management can understand the incredible strength and overall improvement that QA brings to the organization.
Comments
Post a Comment