Quality metrics

Summary

Introduction

Quality

When dealing with quality, it is difficult to define a single index that measures it.

There are as many views on the topic as there are experts in the field. Nonetheless, it is interesting to get a single index that combines the various basics metrics. But this does not prevent from analyzing the trend on those basic metrics to understand what affects the quality index.

In addition, defining an achievable quality score is part of any test and release management strategy (e.g. our goal is to achieve over 50% quality on the version 1, and 78% quality on the version 2).

Remember that this is only a KPI, not true measure of the actual quality. This indeed should be managed as part of your production metrics (e.g. MTBF 3, MTTR 4, nb of patches per period etc.).

Still, being able to take an educated decision about the opportunity and the risk of pushing a new release to production is a must have for any organization.

Looking at the following diagram (Figure 1), we would be aware that pushing this release to production would be a risky business. Although the scope, specifiability and testability are fine, we can see that the quality is a lot behind. More efforts are needed on the execution of the tests, or we need to reduce the scope or we may even have to focus the next sprint on stabilizing this release. This is probably due to the fact that we increased the scope too soon ignoring the quality score. Very bad!


One way to correct it would probably have been to not increase the scope too soon so that the effort on test execution and test correction are more focused.


In some more complex cases, the scope will be quite dynamic and will impact immediately the specfiability, testability and quality curves. In such case, having a real-time quality score graph is a must have to undertake the right corrective actions as soon as possible to get a proper quality at the end of the sprint.