As the post ‘Tools Do Not Improve Quality’ by my colleague Nils already destroyed the naive dream of simply installing a tool and get all maintenance nightmare healed over night, I would like to address a follow-up question in this post: If tools are not sufficient, how can we improve quality then?

Several years ago when we still were blue-eyed researchers on software quality, we started out developing different kinds of quality analyses in the context of our tool ConQAT. We worked hard on eliminating false positives and achieved highly precise analyses at the end. We thought developers will love adressing each issue these tools emit. When we presented analysis results to developers, they usually agreed on the relevance of the deficits we identified. Thus, we integrated our tools into every nightly build we could find at our partners. However, our enthusiasm was suddenly dampened when looking at the quality trends recorded by the dashboards. Although the analyses worked great, the results were available on a daily basis and the developers were willing to produce high-quality code, the systems decayed. So we learned the hard lesson that tools alone definitely do not improve quality - and even the most motivated teams of developers will hardly stop quality decay as an incidental side-effect.

The chart above is taken from an industrial project driven by really motivated developers and shows a typical situation after introducing new quality analysis tooling. In July 2008 a decrease in the number of defects the tool identified in the system appeared. This was exactly the time the first ConQAT dashboard was introduced, several workshops to explain the analyses were held and explicit examples of bad findings were discussed. This awareness caused several rather uncoordinated clean-up activities. However, as the chart shows this was only a very shot-term effect that did not last for very long - a flash in the pan. Few months later the general trend of increasing quality defects continued as before.

Even today we frequently encounter developers that point to similar quality dashboards in their nightly build when asked about their quality assurance methods. Having a closer look at the dashboards usually uncovers that their content is widely ignored during the daily work. Thus, the tools solely serve as a fig leaf for the project when asked about their strategy to ensure internal quality.

So how to really achieve a lasting effect on a system’s quality? In many discussions with developers we realized that proposing deficits - even clear bugs - for clean-up, led to a reject with arguments like “this is a very old component - we will not touch anything in there”. So the challenge was that the teams feared the risks of unwanted side-effects when changing certain pieces of code. In many cases the developers creating these components were no more available. Usually there were no or very poor automatic tests available. Thus, we very often had to admit that it in deed would be a bad idea touching these pieces of code without a clear mandate from the management and problems reported in production by key users. On the other side of the scale we encountered developers that happily cleaned-up even minor deficits in their code simply because they were working on these components anyway. They were very familiar with the context of the deficits and had the security that their code will be tested in-depth before the next release because it implemented new or heavily modified functionality.

All in all we found out that the point in time a quality defect has been introduced into the system has a major impact on the willingness to remove them from the code-base. The clean-up of recently introduced deficits fits much better to the overall development context of a project, faces lower risks of introducing additional bugs and comes up with lower efforts for program comprehension (as developers still know about that code) and testing. So why not proactively use this effect and prioritize deficits within the latest changes of the source code?

So we set out to implement quality control processes at our customers that regularly assess the quality of the latest changes to systems. The quality engineers driving this process file tasks for quality improvement every 1-4 months (depending on the development activity in the projects). Furthermore, they report about the degree of quality improvement (or decay) since the last interval (e.g. a development iteration). The tasks are then scheduled like other change requests for the next iteration(s).

Using this quality control process the quality decay of a software system can be stopped, even an incremental improvement of grown-systems can be observed. The chart above shows the further evolution of the above mentioned system. The vertical lines show the points in time when quality reporting was done by the quality engineers. Although the system still grew over time, a significant decrease of the overall amount of quality defects took place.

However, this process raises additional requirements for quality analysis tools as they have be able to distinguish old deficits from newly introduced ones. Most tools today simply draw trend charts but do not allow for assessing the root causes for positive or negative tendencies. The long lists of up to several thousand quality issues reported by tools already on medium-size code-bases cannot be tracked manually. This is one of the main reasons why we started the development of Teamscale, the first tool that is aware of the birth and death of every individual quality deficit identified by different analyses. It even recognizes renaming operations or moves of code to new locations without claiming the contained deficits to be new. This tracking of deficits works better, the closer two analyzed revisions of a system are to each other (edit-distance). Thus, it is much more precise when not only executed in the nightly build, but after every single commit using incremental analysis techniques. This approach even comes with the great effect that developers get an immediate response on the changes they performed.

Although tools are not sufficient to achieve quality improvement, more sophisticated tools are needed in order to implement effective quality control processes.