As a global company, the IT service provider Allianz Technology GmbH invests heavily in agile working methods, a uniform tool stack and process automation in order to achieve and maintain high quality standards across the company. Of course, this also applies to software testing in particular. In this context, the company aims for a high level of transparency in order to be able to carry out continuous and risk-based testing during the entire development and maintenance cycle. In the spirit of »shift left«, the tests should be created as early as possible in the development cycle and should be progressively run automatically—and this is where Teamscale’s Test Gap analysis helps. 

Too Many Field Errors as Drivers

At Allianz, Albert Bischof and Nadezhda Tonchev are responsible for the quality tools and software testing strategy of around 60 teams that develop and maintain a portfolio of around 40 applications. At the beginning of 2019, they were faced with the problem that too many field errors were occurring across the entire portfolio. And all this despite the high testing effort Allianz had already invested, as well as the ever-increasing number of automated tests! The fact that different teams were responsible for the tests which were implemented using a variety of different technologies and were spread out across several test levels made it extremely challenging to identify the causes of this situation.
However, in order to identify these causes all the same and to consequently create the basis for effectively improving testing processes at Allianz, Albert and Nadezhda made it their goal to make all their testing processes more transparent, in accordance with the company strategy, and across the entire testing pyramid, i.e., across all test levels and technologies. In their search for suitable approaches and tool support, they learned about Teamscale’s Test Gap analysis at a presentation at the Software Quality Days conference in Vienna and were very impressed. In May 2019, an on-site workshop was held at Allianz together with CQSE, after which all participants were convinced that the analysis would provide them with the desired insights into the development and testing process.

Test Gap Analysis as a Solution

The Test Gap analysis evaluation with the support of CQSE already began in the summer of 2019. For that purpose, the technical requirements were first created in order to establish test coverage in the Java backend as well as in the JavaScript and TypeScript frontends, both for automated and manual tests and, of course, at all test levels. Afterwards, the relationship between code changes and test activities was measured over a period of 2.5 months. This revealed that more than half (58 percent) of all changes in the test process remained entirely untested. For many of the tickets implemented during this period not a single code change was covered by the tests!

Despite substantial testing at Allianz Technology, it was measured that over a period of two-and-a-half months almost 58 percent of all changes were not executed by the tests. 

In addition to this fundamental insight, Test Gap analysis also made it possible to investigate the causes because the analysis takes the chronological sequence of code changes and test events into account. This is how Albert and Nadezhda found out that the complex software architectures and under-specified requirements often resulted in changes that were not foreseeable for test planning based on the description of the requirements. Tight schedules and limited resources for the test made the situation even worse. Furthermore, they identified situations in which the timing between the change being made and the test being executed simply did not match and this led to, for example, tests being run before the corresponding changes had been made to the test environment.

Transparency as the Key to Success

Overall, the results confirmed Nadezhda’s and Albert’s approach because in particular the transparency needed to align the test efforts with the development process was missing. And with Test Gap analysis, they now had just the right tool at hand, which they have been rolling out for more applications in their portfolio since the successful evaluation.

Today, the quality assurance champions at Allianz Technology use Test Gap analysis in order to steer their team’s test activities and deployment decisions based on data rather than merely having to trust their gut feeling. Detailed information on test gaps is during development already directly available in Jira at ticket level and is updated automatically. As a result, the metric can be used as an acceptance criterion by product managers, and this has already reduced the test gap in the application portfolio at Allianz by an average of 20 percentage points and has increased overall test coverage by 48 percentage points, while putting a clear focus on the code areas that are most at risk. At the same time, this systematic approach, with the remaining test gaps in mind, helps to keep redundancies in test suites to a minimum and thus sustains the test suites’ maintainability.

In the Software Intelligence Talk on October 2nd, Nadezhda, together with Fabian Streitel (Head of the "Test Intelligence" team at CQSE), will discuss the challenges Allianz faced in 2019, the lessons learned from implementing the analysis and the initial data measurement, and the benefits that the high transparency has brought to Allianz since then. The talk will be in German language.
Register now (online & free)!