Software grows with every release. At the same time, release cycles get shorter and shorter. As testers, we thus face the challenge to test more and more functionality in less and less time.

Unfortunately, our historically grown test suites are often unhelpful, since they test too much and too little at the same time. Too much, since they contain redundant tests that incur execution and maintenance costs but provide little value over similar tests. And too little, because we lack tests for important functionality in changed or new code.

Instead of doing more and more work in less and less time, our tests are, thus, costly and slow while still letting too many bugs slip into production.

In order to solve this challenge, we need to stop treating our testing efforts like a black box. Instead, as our conference motto suggests, we need to better understand what we are doing here, and act accordingly.

In this talk, Elmar will demonstrate how data from our own test and development process shed light into this black box and allow us to both optimize our tests and align them with development activities: test gap analysis reveals which code changes have not yet been tested by neither automated nor manual tests. Pareto optimization of test suites and Test impact analysis identify the tests that most quickly find remaining bugs. And history analyses of the version control system show where the most bugs occurred in the past, potentially revealing fundamental process issues that we need to fix.

For each analysis, Elmar outlines the research foundation and its limitations. He gives examples and experiences from practical applications in different domains, and shows how they helped teams to find more bugs in less time.