Conference Talk
Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical Analysis
Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical Analysis
Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical Analysis

Both automated and manual software testing are widely applied in practice. While being essential for project success and software quality, they are very resource-intensive, thus motivating the pursuit for optimization. Goal : We aim at understanding to what extent test optimization techniques for automated testing from the field of test case selection, prioritization, and test suite minimization can be applied to manual testing processes in practice. Method : We have studied the automated and manual testing process of five industrial study subjects from five different domains with different technological backgrounds and assessed the costs and benefits of test optimization techniques in industrial practice. In particular, we have carried out a cost–benefit analysis of two language-agnostic optimization techniques (test impact analysis and Pareto testing a technique we introduce in this paper) on 2,622 real-world failures from our subject’s histories. Results : Both techniques maintain most of the fault detection capability while significantly reducing the test runtime. For automated testing, optimized test suites detect, on average, 80% of failures, while saving 66% of execution time, as compared to 81% failure detection rate for manual test suites and an average time saving of 43%. We observe an average speedup of the time to first failure of around 49 compared to a random test ordering. Conclusion : Our results suggest that optimization techniques from automated testing can be transferred to manual testing in industrial practice, resulting in lower test execution time and much lower time-to-feedback, but coming with process-related limitations and requirements for a successful implementation. All study subjects implemented one of our test optimization techniques in their processes, which demonstrates the practical impact of our findings.

Problem solving

Invited Talks

Your location, our expertise

We are happy to come visit you in your office for an internal conference or a workshop. Our list of topics includes quality analyses, quality control, but also test control or introducing peer reviews. You are also welcome to pick a topic of your choice

60-90 minutes talk
About a software quality topic
In your office or remote
Our travel expenses only
Date by arrangement