We have to test more and more functionality in less and less time, as successful software grows from release to release, but release cycles are getting shorter and shorter.
Historically grown test suites are often not up to this challenge, since they test too much and too little at the same time. Too much, since they contain redundant tests that cause execution and maintenance costs but provide little value over similar tests. Too little, since important functionality remains untested. We must make these test suites more effective (i.e. find more bugs) and more efficient (i.e. faster/cheaper) to succeed in the long run.
But how? Luckily, our research community has worked on approaches to increase test effectiveness and efficiency for decades.
In recent years, AI-based approaches appeared that also promise to help us find more bugs faster. In this deep dive, I present various approaches to find more bugs in less time: History analyses of the version control system show, where most bugs occurred in past releases. This often uncovers process flaws that are root causes of future bugs.
Test gap analysis reveals, which code changes have not yet been tested and are the most error prone. Pareto optimization of test suites, test impact analysis and predictive test selection identify tests that have the best cost-benefit ratio right now. And finally, defect prediction uses AI to predict where future bugs will occur.
We have implementing each of these analyses, done empirical research on how well they work and employed them in our own development and at our customers. For each analysis, I outline its research foundation and show how well it works – while some excel, others are do not work at all – to answer how well AI can really help us to find more bugs in less time.
What you will learn
Our development and test processes contain a trove of data that we can analyze (with or without AI) to help us find more bugs in less time.
There is no one-size-fits-all approach, as test processes differ. But I have yet to see a project that cannot benefit at least in some ways from these approaches.
A key strength of some AI-based approaches is their easier applicability to a broader range of test types (and not their better results) compared to non-AI-approaches.
Session Details
Introductory, 90 minutes, Focused Deep Dive, Innovative testing strategies