Conference Talk
Fast Feedback despite Slow Test Suites? Test-Impact-Analysis for the Robot Framework
Fast Feedback despite Slow Test Suites? Test-Impact-Analysis for the Robot Framework



Test suites grow over time. We see many test suites that take hours or days to execute. This is painfully slow.Test-Impact-Analysis automatically selects those tests that find new bugs quickly. This helps to get rapid feedback even for slow test suites. In the talk, I present both research foundations and practical examples for the Robot Framework.

Successful software grows, and so does its test suite. Increasingly more of our customers have test suites that take several hours of even days to execute in full. However, the longer the time between the introduction of a bug during development, and its discovery through a test failure, the harder it gets to pinpoint the change that introduced this bug. The longer the test execution time, the larger the pain for all involved developers and testers.

If the execution of the entire test suite takes too long, a simple solution is to only execute a subset of the suite that require less time. If this subset is chosen well, its execution can find the majority of the bugs very quickly. The remaining tests can then still be executed every night / week to detect the remaining bugs. This way, we achieve rapid bug discovery times for most new bugs even for large, slow test suites.

However, how well this works in practice depends on how we select the subset of tests that gets executed more frequently. In recent years, we have experimented with many various selection techniques, both in research projects and together with our customers. Two work very well: Test-Impact-Analysis and Pareto-Optimization of test suites.

In this talk, I present both approaches. I share their basic ideas and our research findings. I also present how they can be applied to the Robot Framework and give examples from the Robot Framework and our customers.



Problem solving

Invited Talks

Your location, our expertise

We are happy to come visit you in your office for an internal conference or a workshop. Our list of topics includes quality analyses, quality control, but also test control or introducing peer reviews. You are also welcome to pick a topic of your choice

60-90 minutes talk
About a software quality topic
In your office or remote
Our travel expenses only
Date by arrangement