Test Selection: Find defects earlier & speed up test runs

Test
Selection

Test Selection runs exactly those of your tests that are most likely to discover defects early on, giving you fast feedback and safeguarding your expensive test stages.

Request a Demo

Test More. Test Faster.

Challenging demands

You have to test ever larger amounts of software in ever smaller periods of time, because software systems grow more complex by the day and the market demands ever shorter release cycles.

In response, you invested into test automation and continuous integration (CI), but now even your automated test suites run hours or even days and just don’t keep up with the frequency of changes.

The wrong turn

Since your test suites take so painfully long, you decided to run them less frequently. Now you’re struggling to analyze and fix test failures among the large amounts of changes that happen in between test runs. This slows you down even more.

A way out

You select a subset of your tests to run more frequently. If chosen well, this subset allows you to identify most defects very quickly, while the rest is caught in a later stage, still running all tests.

Pareto Testing

Bring tests to order

A single test usually covers only a small part of the system. Similar tests overlap and, thus, cover some parts redundantly.
Teamscale’s Pareto Testing reorders tests to cover as much of the system as quickly as possible. Such a smoke test may reveal 90% of the defects in the first 11% runtime – an effective quality gate for expensive test runs.

Test-wise coverage

Pareto Testing uses test-wise coverage and test runtime to prioritize tests.

Test-wise coverage captures which code of your software system each individual test executes. You may capture this with a coverage profiler.

Integration

1. You decide how much time to spend on each smoke test.
2. Teamscale records test-wise coverage and computes an optimal list of tests for this time limit.
3. Your test runner uses this list as input.
4. You repeat the whole process regularly, e.g., once a quarter.

Test Impact Analysis

Align tests with changes

When you test frequently, the amount of changes under test is small compared to the overall system. Hence, many of your tests are unrelated to these changes and will tell you nothing new.
Teamscale’s Test Impact Analysis selects exactly those tests that run through code changes and reorders them to cover as much of these changes as quickly as possible. Such an optimized test set may reveal 90% of the defects in only 2% runtime.

Test-wise coverage and changes

Test Impact Analysis uses code changes, test-wise coverage and test runtime to select and prioritize tests.

Code changes are automatically collected from your version control system (e.g., Git or Subversion).

Test-wise coverage captures which code of your software system each individual test executes. You may capture this with a coverage profiler.

Integration

Teamscale fetches changes from your version control system and integrates with your test runner, to run only the tests selected by Test Impact Analysis. During execution, it records test-wise coverage of the respective tests and feeds this data back to Teamscale.

Put it to Use

Run selected tests frequently

Get early test feedback on any change, base the earliest possible test stage on Pareto Testing or Test Impact Analysis. This helps you catch most defects early on, when they are still cheap to fix.

Keep running all tests

Neither Pareto Testing nor Test Impact Analysis find all the defects that your entire test suite finds. Therefore, you need to keep running all the tests regularly, knowing that you caught most defects earlier.

Always run critical tests

Sometimes you have tests that are of such importance that they need to run in any case. Simply always run such tests and tell Pareto Testing and Test Impact Analysis to exclude them from their results.

Experience Fast Testing

Earlier test feedback

Find 90% of the defects with only 2% of the test runtime, using Test Impact Analysis, or with 11% of the test runtime, using Pareto Testing, depending on how much speedup you need.

Cost reduction

Safeguard your more expensive test runs (e.g., a full regression test) with an efficient quality gate.

Support

FAQs

Everything you need to know about Test Selection. Can’t find the answer you’re looking for? Please chat to our friendly team.

How come Test Selection provides so much speedup?

Admittedly, this is not only because Pareto Testing and Test Impact Analysis are such amazing ideas. It's also because simply running all tests all the time on rather small changes is highly inefficient to begin with.

What's the difference between Pareto Testing and Test Impact Analysis?

Test Impact Analysis tightly integrates with your test runner, to optimize tests to the changes under test. This yields exceptional speedup, but requires a complex technical setup and quite some maintenance effort. Pareto Testing, on the other hand, computes offline at larger intervals. This is much easier to set up, but brings less speedup. The choice really depends on your context. Why not try both and then take your pick?

Do the selected tests find all defects?

No. Neither Pareto Testing nor Test Impact Analysis guarantee that. There are approaches that, in theory, provide such a guarantee, but they tend to select the entire test suite in practice, so you gain nothing.

We strongly recommend you keep running all your tests regularly, with the good feeling that you caught most defects already.

Do I still need to run all my tests?

Yes. Neither Pareto nor Test Impact Analysis find all defects that your entire test suite may find. Therefore, you still need to run all tests regularly. Test Selection helps you to find most defects at an earlier stage.

Can Test Selection consider priority of tests?

No. Instead of trying to ensure high-prio tests are always selected, simply ensure you always run them and tell Pareto Testing and Test Impact Analysis to exclude them.

Can I use Test Selection to discard unnecessary tests?

Not by itself. Neither Pareto Testing not Test Impact Analysis assess test redundancy or quality. You may, however, take a closer look at those tests ranked last by Pareto Testing. Those are the slowest tests relative to the additional coverage they provide. If they turn out unnecessary, deleting them yields high return.

Experience Exchange

Would you like to exchange experiences on Test Selection?

Any complex analysis raises questions. Is it applicable to you at all? What experiences have other companies in your industry had with Test Impact Analysis or Pareto Testing? Are the technologies you use supported? To name just a few.

I have dealt with Test Impact Analysis and Pareto Testing for many years. In research papers, speaking at industry conferences, talking to testers and test managers, and working with customers who have been using it for years.

I’m happy to chat with you about Test Selection!

Contact Fabian Streitel