Suites

Why use suites?

Suites are great for grouping test cases to run them in parallel. They are simply groups of multiple tests. Organise your tests into Suites to quickly run several tests at once.

By default BugBug has one suite called "All tests", where you will find all tests you created in a project. All newly created tests will be automatically added to this suite.

Suites can be scheduled to run in the cloud and are useful for working with different environments.

What suites should I have?

Here are some examples of suites that usually help with test automation workflow.

• you can have a suite that monitors only core features on your production, scheduled every hour

• you can have a full regression suite that you run manually before the release

• you can have a "feature branch" suite that has work in progress tests that are not yet ready to be run on production, but after the release you will add them to the production suite

• you can have suites that run the same tests with different profiles, for example checking your app on multiple languages

Create new custom suite

You can create custom suites in the Suites tab:

1. Click Create new suite button

2. Add Suite name

3. Choose tests that you want to include in a suite

4. Choose options for a suite

5. Save

Auto-retry failed cloud tests to prevent flaky tests notifications

If your suite sometimes randomly fails, but there is no certain reason for that, you may be annoyed by the "failed" false positive notifications. The industry term for tests that fail randomly for no reason is flaky tests.

Flakiness can be caused by many different factors:

• a slowdown in the internet connection

• busy server side

• temporary machine CPU overload

You probably don't want to get notifications for such randomly failed tests - your app works as it should, it was just a temporary problem that doesn't require attention. BugBug allows you to prevent such unnecessary alerts.

When auto-retry is enabled:

• If one of the tests in a suite fails, it will be automatically run again

• if another attempt passes, we will mark the suite as "passed", but you can still see the failed test attempt in the test runs history, marked as "auto-retried"

• If the test fails for the second time, it will be again restarted

• if the test fails 3 times, we will mark the suite as "failed"

Example suite run history with auto-retry enabled. There were 3 attempts to run the same test.

When auto-retry is disabled:

• If any of the tests fail, the suite will continue running and after all the tests are finished it will be marked as "failed"

Example suite run history with auto-retry disabled. There was just one attempt to run the test.

Important! Auto-retry only works in suites run in the BugBug cloud. Pro subscription is required.