Suites are great for grouping test cases to run them in parallel.
Why use suites?
Suites are simply groups of multiple tests. Organise your tests into Suites to quickly run several tests at once.
By default BugBug has one suite called "All tests", where you will find all tests you created in a project. All newly created tests will be automatically added to this suite.
Here are some examples of suites that usually help with test automation workflow.
you can have a suite that monitors only core features on your production, scheduled every hour
you can have a full regression suite that you run manually before the release
you can have a "feature branch" suite that has work in progress tests that are not yet ready to be run on production, but after the release you will add them to the production suite
you can have suites that run the same tests with different profiles, for example checking your app on multiple languages
Create new custom suite
You can create custom suites in the Suites tab:
1.
Click Create new suite button
2.
Add Suite name
3.
Choose tests that you want to include in a suite
4.
Choose options for a suite
5.
Save
Auto-retry failed cloud tests to prevent flaky tests notifications
If your suite sometimes randomly fails, but there is no certain reason for that, you may be annoyed by the "failed" false positive notifications. The industry term for tests that fail randomly for no reason is flaky tests.
Flakiness can be caused by many different factors:
a slowdown in the internet connection
busy server side
temporary machine CPU overload
You probably don't want to get notifications for such randomly failed tests - your app works as it should, it was just a temporary problem that doesn't require attention. BugBug allows you to prevent such unnecessary alerts.
When auto-retry is enabled:
If one of the tests in a suite fails, it will be automatically run again
if another attempt passes, we will mark the suite as "passed", but you can still see the failed test attempt in the test runs history, marked as "auto-retried"
If the test fails for the second time, it will be again restarted
if the test fails 3 times, we will mark the suite as "failed"
Example suite run history with auto-retry enabled. There were 3 attempts to run the same test.
When auto-retry is disabled:
If any of the tests fail, the suite will continue running and after all the tests are finished it will be marked as "failed"
Example suite run history with auto-retry disabled. There was just one attempt to run the test.
Important! Auto-retry only works in suites run in the BugBug cloud. Pro subscription is required.