Regression Testing is an important step in ensuring product quality and it has long been used in formal testing efforts. The focus of regression testing is to ensure that nothing that worked fine previously in the product has been broken due to a) new code changes, b) changes to content files or c) any deployment/build changes and upgrades. When regression testing has issues around coverage, it can have an adverse impact on:

a. overall product quality
b. team morale increasing time spent and effort expended
c. the debugging and troubleshooting process making them even more challenging, creating deep and nested defects

To be safe, testing groups would often prefer to go with choosing a larger than required regression test suite. However, this is not a preferred route to take for several reasons such as:

a. In the current day development world, all product teams operate within very tight timelines where one does not have the luxury of building an elaborate regression test suite

b. More importantly, a well chosen and optimized regression test suite, is what will give one the required coverage in ensuring product quality. A larger suite chosen purely by the number of test cases is not going to give any measurable results in enhancing the regression test coverage and product quality

That said, choosing such an optimized regression test suite is easier said than done because of the complexity of product scenarios, geographically distributed teams, tests that are run in an exploratory manner, scenarios that are supported on multiple platforms (mobile and non-mobile devices, operating systems, browsers etc.). Once these challenges are mitigated and the right regression test strategy has been arrived at, it is a great win to the entire product team and more importantly the users in experiencing a product of great quality.

We at QA InfoTech suggest Regression Testing in 3 stages, so the effort is spread throughout the testing cycle and regression issues are caught early on rather than waiting until the end of the test pass:

1. An informal yet very valuable regression test phase with every feature release/new build deployment: Identify and include core workflow scenarios to include as part of your sanity testing every time a new build is deployed. These could be in the form of build verification tests, but will also touch upon regressions and any known areas which are especially unstable and prone to breaks/failures. Tests are largely automated here, but could also cover some manual scenarios

2. Regression with every defect fix: Individual testers will be encouraged to carefully test not just for the defect fix but also holistically look at the defect and possible areas that might be impacted by this fix and test them from a regression angle. Test are largely manual in nature here

3. A formal regression test pass at the end of every testing milestone. In this test pass, identify E2E scenarios and tests such that running them will give you maximum coverage with minimum number of tests and time to execute them. A best practice at this stage would be to use instrumented builds and enable code coverage such that you can measure the traceability of your tests to the code, both in your manual and automated test runs

In addition to the above, the team needs to think through the right balance of manual and automated tests, maintaining a dynamic regression suite, encourage the team to closely analyze exploratory tests from a regression angle, promote cross group collaboration to understand pain areas, look at past test results and issues from the field in building a strong regression suite. While this may seem laborious, once the engine is established it will only be a marginal increment in efforts to plan subsequent regression cycles whereas the returns on investment will be very high in building a solid product of exceptional quality.