In the world of software testing and quality assurance, we all know the value Test Automation brings in, to improve test coverage, overall product quality and the tester’s productivity. But all of this value flows in and the ROI is reaped only when the automation code is robust and reliable and can produce consistent results to catch product bugs. The term “product bugs” is very important here. If the automation code does not catch bugs (if this is truly because the product has reached a steady state and is largely bug free, then it is acceptable) or shows more false negatives (due to test code issues rather than product issues), there will be a lot of wasted effort including:
• Test Automation Effort – code design, implementation and maintenance resulting in wasted time, cost and human resources
• Triage time – involving the product team to look into the invalid bugs reported resulting in expended time, cost and human resources and more importantly the reputation of the test team
• Resource Usage – Machine, other infrastructure and software usage for automation execution.
One of the traditional solutions adopted to solve this problem and make the test code reliable and robust is to “test” the test code itself, as if it were product code. While this sounds great in theory, are all teams really practicing it? Also, it is all the more important to carefully chose a smart strategy and decide “How much to test” your test code because your goal is very different in this test effort compared to product testing goals. Any extra test effort here is again going to lead to a lot of wasted resources and is not going to add any value. Choosing what areas to test, how to test, and agreeing upon your goals of testing the test code are vital. Here are some best practices on “Testing your test code” which will help you make these important decisions.
1. Static tests around code reviews are very important. Testing your code to ensure coding standards are adopted is very important to ensure automation maintenance effort is minimal down the road. In such reviews check for modular test design, including things like reusable methods, decoupled test data to ensure code is clean and easy to work with.
2. Your test code’s functionality testing is what is most important – specifically focus on whether your code is testing what is intended. It would be good to pick a few core tests and run them both manually and through automation to ensure consistent results are produced
a. Focus more on unit testing your code. Leverage simple unit testing tools such as NUnit, JUnit etc. coupled with manual testing efforts using simple checklists to achieve your goal without making this process a complicated one
b. Testing the code that you write to verify builds (Build Verification Tests) in detail is very important because this is an area where you will use your test code very frequently. If build breaks happen due to test code issues it is not only a lot of wasted overhead but also adversely impacts the visibility of the test team within the overall product team
3. Before you actively start using your test automation code and share it with the product team, carry on a few trial runs where you execute the code on a few builds (if required use even local builds) to ensure the code yields consistent results (e.g. consistent passes and fails between builds) and closely evaluate the kind of failures returned to ensure they are indeed product bugs
4. Schedule your code to run in sequence a few times (e.g. 10-15 times) over a day in an unattended mode to ensure there are no built in dependencies which affect the test runs. E.g. access issues, any versioning issues, any software / hardware dependencies etc. Also, while a few initial trial runs could be on local environments, get your test code deployed and run against the actual test environment (at least on a trial basis) as soon as possible as this is what will reveal any other deployment and cross module dependencies
5. Code coverage – Running your test code with code coverage tracking on, is a very good test to determine the coverage achieved. This is a good objective evaluation of your test automation’s ROI also helping you to find out areas where you can beef up your automation and areas of product code that need to be removed
6. Don’t focus as much on areas such as performance, security, UI, usability, globalization, compatibility etc. while testing your automation code. In fact, you can almost eliminate a few of these, such as globalization out of the mix to purely focus on the functional aspects of your test code
7. Beware of any tool/framework limitations to ensure they don’t impact your test results including any support clauses to help build your automation on a sustainable platform
8. Test for your test code’s reporting capabilities – This is one area where you can focus on UI and usability aspects since your test code results are often shared with various levels in the management all the way from even the executive management down to the grass root level. Test for your test code’s accuracy, level of detail, archiving capabilities and usability aspects when you test the reporting module’s code
9. Set clear exit criteria – Similar to product testing, you will need to set clear exit criteria to sign off on your test automation code and move it to production (where you start using the code for actual product testing). Exit criteria need to be very stringent (I would even say 100% reliability at all times) in this case to ensure test automation code is reliable and can be leveraged to test the product. If specific scenarios/tests are delaying your process, de-activate them, and move them to a queue to be analyzed later. I say this because although the test code needs to be reliable, the team should also understand that testing the test code is only a supplemental activity and that the larger goal is really to leverage this and focus efforts on actual product testing. So, the test team has to lay down its priorities right in expending the required amount of time to test and strengthen the test code while still focusing on the quality assurance of the prod3 end user product
Try these out and do share any other tips/inputs you have around “Testing” your test code. I would love to hear what you have to say!