Being able to reproduce defects is a very important step in defect management to aid quick defect analysis, fix and closure. Screen snapshots highlighting the defect, a video of the series of steps to see the defect have been traditionally used as extras, to help reproduce the defect in addition to the “Steps to Reproduce” that a tester outlines in his defect report. Whether the tester is working in a paired development mode, sitting beside his developer, or is co-located yet not in the same room as that of his developer or is remotely located from his development team (especially in cases where a testing services vendor may be hired), the importance of reproducing defects can never be underestimated. In a large sense, this also speaks for the reputation that tester has on his team. Testing in the current day has become very complex having to deal with varied parameters in its scope of operations – for example, what are the chances I will be able to consistently reproduce a defect filed by a tester in Africa who has found an issue on a specific version of his mobile device? Given the variables at play, it is now more important than ever before for a tester to take conscious steps to maintain the sanctity of “defect reproduction”. Hereare some steps to promote better defect repeatability in our test efforts:
1. Be cognizant of your test configuration even in an exploratory test effort

2. Try to bring in a sequence and logical flow in your test effort rather than trying haphazard and random steps. This will help you sequentially troubleshoot and determine your steps to reproduce the issue with improved chances of seeing the issue. This does not mean curbing your creative juices in exploring newer tests, but this is where the convergence of creative and critical thinking, plays an important role

3. Take time to analyse an issue once found from possible other areas that might be affected – for example, other locales, other devices, other operating systems which might face the same issue

4. Stay current with forums that discuss issues with specific devices, browsers, operating systems that help you differentiate product specific issues from external issues

5. Give time to reproduce certain issues. Some issues show up only with time gaps. Experts suggest taking breaks at times (say a coffee break) to come back and revisit an issue. Leave the system on, at a logical state then to enable it to process the information you have provided. For all you know when you come back you would see the issue reproduced. Such gaps also help you come back refreshed to look at the issue from a possibly different perspective

6. Discuss the issue with your team members/developers. For all you know, they may have views on the issue, that you have not thought about

7. Explore ways to give the developer access to your machine/environment to see the issue, wherever possible, instead of giving just snapshots and video clippings

8. If in doubt, take the time yourself to reproduce the issue a few times, at intervals before reporting the issue. For all you know the issue shows up only at specific times of the day influenced by users and their usage patterns at that point. Such extra details go a long way in building credibility for the defect and for yourself
While every attempt is made to reproduce a defect, sometimes it may just not be worth the effort, especially when it has been reported by a customer and it appears to be very time consuming to try to reproduce it. In such cases the team would be better off brainstorming what might have happened, where, why and proceed with analysing the code to see how to fix it. The tester in this case, plays an important role in seeing if any quick reproductions can be attempted; if not trying to see how to help with the debugging and troubleshooting with the goal of fixing the issue and regressing it. The initial analysis that was done to see if it can be reproduced, will help the tester at a later stage when the regression effort is taken on.
So, while the tester takes every sincere attempt to reproduce defects and provide as much detail as possible to improve the chances of reproducing it, the team as a whole needs to buy in that at times, thereproduction may not be worth the effort and it might be a better use of everyone’s time to jump into fixing the issue directly through code reviews and tracing code paths.

I happened to read a really engaging paper today, by a leading testing consultant and speaker, Zeger Van Hese on “Testing in the Age of Distraction”. He talks about how there are distractions all around us and how they are inevitable. However not all distractions are bad; some are good and help us further boost our thought process and productivity. Of the several points that his paper talked about, I wanted to pick on one which was the most intriguing to me – “critical thinking vs. creative thinking in testing” and what share each of these play at different stages in the testing lifecycle. He has referenced this from the original blog post by John Stevenson.

While there are ample definitions of these terms critical and creative thinking, by experts, available online, I like his simple interpretation of it which reads:
Creative thinking is divergent while critical thinking is convergent.

There is no denying that both are equally important to inculcate as a tester but what is even more important is knowing what to use when and where, to bring their combined benefits into the testing process – this is important especially in the current day, where our distractions are high and just continuing in a flow of creative thinking might not yield us any convergent benefits that translate into better product quality.

He talks about the various stages including: documentation review, test planning, test execution, test analysis and test reporting touching upon what activities are taken up at each of these stages. At a high level, critical thinking outweighs creative thinking in the documentation review and test analysis stages where you are more into an information gathering/assimilation and analysis modes. Creative thinking takes a stronger hold in the test planning stage when you are diverse in your thought process, letting your creative flow guide you into what kinds of tests you want to carry out, what your testing strategy needs to be, looking at how to bring in newer test ideas which are not just intrinsic but also extrinsic in nature.

Critical and creative thinking have a relatively equal share to play in the test execution stage where you put your ideas into action. While being creative here lets you test newer possibilities including ones along the unbeaten path, critical thinking herein is that soft leash that keeps a check on your creative thinking, to ensure they translate into measurable results to align with your product quality. Finally in the test reporting stage again both critical and creative thinking are used, but creative thinking should be used more as it helps you interpret your results into actionable items for the team at all levels, be it your senior management or your immediate team. So, you are diverse in your thinking to interpret results and report them not as mere numbers but as information that is relevant and meaningful in empowering various entities take important decisions. While your creativity and diverse thinking helps you with this process of embracing everyone on the team, your critical thinking is important in converging all of these results into one single message that represents your product’s quality.

In my next testing assignment, I for sure am planning to try this path of making a conscious effort and evaluating how my critical and creative thinking hats need to be used at different stages. Hope you would be excited to try this too and share your feedback. Thanks John for this insightful post which has already unleashed my critical and creative thought processes.