Performance testing as a non-functional testing attribute has typically been encouraged to be taken up after a product is functionally stable. This helps bring in focus on the core elements of performance be it load, stress, endurance, scalability etc., not having to bother about functional issues that may come in, along the way. This was especially prevalent during the days of waterfall style of development. With Agile, obviously a lot of things changed for the good. Performance testing and associated engineering efforts was one area which still remained independent of the ongoing Sprints, given the nature of work – both from the complexity to contain within one Sprint as well the need for functional stability. Most often it was taken up one to two Sprints behind.
Then came in a phase where specific non-functional attributes, especially performance testing services, were taken up as a horizontal stretching across Sprints – this ensured the due diligence done from a Performance testing standpoint too, although it was not taken up at a Sprint level. The performance engineers closely worked alongside the core product team, were completely plugged in into the product architecture, feature set, performance requirements, benchmarks etc. and continued to operate horizontally. What this meant was, unlike in the past, a dedicated performance testing team was now available for the core product team, working horizontally.
In this evolution process, where do we currently stand? – With the momentum brought in by DevOps, more robust performance testing services had to be explored rather than a suite of tests / scenarios that aligned with business requirements. Typically, performance issues are vague – as in, debugging them takes a lot of time. The defects may not be actionable enough for the devs to immediately work on. These needed to be improved on, even as it continued to be horizontal – some better granularity had to be brought into the DevOps cycle, both for time and faster issue debugging and resolution. A best practice that we have introduced herein is unit testing at the performance level. These tests are integrated in the DevOps cycle, making modules very granular, results quicker to be ascertained and issues easier to be debugged. Unit tests for performance testing is one of the best bets to also closely connect the performance engineering team with the core development team and functional quality teams. Triaging and resolutions are also much faster. And when performance is monitored and improved from the unit level, the scenario level performance testing has better chances of faring well. If this is such a straight forward answer, why have organizations not taken this up all along? Defining the right performance unit tests needs a lot of expertise, understanding of the system and architecture. Unlike functional unit tests which are typically taken up by developers, performance unit tests are usually taken up by the performance testers. This requires dedicated bandwidth and expertise, as opposed to serving as a shared performance team across projects. All of these require buy-ins from stakeholders both at business and engineering levels. Given the value the teams have been able to showcase with performance testing in the DevOps cycle, this is now becoming an easier sell across the board and this will only continue to evolve and mature in the coming years, that very soon performance scenario testing may also become an inSprint activity, enabling close monitoring before and after each Sprint release.