Software development has become a very integrated activity over the last decade, where instead of one large piece of code developed in-house, a lot of small pieces flow and fall in place like pieces of a puzzle, to shape the E2E product. Such integrations and collaborations have been necessitated by a lot of drivers in the market such as: Need for faster time to market, leverage domain expertise from specialists rather than building everything in house, feature rich product, technology advancements that make such integrations seamless and possible. For a long time now, cross group collaborations within a company have been encouraged to promote resident experts in a specific area, ease of code maintenance, modular functionality etc. For e.g. in my projects at large ISVs, I have almost always seen such integrations in areas such as login functionality, payment processing, installation and setup. In the more recent years such integrations have extended bounds with ISVs now willing to integrate external code bases with theirs to compete and maintain an edge in the marketplace.

Besides some areas I’ve mentioned above, a few others that have always been excellent targets for external interfacing include analytics and reporting. Large ISVs have quite heavily depended on companies such as Omniture, Web Trends to incorporate core reporting modules into their software functionality. The world of application development is now opening several doors for such external integrations. Having looked at some of the striking advantages such integration brings, one cannot miss the inherent challenges in such third party collaborations. Let’s take a quick look at some of those and also the associated mitigation strategies.

Challenges and some mitigation strategies:

Challenge: Quality of Code – This is a huge challenge since you have a third party team that is working on giving you the source code. Very often, product teams’ face challenges in imbibing coding standards within their own team. In such scenarios, how do you ensure code quality from an external team where you do not have complete visibility into and control on?

Mitigation Strategy:

• Define clear protocols and coding standards to use and ensure they are consistent with what is being used in house
• Define metrics for measuring code quality and associate them with Service Level Agreements (SLAs) to reward (for met and exceeded expectations) and penalize (for expectations not met)
• Define and implement quality gates/checkpoints, before integrating third party code into your code base

Challenge: Getting the external team to understand your business requirements, time, cost and quality constraints and effectively customize their code to meet your needs

Mitigation Strategies:

• Involve the external team sooner than later in your product development life cycle and engage them in all important business and product conversations, including any field trips, client visits
• Encourage free flowing yet managed communication channels between your business team and external team. I say managed as unless this is monitored, the entire flow can soon become very haphazard
• Ensure transparency of communication at all times and insist such interactions from their end also to understand their product timelines, challenges if any
• Get the third party team to spend adequate cycles on using your product hands on; unless this is done, they will only get a theoretical view of the product and will not be able to deliver a quality integration piece

Challenge: Defect Management – Disconnected defect management practices often result in huge gaps in timelines to fix defects, incomplete defect fixes, ineffective hand offs between teams etc. all of which lead to delays in project timelines and increase in project total costs

Mitigation:

• Establish clear defect management protocols at the start of the project including acceptable timelines, entities involved in the hand shake between the two teams, tools to use etc.
• Define service level metrics around number of defects, type of defects, severity of defects, defect fix timelines, regressions etc. at various stages in the product lifecycle
• Get the third party team to spend adequate cycles on using your product hands on; unless this is done, they will only get a theoretical view of the product and product development and defect fixes will not be comprehensive

Challenge – Maintenance of external code, post the initial release is often a huge challenge

Mitigation:

• Think about code sustenance upfront. If you want the external team to continue to own the code, understand and align their release cycles to yours, so their code is up-to-date in user features and functionality, use of technology, defect management etc.
• If code is to be transferred to your team, ensure you have the right team identified for training and take over; have one person assigned from the external team for future questions and clarifications

Having talked about some of the core challenges and mitigation strategies, let’s now look at some vital quality gates that need be implemented in the product team in addition to the quality checks done by the external team before and after code integration.

• Static analysis of code – This is important to ensure the code meets the standards defined by the product team and is safe and free of vulnerabilities before it can be integrated main stream. Code needs to be scanned especially from a security standpoint at this static level, to ensure no malicious snippets are being integrated. Such static analysis can either be done manually (not a scalable and fool proof option though) or with the use of home grown or commercial code scanners. Such scanners run through all possible execution paths of the code and spit out issues / errors (including critical ones), warnings, vulnerabilities etc. This is a great checkpoint before code integration happens and is also one of the right situations at which Service Level Metrics can be enforced

Dynamic Analysis of code: At the dynamic level when the code is run, the following checks become important at the black and white box levels:

• API level checks – automated tests at the interface level and database level to ensure interfaces work fine, are secure, have the right schemas and are consistent with the core product architecture and technology
• Performance Testing – check for configuration files, setup files etc. to ensure the right parameters are being used. If the integrated external code is not on par with the core code, it can often pull down the performance of the entire product. This is also  one of the right stages for the product team to mandate SLAs to the external team
• Functional testing and UI testing at the black box level to ensure the E2E functionality works fine, the UI is consistent overall and that the integration points are not apparent to the end user. Solid integration testing herein needs to be undertaken to ensure a seamless end user experience. If the external team has done specific customizations to align with your product line, such customizations also need to be tested for
• Security Testing – This is one of the most important aspects to check for to ensure no vulnerabilities have been introduced, especially when third party code (on which the product team may not have complete control) is being integrated into your mainstream code

In a nutshell, third party code integrations are becoming the need of the day to enhance your core product’s functionality. Do not be intimidated by the challenges it poses. Be aware of them and attack them at the very early stages to fully benefit from such external code; for which, as you have seen above, quality assurance right from the early stages is a very important wand!

In an earlier blog a few weeks back, I had talked about defect metrics that help determine the quality of test effort. In the current blog, I will be discussing what defect metrics will serve as good indicators of the quality of product under test. While both these sets of metrics are equally important to track and base important test management decisions on, this current set is all the more important in helping a test manager decide “whether the test team is ready to sign off on the product under test”. This set of metrics bring in objectivity in making this decision more accurate and reliable, rather than when based on subjective factors. Also, as a best practice, I would recommend using these metrics on an ongoing basis (say once in 15 days or so) to keep track of how the product quality is shaping up. This will help the test management proactively recommend any corrective actions to the product team as opposed to reactively accepting poor quality later in the game. Here we go, with the magic set of metrics that I have been alluding to.

1. Number and timing of high priority and severity bugs reported – High priority and severity bugs typically indicate a major crash or failure in product functionality. These strongly denote the health of the product regardless of when in the product life cycle they are reported. These are especially critical warranting a closer look at product quality, if found later in the product life cycle. The product triage team pays very close attention to high priority and severity bugs throughout the life cycle for these right reasons, and in addition, if the test manager notices any patterns/trends such as high pri/sev bugs concentrated in one module, showing up too frequently, showing up too late in the game etc. it would be worthwhile to further analyze and report findings to the product triage team.

2. Number of regressions – The higher the number of regressions, the more the product quality needs attention to. Such regression numbers often indicate the performance of the development team, but more importantly their performance is directly indicative of the product quality. Regressions are often strong indicators of the comprehensiveness and fool-proofness of bug fixes that are checked in. This is a very good discussion point the test manager can take to the round table to highlight the need to improve product quality

3. Number of build breaks or Build Verification Test (BVT) failures – Since these are basic tests that are performed either by the development or test team, to qualify a build for further testing, these are often tests that represent the product’s core functionality or the bare minimum “must run scenarios”. Also, these tests once built at the beginning at the product life cycle remain quite static and only tend to grow with additional functionality that gets developed. Given that these are tests that are run over and over with every newly released build, it is an expectation to hit 100% BVT pass percentage in all builds. Frequent build breaks not only waste the product team’s time in resolving them but more importantly are indicative of the poor build quality that needs immediate attention.

4. Number of functional, performance and security bugs – Bugs of any and all types are valuable to help improve product quality. However, the ones that are most indicative of product quality are bugs that are reported in areas of product functionality, performance and security. Cosmetic bugs such as UI, usability etc. are also important to be looked at but often times these are the ones that can be lived with, whereas this other category of bugs discussed here are ones that cannot be compromised or will at least need serious collaborative consensus before product release. The test manager should specifically focus on these bugs and probably further dissect this data from various views such as: number present in a given module, a combination of these bugs in a given module, priority/severity of these bugs, when in the product life cycle do these show up, potential regressions these may introduce etc.

5. If product is post v1, number of post release patches, bugs – If the product under test is not a V1 product there will be several useful data points from past releases that the test manager can leverage to track product quality in the current release, which include how the product fared in past releases (both pre and post release), the number and kind of emergency patches that were required post release, defect metrics from past releases etc. as a start point.

Herein above, I have discussed core metrics that can be tracked and acted upon to monitor and improve product quality. Obviously, this is by no means an exhaustive list but is a great start towards having the test team move towards the driver’s seat in proactively playing an important role in owning product quality. Feel free to send me your comments and experiences from your test and/or development assignments, which helped you ship a product of great quality.

With strong support from the government and private companies, Education is one of the core domains where a lot of research has been going on in the US, over the last few years. There is a greater push to strengthen the educational foundation by leveraging technology in possible places. Obama’s government has pledged $118 million to be raised through private companies to fund for educational research. As recent as in July 2011, Microsoft has invested $15 million for research in use of gaming in education.

Couple of promising avenues to introduce technology in education is the use of videos and games. These are being viewed as great tools to make education more effective without overwhelming the learning process. Some facts to substantiate this are as seen below (Source: several online sources call out these stats, dating back to 2006, from an original research conducted by ESA – Entertainment Software Association):

1. 35% American parents play games and 66% believe games bring their families closer together
2. 69% of American households play computer and video games
3. Gaming surpasses age bounds. Average gamer’s age is 33.

With this level of penetration that games and video have in our daily lives, educational researchers are seeing clear benefits of introducing them in enhancing learning for all age groups. Major benefits they see include: Improved analytical thinking, problem solving skills, communication and articulation skills, subject matter retention and attention span. With increase in use of software games and video in education, comes in a whole new career avenue for software testers. What can you as a software tester do to prepare yourself to fit such jobs and what can software testing companies do to build a line of expertise in this area?

There are three parts to building a career in *Testing for Educational Gaming* viz. Testing, Educational domain knowledge, Gaming domain knowledge. Before you invest time, effort and money to train yourself in this space, there are some core traits to self-evaluate yourself on, to candidly analyze your fitment for this job. Such traits include: Analytical skills, passion for games and video entertainment, interactive working, a perennial learner and problem solver. If you see a pattern here, some of these are core traits for a regular software tester as well. Of the three parts I’ve outlined above, I would prioritize them in the following order (highest to lowest), for you to work on, to be successful in this career: Gamer, Tester, Educator. My pick for this order is because once you have the core gaming skills, you have automatically acquired some core testing skills as well. Typically a great gamer is a great tester to start off with, as he is always looking for ways to win over the game he is playing, exploring all routes to score the most, focusing on finishing the game in the fastest possible time etc. These translate to the person really understanding the functionality and the performance of the game and a core aspect of out of box thinking which goes a long way in grooming him as a successful tester. So, once you are a great gamer to begin with the other pieces of the puzzle automatically fall in place. If one works on this foundation to strengthen his/her software testing skills and also builds core domain knowledge in the specific educational discipline that he/she would be working on, the future looks very bright. Educational domain knowledge can often be built on the job, unless the job calls for some very specific expertise with people who have been formally trained in that area. One can pursue this job even part time, by working a few hours which can be beneficial to both the employer and the employee by providing time and cost flexibility, yet bringing in the domain knowledge needed to test the product.

As for a software testing company looking to build this expertise, I would strongly recommend hiring avid gamers and train them on software testing. At QA InfoTech, we’ve found this route very successful in several areas of testing in the past – one being, building a team to test call center software, where we actually hired support engineers and trained them on software testing. Clearly this is a robust strategy to build such a talent pool in a company. Gaming in Education is an up and coming trend with a lot of evolution in the offing this coming decade. This is definitely an area that will pay off if invested in, at this point of time.