Test tools can be a long-term investment, perhaps extending over many iterations of a single project, and/or applicable to many projects. A prospective tool must therefore be considered from different viewpoints.

• For the senior management, a positive ROI is required.
• For the support and operations team, a limited, but necessary number of tools used across the organization is preferred. Maintaining a larger number of tools, keeping track of their licenses, and managing the tool stack should not be cost or time-consuming.
• For the project leads, the tool must add measurable value to the project or organization.
• For the people who use the tool, usability is very important. Usability includes, for example, support for given tasks, learnability, and operability.
• For the operational staff members, maintainability is important.

Features must be analyzed for each business and technical type of tool. Different perspectives and interests have an influence on that analysis: test management, (technical) test analysis, test automation or development. The person in the organization that is responsible for the tool (the tool owner) must make sure the analysis is accomplished, and the above-mentioned bullet points are considered.

All tools introduced into the test process should also ensure a positive ROI to the organization. It is the responsibility of the test manager to take care of the calculation and further evaluation of the ROI. In Agile software development it may be the responsibility of the entire development team.

A cost-benefit analysis should be performed before acquiring or building a tool to ensure it benefits the organization. This analysis should take both recurring and non-recurring costs into account.

Non-recurring activities and costs include the following:

• Defining and determining tool requirements to meet objectives
• Evaluating and selecting the correct tool and vendor, proof of concept
• Purchasing, adapting or developing the tool for initial usage
• Defining guidelines for the usage of the tool
• Initial training for the tool
• Integrating the tool into the existing tool landscape
• Procuring hardware/software needed to support the tool

Recurring activities and costs include the following:

• Recurring licensing and support fees
• Maintenance costs
• Ongoing training costs
• Porting the tool to different environments

Opportunity costs must also be considered. This means that the time spent on evaluating, administering, training, and using the tool could have instead been spent on actual test tasks. Therefore, more test resources may be needed before the tool can be used for intended activities.

The following risks regarding ROI should be considered when selecting tools:

• Immaturity of the organization can lead to inefficient use of the tool
• Changes in the maintenance policy of the vendor can increase workload
• Higher costs than expected
• Lower benefit than expected

The following benefits may apply to test tools:

• Reduction of manual repetitive work (e.g., regression testing)
  • Speed-up of test cycle-time through automation
  • Saving test execution costs by a decrease in manual activities
  • Increased coverage for certain test types supported by the tool
• Reduction in human error due to fewer manual activities
• Quicker access to information about tests

Additional benefits and risks, especially for test automation tools, can be found in the ISTQB® Foundation Level Syllabus version 4 and ISTQB® Test Automation Engineer Syllabus.

In general, a test organization rarely uses a single tool. The total ROI for an organization is usually a mix of the ROI of all tools that are used for testing. Tools need to share information and work cooperatively. A long-term, comprehensive strategy for test tools that includes risks, costs, and benefits is advisable.