Human beings make errors (mistakes), which produce defects (faults, bugs), which in turn may result in failures. Humans make errors for various reasons, such as time pressure, complexity of work products, processes, infrastructure or interactions, or simply because they are tired or lack adequate training.

Defects can be found in documentation, such as a requirements specification or a test script, in source code, or in a supporting work product such as a build file. Defects in work products produced earlier in the SDLC, if undetected, often lead to defective work products later in the lifecycle. If a defect in code is executed, the system may fail to do what it should do, or do something it shouldn’t, causing a failure. Some defects will always result in a failure if executed, while others will only result in a failure in specific circumstances, and some may never result in a failure.

Errors and defects are not the only cause of failures. Failures can also be caused by environmental conditions, such as when radiation or electromagnetic fields cause defects in firmware.

A root cause is a fundamental reason for the occurrence of a problem (e.g., a situation that leads to an error). Root causes are identified through root cause analysis, which is typically performed when a failure occurs or a defect is identified. It is believed that further similar failures or defects can be prevented or their frequency reduced by addressing the root cause, such as by removing it.

It is necessary to understand there are degrees of failure. Just because a security vulnerability is not detected and resolved, it doesn’t necessarily mean the security testing approach failed. There are too many possible security vulnerabilities, with new ones being discovered daily. However, there are other cases where security test approaches have been inadequate to effectively identify security risks, which have led to sensitive data and other digital assets being compromised.

Root cause analysis can help identify why a security testing approach may have failed. Possible causes include:

• Lack of executive leadership in establishing security testing
• Lack of executive provision of resources needed to implement the security testing strategy (such as lack of funding, lack of time, lack of resources)
• Lack of effective implementation of the security testing approach (such as the lack of skills needed to perform the required tasks)
• Lack of organizational understanding and support of the security testing approach
• Lack of stakeholder understanding and support of the security testing approach
• Lack of understanding of security risks
• Lack of alignment between the testing approach and the organization’s security policy
• Lack of alignment between the testing approach and the organization’s security testing policy and strategy
• Lack of understanding of the purpose of the system
• Lack of technical information about the system (causing wrong assumptions)
• Lack of effective tools for security testing
• Lack of skills in security testing

The benefits of testing are offset by quality costs. A means of quantifying the total cost of quality-related efforts and defects is called cost of quality. Cost of quality involves classifying project and operational costs into four categories related to product defect costs:

• Defect prevention costs: The cost of all activities that are planned and proactive to prevent poor quality (e.g., qualification of the developers for their tasks such as training in the creation of maintainable or secure code, reviewing the test basis as early as possible, and appropriate communication within the team)
• Appraisal costs: The cost of all activities aimed at defect detection (e.g., performing static testing and dynamic testing and reviewing work products)
• Internal failure costs: The cost of all reactive activities (e.g., fixing defects found during testing, providing workarounds)
• External failure costs: The cost of all non-value added and reactive activities (e.g., loss of either revenue, assets, human health, human life, or the environment, legal costs related to defect fixing, testing, deployment, and support because of defective product being delivered to the customer (“post release”), fixing field defects (flagged by customers)).

The total appraisal costs and internal failure costs are usually significantly less than the external failure costs. This therefore makes testing extremely valuable. By determining the costs in these four categories, test managers can create a convincing business case for testing.

There are more approaches that can be considered for defining the cost of quality. The ISTQB® syllabus supports two of them. This syllabus is based on the Feigenbaum’s approach, and the ISTQB® Foundation Level Syllabus V.4 presents Boehm’s approach (see the ISTQB® Foundation Level Syllabus V.4, Section 1.3, Testing Principles). These two approaches have been selected to reach a broader understanding of the cost of quality. Feigenbaum’s approach (Feigenbaum, Nov/Dec 1956) considers quality as a customer-oriented and company-wide process, while Boehm’s approach (Boehm, 1979) focuses on the trade-off between the cost of prevention and the cost of failure in software development (Hadjicostas, 2004).

Piloting a proposed improvement is an effective way of reducing the risk of failure, gaining experience, building support and reducing the risk of implementation failure. This is especially important where those improvements involve major changes to working practices or place a heavy demand on resources.

Selection of a pilot should balance the following factors:

• Realism - Is the pilot representative of the “real world”? Care should be taken not to select a pilot which is unrealistic just because it offers the chance of a particularly quick or easy implementation.
• Scalability of solution - Can the results from the pilot be used in all contexts? If the pilot is not representative of the complexity and size of real projects there is a risk that the implemented improvement will not scale.
• Impact on current projects - Pilots should not be performed on current projects unless the impact is acceptable. Particular care is required if existing practices are to be replaced by the improved practices for the duration of the pilot. A better solution is to run the new practices in parallel with the existing practices, although this may create a resource problem (we cannot expect project employees to perform the same task twice because of the pilot).
• Risk of failure - Even though the use of pilots is a risk-reduction measure, the risk of pilot failure must also be evaluated. The above-mentioned aspects are significant factors in the evaluation of the pilot, which should consider both the financial and motivational risks of failure to the entire improvement project.