70 results found
Search Syntax
"regression testing" Exact phrase match
term1, term2 Search for multiple terms (OR)
test plan OR test strategy Same as the comma: either term (OR in capitals)
-keyword Exclude results with keyword
"test plan", "test strategy" Multiple exact phrases
🧠 Smart Search powered by AI semantic matching
ISTQB® Knowledge Assistant
Analyzing your query...
AI-generated summary based on ISTQB syllabus content. Always verify with source material.
Results Overview
70 results
Chapter Select a syllabus first
K-Level
Sort
Refine
Type a keyword to narrow down the results already shown. This filters by title, content, and section — no new search is performed.
 
CTAL-TM v3.0 Ch.2 – Managing the Product

2.3.5 – Defect Report Information

TM-2.3.5: Use the data and classification information that should be gathered during defect management
Related exam questions
K3 Apply 90% match
Go to Syllabus p.59

The information on a defect report should suffice for the following purposes:

  • Management of the defect report through the defect lifecycle
  • Assessment of overall project status, especially in terms of product quality, and test progress
  • Assessment of the status of a product increment in terms of product quality
  • Assessment of process capability

The information needed for defect management and project status can vary depending on when the defect is detected in the SDLC. In addition, defect reports related to non-functional quality characteristics may need more information (e.g., load conditions for performance issues). However, the core information gathered should be consistent across the SDLC and ideally across all projects in an organization to allow for meaningful comparison of defect data throughout the project and across all projects.

Many data items can be collected in a defect report. The test manager should decide which information is appropriate for effective defect management for a given project context. Due to the fact that each additional attribute increases the time spent on defect reporting and may increase confusion by the person who is entering the defect report, it is advisable to only collect data that is needed for defect management in the given context and/or will be used for process improvement.

To manage the defect report in most environments, the following are mandatory:

  • A defect title with a short summary of the anomaly
  • A detailed description of the anomaly preferably including steps to reproduce the failure
  • Severity of the impact on the system under test and/or the product stakeholders
  • Priority to fix the anomaly

Additional important data items are often created by the defect management tool:

  • Unique identifier for the defect report
  • Date/time of creation of the defect report
  • Name of the person who discovered and/or reported the anomaly
  • Project and SDLC phase in which the anomaly was discovered
  • Current state of the defect report
  • Current owner (i.e., the person currently assigned to work on the defect)
  • Change history such as the sequence of actions, including date/time information, taken by project team members to isolate, repair and confirm the defect as fixed
  • References (e.g. to test case, to connected defects).

Depending on context, further information (e.g. traceability) may also be collected in a defect report (see the ISO/IEC/IEEE 29119-3 for more information). The following bullet points group the information according to the intended purpose:

  • To help defect resolution : The subsystem or component in which the defect lies, the specific test item and its release number in which the anomaly was observed or the test environment in which the defect was observed
  • To assess the overall project status: Information to monitor progress, (e.g., risks, costs, opportunities, and benefits associated with fixing or not fixing the defect, a description of any available workaround, or requirements affected by the defects)
  • To assess the status of a product increment in terms of product quality: The type of defect (usually corresponding to a defect taxonomy), the work product in which the defect was introduced, or the quality characteristic/sub-characteristic affected by the defect
  • To assess the process capability : Information to monitor the effectiveness and efficiency of the development processes (e.g., the SDLC phase of introduction, detection, and removal for the defect or defect root cause)
Show content Hide content
CTFL v4.0.1 Ch.5 – Managing the Test Activities

5.5 – Defect Management

Since one of the major test objectives is to find defects, an established defect management process is essential. Although we refer to "defects" here, the reported anomalies may turn out to be real defects or something else (e.g., false–positive result, change request) - this is resolved during the process of dealing with the defect reports. Anomalies may be reported during any phase of the SDLC and the form depends on the SDLC. At a minimum, the defect management process includes a workflow for handling individual defects or anomalies from their discovery to their closure and rules for their classification. The workflow typically comprises activities to log the reported anomalies, analyze and classify them, decide on a suitable response such as to fix or keep it as it is and finally to close the defect report. The process must be followed by all involved stakeholders. It is advisable to handle defects from static testing (especially static analysis) in a similar way.

Typical defect reports have the following objectives:

  • Provide those responsible for handling and resolving reported defects with sufficient information to resolve the issue
  • Provide a means of tracking the quality of the work product
  • Provide ideas for improvement of the development and test process

A defect report logged during dynamic testing typically includes:

  • Unique identifier
  • Title with a short summary of the anomaly being reported
  • Date when the anomaly was observed, issuing organization, and author, including their role
  • Identification of the test object and test environment
  • Context of the defect (e.g., test case being run, test activity being performed, SDLC phase, and other relevant information such as the test technique, checklist or test data being used)
  • Description of the failure to enable reproduction and resolution including the test steps that detected the anomaly, and any relevant test logs, database dumps, screenshots, or recordings
  • Expected results and actual results
  • Severity of the defect (degree of impact) on the interests of stakeholders or requirements
  • Priority to fix
  • Status of the defect (e.g., open, deferred, duplicate, waiting to be fixed, awaiting confirmation testing, re-opened, closed, rejected)
  • References (e.g., to the test case)

Some of this data may be automatically included when using defect management tools (e.g., identifier, date, author and initial status). Document templates for a defect report and example defect reports can be found in the ISO/IEC/IEEE 29119-3 standard, which refers to defect reports as incident reports.

Show content Hide content
CTAL-TM v3.0 Ch.2 – Managing the Product

2.3.1 – Defect Lifecycle

TM-2.3.1: Implement a defect management process, including the defect workflow, that can be used to monitor and control defects
Related exam questions
K3 Apply 79% match
Go to Syllabus p.55

Each phase of the SDLC should include activities to detect and remove potential defects. For example, static testing techniques (i.e. reviews and static analysis) can be used on design specifications, requirements specifications and code prior to delivering those work products into subsequent activities. The earlier each defect is detected and removed, the lower the overall cost of quality for the product. Cost of quality is minimized when each defect is removed within the same phase in which it was introduced (i.e., when the software process achieves perfect phase containment).

During static testing we search for defects. During dynamic testing the presence of a defect is revealed when it causes a failure, which results in a discrepancy between the actual results and the expected results of a test (i.e., an anomaly). In some cases, a false-negative result occurs when the tester does not observe the anomaly. When an anomaly is observed, further investigation should be performed. This investigation usually starts with completing a defect report in accordance with the defined test and defect management process. A failed test does not always result in the creation of a defect report. (For example, in test-driven development, where component tests, usually automated, are used as a form of executable design specification). Until the development of the component is complete, some or all of the tests must initially fail. Until the development of the component is complete, some or all of the tests must initially fail. Therefore, the result of such a test is not necessarily caused by a defect and is typically not tracked via a defect report.

A defect report progresses along a workflow (for simplicity and consistency with most defect management tools we will further use the term “defect workflow”) and moves through a sequence of defect states. In most of these states, one person owns the defect report and is responsible for carrying out a task (e.g., analysis, defect removal, or confirmation test). The following diagram represents a simple defect workflow:

Figure 2: A Simple Defect Workflow

Figure 2: A Simple Defect Workflow

A simple defect workflow may cover the following defect states:

  • OPEN (may be called NEW): The initial state when the defect report is created.
  • IN PROGRESS: The team is working on the defect report analysis and/or fix.
  • REJECTED: A defect report is rejected by the person who processed it (usually a developer or an analyst). There may be many reasons for rejection (e.g., invalid information, incorrect test, duplicate defect report) and this information is added to the defect report.
  • RESOLVED (may be called FIXED, READY FOR RETEST): A tester runs a confirmation test often following the steps to reproduce the failure from the defect report itself to determine whether the fix has indeed resolved the defect.
  • CLOSED: The defect report has reached its terminal state, and no further work is intended to be done. The tester transitions the defect report to this state either after a successful confirmation test or to acknowledge rejection of the defect report.

A simple defect workflow is used in many organizations and is extended by the use of other defect states relevant for a given context (e.g., RE-OPENED, ACCEPTED, CLARIFICATION, or DEFERRED).

The defect workflow may vary in different organizations in terms of different names of the defect states, rules for transitions among defect states and roles responsible for tasks in given defect states. Often the defect workflow is more simple in Agile software development than in sequential development models. The defect workflow should be adapted to a given context. When designing the defect workflow, it is advisable to respect several good practices:

  • If possible, the defect workflow should be defined organization-wide to provide unified defect management across all projects
  • Duplicate and false-positive defects should be represented by a separate state or a combination of the REJECTED status with choosing the reason for rejection. They may be helpful in further defect analyses with the aim of improving the test process.
  • It is recommended to use only one terminal state (e.g., CLOSED). The transition to this state often requires choosing a reason for closure, useful for process assessment and process improvement activities.
  • The names of states in the defect workflow should be the same as for analogous states used for other entities (e.g., user stories and test tasks) to simplify working with them.
  • Consecutive defect states should belong to different responsible roles. If two or more consecutive states belong to the same responsible role, there should be a good reason (e.g., to measure the time spent in a defect state).
  • Each defect state, except for the terminal state, should have more than one outgoing transition to allow the responsible role a decision regarding the next step. Exceptions of this rule should be justified (e.g., to monitor time spent on a given activity).
  • The set of attributes required to be entered when performing a state transition should be limited to those which give substantial value to defect management.
Show content Hide content
CT-AcT v2019 Ch.5 – Collaborative Acceptance Testing

5.2.1 – Defect Analysis

AcT-5.2.1: Explain how to analyze discrepancies between actual and expected outcomes at the business level in a given context
Related exam questions
K2 Understand 79% match
Go to Syllabus p.31

Testers collect and report discrepancies between actual and expected outcome through defect reports. A defect report contains all relevant information the tester can provide to help the business analyst understand what happened and to assess the deviation.

Defect analysis is a joint activity of testers and business analysts. Usually, the tester identifies the acceptance criteria that are not satisfied. The business analyst may then be asked to analyze its impact on the related business processes. This includes determining the priority of the defect (e.g., low, medium, high, critical) with respect to its potential business impact on system usage.

To analyze the business impact of a defect, the business analyst and tester may do the following:

  • Explore the path(s) in the business process models in which the defect(s) was found
  • Explore the business rules which were not implemented correctly and analyze the priority of the defect from a usage point of view

The impact analysis and the resulting decision regarding further actions to be taken are documented in the defect report.

Show content Hide content
CTAL-TM v3.0 Ch.2 – Managing the Product

2.3.6 – Defining Process Improvement Actions Using Defect Report Information

TM-2.3.6: Explain how defect report statistics can be used to devise process improvement
Related exam questions
K2 Understand 76% match
Go to Syllabus p.60

As discussed in Section 2.3.5 of this syllabus, Defect Report Information, defect reports can be useful for project status monitoring and reporting. While the implications of metrics on the test process are primarily addressed in the Expert Test Management Syllabus, at the Advanced Test Management Level, test managers should be aware of what defect reports mean to assessing the capability of the software development and testing processes.

In addition to the test progress monitoring information mentioned in this syllabus, in section 2.1.2, Monitoring, Control and Completion, and in section 2.1.3, Test Reporting, defect information should support process improvement initiatives as discussed during retrospectives. Examples include:

  • Using information about the phases of introduction, detection, and removal of defects to assess phase containment and/or perform cost of quality analysis with the aim of suggesting ways to improve defect detection effectiveness in each phase and minimize the cost associated with defects
  • Using information about the phase of introduction for analysis of the phases in which the largest number of defects are introduced, to enable targeted improvements for defect prevention
  • Using defect root cause information to determine the underlying reasons for defect introduction, to enable process improvements that reduce the total number of defects
  • Using defect location information to perform defect cluster analysis, to better understand technical risks (for risk-based testing) and to enable re-factoring of troublesome components
  • Using information about re-opened defects to assess the quality of debugging implementations
  • Using information about duplicate and rejected defects to assess the quality of the defect report creation
  • Enable process improvements that reduce the total number of defects by introducing pro-active measures to avoid errors upfront

The use of metrics to assess the test process effectiveness and efficiency is discussed in the Expert Test Management Syllabus.

In some cases, teams decide not to track defects found during some or all phases of the SDLC. While this is often carried out in the name of efficiency and for the sake of reducing process overhead, it greatly reduces visibility into the process capabilities of software development and testing. This makes the improvements suggested above difficult to carry out due to a lack of reliable support data.

Show content Hide content
CTAL-TM v3.0 Ch.2 – Managing the Product

2.3.2 – Cross-functional Defect Management

TM-2.3.2: Explain the process and participants required for effective defect management
Related exam questions
K2 Understand 73% match
Go to Syllabus p.57

Although the test organization and the test manager often own the overall defect management process and the defect management tool, a cross-functional team is generally responsible for managing the defects for a given project. This team, sometimes called the defect management committee, may include the test manager, representatives of development, suppliers, project management, product management or product owner and other stakeholders who have an interest in the software under test.

As anomalies are discovered and entered into the defect management tool, the defect management committee should determine whether each defect report represents a valid defect and whether it should be fixed (and by which party in case several development teams are participating in delivery), rejected, or deferred. This decision requires the defect management committee to consider the benefits, risks and costs associated with fixing the defect. It is beneficial to discuss the consideration in a meeting (often called the triage meeting). If the defect is to be fixed, the team should establish the priority of fixing the defect relative to other tasks. The test manager and test team may be consulted regarding the relative importance of a defect and should provide the available objective information.

On very large projects the appointment of a full-time defect manager may be justified by the effort needed to prepare for and to follow-up on the decisions made by defect management committee meetings, at least during those SDLC phases when testing is at its most intensive. In other situations, several large projects may share a defect manager.

A defect management tool should not be used as a substitute for good communication, nor should a defect management committee be used as a substitute for effective use of a good defect management tool. Communication, adequate tool support, a well-defined defect workflow (incl. defect report properties) and an engaged defect management team are all necessary for effective and efficient defect management.

Show content Hide content
CTAL-TM v3.0 Ch.2 – Managing the Product

2.3.3 – Specifics of Defect Management in Agile Teams

TM-2.3.3: Explain the specifics of defect management in Agile software development
Related exam questions
K2 Understand 72% match
Go to Syllabus p.57

Defect management in organizations using Agile software development is often lightweight and/or less formal than in sequential development models. If Agile teams are co-located or have well established communication means available, information about a defect or failure is often exchanged among testers, customer representatives and developers without a formal defect report. Defect reports should, however, be created for:

  • Defects that block other current sprint activities (i.e., development, testing, or other) and could not be fixed immediately within the Agile team
  • Defects that cannot be resolved within the same iteration. Some Agile teams have a rule of creating a defect report if the defect cannot be resolved during the day the failure is found.
  • Defects which must be resolved by or in cooperation with other teams in multi-team organizations
  • Defects that must be solved by a supplier
  • Defects where a defect report is explicitly requested (e.g., when a developer cannot immediately work on a fix)

Common practice is to add defects that cannot be resolved within the same iteration to the product backlog so that they can be prioritized among other defects and user stories for a later iteration.

Although the foundations of defect management should be set in an organization's test strategy, many aspects including the level of formality, triggers for creation of a defect report, and defect attributes to be captured may be left to agreement among Agile team members. In general, the level of formality of defect management and the approach to creating defect reports should reflect the following:

  • Co-location of team members
  • Distribution of team members across time zones
  • The number of teams that cooperate on product development
  • Maturity of the team(s)
  • Size of the team(s)
  • Risks associated with the product
  • Regulatory, contractual, or other requirements (if and where applicable)

The final decision of the Agile team regarding the details of defect management should always be documented (e.g., with guidelines in a knowledge management tool).

Show content Hide content
CT-STA v1.0 Ch.6 – Interpreting Test Results and Defining Actions

6.1.1 – Key aspects of test report communication

STA-6.1.1: Explain the key aspects of test report communication
K2 Understand 67% match
Go to Syllabus p.44

Reporting security test results requires close cooperation between the STE and STA. The STE is responsible for creating the security test report, and the STA is responsible for following up and prioritizing any corrective actions needed.

Depending on when the security test is performed, critical security vulnerabilities must be discussed immediately by the STE and STA, and actions taken to manage such emergencies according to defined procedures. If a critical defect is found, it is not advisable to wait until a formal security test report is issued. This is irrespective of whether a test environment or the production system was used.

The content and format of a security test report may vary depending on the stakeholders who are to receive it. Developers and architects need as much detail as possible regarding any vulnerability so they can design and implement the needed fixes. To enable this, the security test reports must contain sufficient details to replicate the security vulnerabilities. These detailed security test reports may also contain possible recommendations provided by the security test tools for each security vulnerability found. Those stakeholders who are responsible for making business decisions need high-level information.

This stakeholder management is a critical responsibility for authors of security test reports. While the technical findings may remain unchanged, their presentation must be tailored to the information needs, responsibilities, and decision-making authority of the intended audience. Executive stakeholders typically require a concise overview of business risks, potential impacts, and recommended actions, whereas technical teams need detailed evidence, attack paths, reproduction steps, and remediation guidance. Effective reporting therefore balances accuracy, transparency, and relevance, ensuring that each stakeholder group receives the level of detail necessary to understand the significance of the findings and to take appropriate action. A well-designed security test report translates technical vulnerabilities into stakeholder-specific insights, enabling informed decisions and fostering organizational support for risk mitigation activities.

The STA defines the security test report formats, which determine the sections of the report into which specific test results are to be inserted. In addition, the STA must define communication channels and formats in advance of the report being issued, which includes a distribution list of all those who should receive particular security test reports and states how to send them (e.g., only via encrypted email).

Vulnerability scans report on the presence of publicly known vulnerabilities, known exploits or weaknesses. For example, if the software uses a version of an external dependency with a known vulnerability, the software itself might be vulnerable as well. In this case, looking up the corresponding Common Vulnerabilities and Exposures database (CVE) identifier mentioned with the vulnerability helps avoid ambiguity and reduces the risk of misinterpretation [ISTQB-STE]. Vulnerability details are often publicly available, such as those published in the National Vulnerability Database (NVD) [NVD], which can provide further useful context.

Show content Hide content
CTAL-TM v3.0 Ch.2 – Managing the Product

2.1.3 – Test Reporting

TM-2.1.3: Analyze test results to create test reports that empower stakeholders to make decisions
Related exam questions
K4 Analyze 67% match
Go to Syllabus p.50

Test management should understand how to interpret and use metrics to understand and report test status. For higher levels of testing such as system testing, system integration testing, acceptance testing, and security testing, the primary test basis is typically work products such as requirement specifications, use cases, user stories and product risks. Metrics of structural coverage are more applicable to lower levels of testing such as component testing (e.g. statement coverage) and component integration testing (e.g., interface coverage). While test management may use metrics of code coverage to measure the extent to which their tests exercise the structure of the system under test, reporting of higher-level test results should be tailored to the specific context and needs of the project. For example, in frequently changing environments, code coverage metrics may be useful to monitor the impact of code changes on the test suite and identify potential gaps or risks. In addition, test management should understand that even if component tests and component integration tests achieve 100% of their structural coverage, defects and quality risks remain to be addressed at higher test levels.

The objective of reporting metrics is to provide an immediate understanding of the information for management purposes. Metrics can be reported as a snapshot of a metric at a point in time or as the evolution of a metric over time to evaluate trends.

Product risks, defects, test progress, coverage and related cost and test effort are measured and reported in specific ways at the end of the project.

The following are examples of metrics that can be used for different purposes:

Metrics related to product risks include:

  • Percentage of risks of which all tests passed
  • Percentage of risks of which some or all tests failed
  • Percentage of risks not yet completely tested

These metrics can be used to assess the quality of the test basis and the effectiveness of the test cases in covering the product risks.

Metrics related to defects include:

  • Accumulated number of resolved defects versus the accumulated number of defects
  • Breakdown of the number or percentage of defects categorized by:
    • Test items or components
    • Source of the defect (e.g., requirement specification, new feature, or regression)
    • Test releases
    • Test level or iteration introduced, detected, and removed
    • Priority/severity
  • Root cause
  • Status (e.g., rejected, duplicated, open, closed)

These metrics can be used to monitor the defect detection and resolution process, identify the areas of high defect density or defect severity, and evaluate the test efficiency and effectiveness.

Metrics related to test progress include:

  • Test execution status: Total number of tests planned, implemented, executed, passed, failed, blocked, and skipped
  • Test effort: Number of actual versus planned resource hours devoted to testing

Metrics related to coverage include:

  • Requirements coverage: The percentage of requirements that are covered by test cases
  • Product risk coverage: The percentage of identified product risks that are mitigated by test cases
  • Code coverage: The percentage of code statements, branches, paths, or conditions that are executed by test cases

Metrics related to costs and test effort include:

  • Residual risks for untested components: The potential impact and likelihood of defects in the components that are not tested
  • Test cost: The actual versus planned cost of testing

Additionally, it is useful to combine metrics from different categories (e.g., a metric that shows the correlation between the trends of open defects versus the trends of executed tests, or a metric that shows the quality of the test basis based on the number of defects found in the requirements). When test execution continues and fewer and fewer defects are identified, a decision can be made to terminate the tests. This decision should be based on the reporting of the metrics and the agreed exit criteria.

Show content Hide content
CT-UT v2.0 Ch.5 – Usability Testing

5.6.2 – Usability Findings

UT-5.6.2: Explain how to overcome internal resistance to usability test findings
K2 Understand 67% match
Go to Syllabus p.42

A usability finding is a usability test result that identifies some important issue, or opportunity.

Positive usability findings are important for the following reasons:

  • They make it easier to sell the need for correcting usability issues by giving a balanced view
  • They communicate to the development team which features should not be modified or deleted
  • It enables a complete view of usability to be obtained

A usability defect report should contain the same information as a regular defect report with the possible addition of direct quotes from the user. Defect severity classifications are used to indicate the consequences of the defect on the user experience.

The moderator and the scribe rate usability problems from the test participants' point of view. Sometimes, the severity ratings are allocated in cooperation with a domain expert.

Important parameters that influence severity ratings are:

  • Frequency: How often does the usability defect occur?
  • Impact: How badly does it hurt the user and the user's environment when the usability defect occurs?
  • Persistence: How quickly will users learn to avoid the usability defect?
Show content Hide content
CT-UT v2.0 Ch.5 – Usability Testing

5.8 – Challenges and Frequent Errors

The ten most frequent and serious errors in usability evaluation, in particular usability testing, are:

Type of Error Description
The purpose of the usability evaluation is not clear The purpose should be described in the usability test plan and approved by the stakeholders before the usability evaluation starts. (see section 5.3.1)
Scheduling too late The usability evaluation happens so late in the development process that there is no time to correct any usability defects before the software system is implemented. Note that this mistake is less likely to occur if an agile software development methodology is used, unless the sessions are scheduled at the end of each sprint. If the sessions are only during the hardening sprint, it's likely there will not be enough time available just like any other software development model.
Critical evaluation results ignored  Critical usability evaluation issues are overruled by management due to cost or time restrictions.
Incorrect focus The usability evaluation focuses on minor details like user interface guideline violations and graphic design and does not address serious defects with effectiveness and efficiency.
No post-session interview A short interview after a usability test session is important to summarize the test participant’s impression from the usability test. (see section 5.4)
No engagement of stakeholders The stakeholders should be engaged in various activities of the usability test (e.g., planning, reviewing), as noted in various sections of this chapter.
Less usable usability test report Unusable test report. The usability test report is unusable, for example because it is too long, contains too many findings, or lacks a one-page executive summary
No positive findings reported The usability test report contains no positive usability test findings.
Positive findings are important to get acceptance of the test results.
(see section 5.6.3)
Usability findings are not prioritized The reader of the usability test report must be able to quickly distinguish between critical and minor usability defects. (see section 5.6)
Lack of management support Management support is not available in terms of time and money to perform a good set of test sessions and resolve the defects when found
Show content Hide content
CT-AcT v2019 Ch.5 – Collaborative Acceptance Testing

5.2.2 – Reporting

AcT-5.2.2: Summarize reporting activities for acceptance testing for stakeholders
Related exam questions
K2 Understand 64% match
Go to Syllabus p.31

Reporting activities during acceptance testing address a specific target audience (for example business managers, product managers or domain experts). These stakeholders are experts in the application domain, but they are not always familiar with implementation details. Therefore, information on acceptance test progress, results, and detected defects should be presented without technical details in the language of the target audience.

Using metrics is an important part of reporting test progress. The overall test result is provided in a test summary report. Apart from summarized information on test execution and results of all test phases, the test summary report provides additional information from the impact analysis of open defects. The test summary report also provides an indication of whether the targeted quality criteria have been reached.

Based on the test summary report, decision makers should be able to determine whether the system under test has reached the necessary pre-defined level of quality and may be released to production or not. Several outcomes are possible, including the following:

  • The system may be released “as is” and rolled out without restrictions.
  • The system may be released with reservations. Known errors which have a workaround are communicated to the user and the support team. For defects that represent an important risk, the related feature (or sub-system) may be excluded from roll-out.
  • The system is rejected until critical defects have been fixed and re-tested. Roll-out is either postponed or replaced by alternative solutions.
Show content Hide content
CT-DAT v1.0 Ch.4 – Digital Accessibility Testing Lifecycle

4.3.3 – Best Practices in Reporting and Communicating Test Results

DAT-4.3.3: Apply good practices to report and communicate digital accessibility test results
K3 Apply 62% match
Go to Syllabus p.45

Effective communication is key to confirming that digital accessibility defects are understood and addressed. The goal is to be a collaborative partner, the enabler to overcome the defects, not just a defect finder.

A digital accessibility tester should be able to apply good reporting practices:

  • Foster collaboration and empathy
  • Manage defensive reactions with objective facts
  • Adapt to the organization's digital accessibility maturity
  • Tailor defect report to different audiences
  • Be constructive and educational
  • Use a combination of reporting formats
  • Focus on solutions, not just problems
  • Report positive findings
  • Track and measure progress

Foster collaboration and empathy

Involve cross-functional team members (developers, designers, product managers) directly in the test activities. Inviting them to observe live test sessions - especially end-user testing - is a highly effective way to build empathy and a shared understanding of the barriers users face. Make it easy for them to participate by organizing informal live observation sessions, scheduling them at convenient times, and emphasizing that even observing a partial session is valuable. Communicate clearly and tailor the terminology to the audience, avoiding overly dense digital accessibility jargon when discussing user impact with non-technical stakeholders.

Manage defensive reactions with objective facts

In situations involving conflict or pushback, the tester should rely on objective evidence and applicable requirements. When stakeholders have strong subjective opinions, shift the discussion toward reproducible observations, applicable WCAG success criteria or other requirements, and real-world feedback from users with disabilities.

Adapt to the organization's digital accessibility maturity

The W3C Accessibility Maturity Model [W3C AMM] is non-normative organizational guidance for assessing how accessibility is integrated into organizational processes. It can help a tester tailor communication to organizational maturity, but it is not a product-conformance standard.

Tailor defect report to different audiences

Provide developers with a detailed technical defect report and, where appropriate, suggestions or guidance for remediation. Identify the applicable accessibility requirement or WCAG success criterion when the finding relates to one, while recognizing that some user barriers or quality findings may fall outside WCAG. For product managers or executives, present a concise summary of risk, scope, priorities, and trends. For designers, emphasize early evaluation of visual and interaction patterns such as contrast, focus, structure, and layout.

Be constructive and educational

Beyond reporting findings, testers can provide educational context and point teams to reliable accessibility resources. Explain the user impact and the applicable requirement where relevant, while keeping responsibility for design and implementation decisions with the appropriate project roles.

Use a combination of reporting formats

There are different ways to report defects depending on the project phase and project need. The digital accessibility defects to be reported should use a dedicated defect management system. Log individual, actionable tickets with all the criteria listed in Section 4.3.2 above.

For formal reporting, prepare the type of report required by the context. An internal or project test report can summarize scope, methods, risks, findings, and remediation status. Where procurement or external conformance reporting requires an Accessibility Conformance Report (ACR), use the applicable VPAT® edition or other required format (see Section 4.3.4).

Focus on solutions, not just defects

Work with developers and designers to brainstorm and validate solutions. Help to perform confirmation testing for fixes and provide positive feedback when defects are resolved correctly. Product manager input is as important from the business side as the insights from developers and designers.

Report positive findings

Digital accessibility audits can often feel like a long list of failures for the development team. Always highlight features that work well in summaries and presentations (e.g., “The application's focus management and color contrast are excellent across all pages”). ncluding positive findings provides a balanced view, boosts team morale, and helps developers know which accessible features and patterns should not be modified or deleted.

Track and measure progress

Use dashboards to track the number of open versus closed digital accessibility defects over time. Monitor trends to see if the number of new defects being introduced is decreasing, indicating that the team is learning and improving.

Show content Hide content
CTAL-TM v3.0 Ch.2 – Managing the Product

2.3 – Defect Management Introduction

The ISTQB® Foundation Level Syllabus V.4 describes the activities which begin after observing actual results that differ from expected results. The syllabus refers to these activities as defect management. Other standards use the term “incident management” ISO/IEC/IEEE 29119-3 Standard or “anomaly management” (TMAP) to emphasize the fact that at the beginning of the process we may not know if the discrepancy is caused by a defect in a work product, or due to something else (e.g., test automation failure or a misunderstanding of the requirements by the tester). Defect management and the tool used to manage defects are of critical importance to the testers and to other team members involved in software development. Information from an effective defect management process allows the test team and other project stakeholders to gain insight into the state of a project throughout its SDLC. Defect management is also crucial for deciding which defects will be fixed. This ensures that effort is spent on working with the correct defects. Collecting and analyzing defect-related data over time can help to locate areas of potential improvement both for testing and for other processes within the SDLC (e.g., better defect prevention by improved architecture and technical design).

In addition to understanding the overall defect lifecycle and how it is used to monitor and control both the software development and testing processes, the test manager, and the testers (or whole Agile team in Agile software development) must also be familiar with which data is critical to capture. The test manager must be an advocate of the proper use of both defect management process and the selected defect management tool.

Show content Hide content
CTAL-TA v4.0 Ch.5 – Software Defect Prevention

5.3.1 – Analyzing Test Results to Improve Defect Detection

TA-5.3.1: Analyze test results to identify potential improvements to defect detection
Related exam questions
K4 Analyze 62% match
Go to Syllabus p.52

Test results allow the TA to identify failures, but they also provide feedback to the TA to help improve defect detection effectiveness. Some commonly used techniques to analyze test results are described below.

Predicted versus actual defect cluster analysis . A few components usually contain most defects (see ISTQB-CTFL, v4.0.1, Section 1.3). After testing, the TA can identify the actual defect-prone areas and compare predicted versus actual defect clusters. In the case of discrepancies, more rigorous testing may be applied to areas where more defects were found than expected. When determining the clusters, measurable criteria should ensure clarity and consistency (e.g., defect density and defect severity). Among these criteria, the severity of the defects should play a significant role. Small clusters of critical or major defects are usually more important (i.e., need more rigorous testing) than larger clusters of minor or cosmetic defects.

Defect detection percentage (DDP) analysis . DDP is one of the most important test effectiveness measures for a test level. When calculating DDP, the number of escaped defects should be limited to those that the test level under consideration could have detected. Clear boundaries for defect counting, such as temporal limits (e.g., defects found within a defined time frame after release) and exclusion criteria (e.g., defects in third-party components or specific customer environments), should be established to ensure consistency. A low DDP for a given test level indicates a high percentage of escaped defects, meaning defect detection is ineffective. In such a case, the TA should analyze the reasons and propose measures to improve it, making the test level more focused and rigorous. The DDP is best divided by severity levels, as the priority of reducing escaped defects usually depends on their severity.

Structural coverage analysis assesses the extent to which tests have exercised specific areas of the test object. Identifying low-coverage areas allows the TA to target test efforts in those areas. Increasing their coverage helps to discover new, previously escaped defects. Structural coverage such as statement coverage, branch coverage, or neuron coverage is usually measured with test tools. When selecting the additional areas to be covered, their risk levels should be considered.

Test gap analysis assesses the extent to which tests have exercised recent code changes. This allows the TA to focus additional test effort in areas that are especially error prone (i.e., new changes that have not been tested at all) instead of targeting all areas that have low coverage (e.g., code that has not changed in a long time and has been tested for previous releases).

Defect arrival pattern analysis . The number or density of defects found in successive phases of a project (e.g., iterations) can be compared with patterns that describe the theoretical distribution of these values over time. A classic example of a defect arrival pattern is the Rayleigh model (Elsayed, 2021). It has a single peak and is skewed to the right, showing that the expected number of defects found first increases in time and, after reaching its maximum value, drops down slowly towards zero. Based on the analysis of such a pattern, it is possible to infer the strength of existing test cases and the potential for their improvement. For example, suppose defect detection remains at a constant, low level when the pattern suggests it should be increasing. In that case, it may mean that the existing tests are too weak and unable to detect additional defects.

The analytical methods described above use various defect-related metrics, such as the number of defects or DDP. These metrics are calculated based on the test results (e.g., number of tests passed/failed), defect reports, and structural metrics (e.g., code coverage or defect density). Note, however, that these metrics are not always as easy to read from the test results as they may seem. For example, the number of failed tests is not necessarily the same as the number of defects detected by these tests. To calculate the actual number of defects detected, the results of the debugging process must be carefully analyzed because the relation between test results and defects can be many-to-many. Several tests may detect the same defect or one test may detect several defects. Moreover, the severity of the defects may differ from the criticality of the tests, as a critical test case may fail due to a cosmetic defect.

Show content Hide content
CTAL-TA v4.0 Ch.5 – Software Defect Prevention

5.3.2 – Supporting Root Cause Analysis with Defect Classification

TA-5.3.2: Explain how defect classification supports root cause analysis
Related exam questions
K2 Understand 60% match
Go to Syllabus p.53

Root cause analysis (RCA) is a technique for identifying and addressing the underlying or fundamental causes of a defect rather than only its symptoms. RCA supports a structured approach to quality improvement, and its primary objective is to prevent the recurrence of defects. The TA uses various techniques to identify root causes of defects and failures (e.g., defect taxonomies, the five whys technique, cause-effect diagrams, and Pareto analysis).

The classical RCA involves subject matter experts studying a defect in considerable detail after resolving it. However, there are usually many defects that need to be analyzed. Therefore, it would be very inefficient and time-consuming to have preventive action planning for each defect. One way to approach this problem is to classify defects and then perform the RCA for the defect types occurring.

Defect classification is based on the recognition that individual defects capture a great deal of information about the development process and the system under test. Defect classification allows the TA to extract information about various aspects of the development process from the defect and turn it into a process measurement. This, in turn, gives an insight into the types of errors made during development, which is helpful for process improvement. Defect classification bridges the gap between quantitative defect statistics and qualitative RCA. To effectively support RCA, the defects should be uniformly classified throughout the entire SDLC, from early testing to production.

The TA should support their organization in standardizing software defect classification. This will improve communication and the exchange of information regarding defects among developers and organizations, facilitating the RCA.

Examples of defect classification methods are:

  • orthogonal defect classification (ODC) (Chillarege, 1992), which classifies each defect into orthogonal (i.e., mutually exclusive) attributes, collected both when a defect is reported and when it is fixed
  • IEEE 1044 ( IEEE 1044 , 2009), a standard classification for software anomalies providing a core set of attributes for the classification of failures and defects
  • severity-based classification, which classifies defects based on their severity (e.g., critical, major, minor, trivial)
  • defect taxonomy models, such as (Beizer, 1990) or (Catolino et al., 2019)

Defects can also be mapped to quality attributes using software quality models such as ( ISO/IEC 25010 , 2023) or the FURPS model (Grady et al., 1987).

More information on RCA can be found in (ISTQB-ITP, v1.0).

Show content Hide content
CTFL v4.0.1 Ch.3 – Static Testing

3.2.2 – Review Process Activities

FL-3.2.2: Summarize the activities of the review process
Related exam questions
K2 Understand 60% match
Go to Syllabus p.35

The ISO/IEC 20246 standard defines a generic review process that provides a structured but flexible framework from which a specific review process may be tailored to a particular situation. If the required review is more formal, then more of the tasks described for the different activities will be needed.

The size of many work products makes them too large to be covered by a single review. The review process may be invoked multiple times to complete the review for the entire work product.

The activities in the review process are:

  • Planning. During the planning phase, the scope of the review, which comprises the purpose, the work product to be reviewed, quality characteristics to be evaluated, areas to focus on, exit criteria, supporting information such as standards, effort and the timeframes for the review, shall be defined.
  • Review initiation. During review initiation, the goal is to make sure that everyone and everything involved is prepared to start the review. This includes making sure that every participant has access to the work product under review, understands their role and responsibilities and receives everything needed to perform the review.
  • Individual review. Every reviewer performs an individual review to assess the quality of the work product under review, and to identify anomalies, recommendations, and questions by applying one or more review techniques (e.g., checklist-based reviewing, scenario-based reviewing). The ISO/IEC 20246 standard provides more depth on different review techniques. The reviewers log all their identified anomalies, recommendations, and questions.
  • Communication and analysis. Since the anomalies identified during a review are not necessarily defects, all these anomalies need to be analyzed and discussed. For every anomaly, the decision should be made on its status, ownership and required actions. This is typically done in a review meeting, during which the participants also decide what the quality level of reviewed work product is and what follow-up actions are required. A follow-up review may be required to complete actions.
  • Fixing and reporting. For every defect, a defect report should be created so that corrective actions can be followed up. Once the exit criteria are reached, the work product can be accepted. The review results are reported.
Show content Hide content
CTFL v4.0.1 Ch.1 – Fundamentals of Testing

1.4.3 – Testware

FL-1.4.3: Differentiate the testware that supports the test activities
Related exam questions
K2 Understand 60% match
Go to Syllabus p.20

Testware is created as output work products from the test activities described in section 1.4.1. There is a significant variation in how different organizations produce, shape, name, organize and manage their work products. Proper configuration management (see section 5.4) ensures consistency and integrity of work products. The following list of work products is not exhaustive:

  • Test planning work products include: test plan, test schedule, risk register, entry criteria and exit criteria (see section 5.1). Risk register is a list of risks together with risk likelihood, risk impact and information about risk mitigation (see section 5.2). Test schedule, risk register, entry criteria and exit criteria are often a part of the test plan.
  • Test monitoring and test control work products include: test progress reports (see section 5.3.2), documentation of control directives (see section 5.3) and information about risks (see section 5.2).
  • Test analysis work products include: (prioritized) test conditions (e.g., acceptance criteria, see section 4.5.2), and defect reports regarding defects in the test basis (if not fixed directly).
  • Test design work products include: (prioritized) test cases, test charters, coverage items, test data requirements and test environment requirements.
  • Test implementation work products include: test procedures, manual and automated test scripts, test suites, test data, test execution schedule, and test environment items. Examples of test environment items include: stubs, drivers, simulators, and service virtualizations.
  • Test execution work products include: test logs, and defect reports (see section 5.5).
  • Test completion work products include: test completion report (see section 5.3.2), action items for improvement of subsequent projects or iterations, documented lessons learned, and change requests (e.g., as product backlog items).
Show content Hide content
CT-DAT v1.0 Ch.4 – Digital Accessibility Testing Lifecycle

4.3.4 – The Purpose and Structure of a Formal Digital Accessibility Report

DAT-4.3.4: Summarize the purpose and structure of a formal digital accessibility report
K2 Understand 60% match
Go to Syllabus p.47

While defect management systems manage the day-to-day resolution of individual digital accessibility defects, formal digital accessibility test reports serve a broader purpose. They provide a comprehensive, point-in-time assessment of a product's conformance within a defined scope, against specified accessibility requirements. Such reports should identify the evaluated product/version, scope, applicable standard or guideline, methodology, sample, limitations, and date. These reports support communication of digital accessibility status to stakeholders, regulatory or procurement needs, and remediation planning.

Formal Reporting Formats: Internal versus External

Unlike day-to-day defect reporting, formal reports summarize a defined evaluation for broader stakeholders. The required report type depends on the purpose, such as internal remediation, governance, procurement, or external conformance reporting:

  • Internal Digital Accessibility Evaluation Report. Created for product, engineering, testing, and governance stakeholders. It can summarize the evaluation scope and methods and reference detailed findings maintained in the organization's normal defect or work-management system rather than duplicating every defect in the report.
  • Accessibility Conformance Report (ACR): An ACR documents how an ICT product or service conforms to specified accessibility standards or guidelines. The Information Technology Industry Council (ITI) publishes the Voluntary Product Accessibility Template (VPAT®), which is a template used to create an ACR. Different VPAT editions support reporting against WCAG, Section 508, EN 301 549, or combinations of these requirements. An ACR is not a certification and, by itself, is not proof of legal compliance [VPAT].

Typical ACR structure when using a VPAT®

A completed VPAT-based ACR typically identifies the product and version, report date, contact information, evaluation methods, applicable standards or guidelines, and conformance terminology, followed by requirement-by-requirement conformance information and remarks or explanations. The exact structure depends on the VPAT edition selected.

  • Product and report identification. Identify the product or service, version, report date, and reporting organization.
  • Applicable standards and scope. Identify the VPAT edition and the accessibility standards or guidelines being reported against, together with the scope of the evaluation.
  • Evaluation methods. Describe the methods, tools, assistive technologies, and other evaluation approaches used to support the report.
  • Contact and supporting information. Provide appropriate contact information and any notes needed to understand the evaluation context.
  • Conformance terms. Use the conformance terminology defined by the selected VPAT edition consistently.
  • Requirement-by-requirement results. Record the conformance level and explanatory remarks for each applicable criterion or requirement included in the selected VPAT edition.
Show content Hide content
CT-GenAI v1.1 Ch.2 – Prompt Engineering for Effective Software Testing

2.2.3 – Automated Regression Testing with Generative AI

GenAI-2.2.3: Apply generative AI to automated regression testing
K3 Apply 60% match
Go to Syllabus p.27

As each new iteration or release is completed, the number of regression test cases to be run often increases, making them ideal candidates for automation, particularly in Continuous Integration / Continuous Delivery (CI/CD) pipelines due to the high frequency of test execution. GenAI can streamline this process by assisting in the creation, maintenance, and optimization of automated regression test suites. By dynamically adapting to codebase changes and performing impact analysis, GenAI can identify which areas of the software are most likely to be affected by recent modifications, focusing regression test efforts where they are most needed.

Here are some typical automated regression testing and test reporting activities that can be supported by GenAI prompting:

  • Automated test script implementation with keyword-driven automation : LLMs can be used to implement test scripts based on keyword-driven test automation frameworks, where predefined keywords represent common test steps. GenAI can map these keywords to specific test cases, generate test scripts and assist testers and test automation engineers in their work.
  • Impact analysis and test optimization : GenAI can be used to analyze code changes in order to identify high-risk areas, thereby enabling targeted regression testing where it is most needed.
  • Self-healing and adaptive tests : GenAI can be used to automatically adjust test scripts to handle minor UI or API changes, preventing unnecessary failures from small modifications and ensuring that test suites remain stable over time.
  • Automated test reporting and insights : GenAI enables the generation of detailed, timely available test reports with success metrics, failures, and key insights, providing stakeholders with dashboards that highlight testing trends and offer predictive insights on potential failure points.
  • Enhanced defect reporting and root cause analysi s: GenAI can facilitate the automatic compilation of comprehensive defect reports with test logs, screenshots, and test environment data.

These activities can be applied to a variety of regression tests, including functional and non-functional regression tests. However, the testers must be aware that GenAI can make mistakes. The generated output must therefore be carefully checked, depending on the associated risk (see chapter 3).

Furthermore, GenAI can assist end-to-end GUI and API-based automated regression tests, each with its distinctive challenges and solutions. GUI tests frequently become unstable due to recurrent changes to the user interface. GenAI can automatically adapt test scripts to handle changes like dynamic locators and modified interactions, reducing the need for manual intervention. API regression tests face challenges such as changing request/response formats, endpoints, and authentication. GenAI can adapt test scripts automatically to evolving API specifications and generate diverse test data, maintaining comprehensive coverage and reducing the need for manual updates.

Hands-On Objective 2.2.3a (H2): Practice few-shot prompting to create and manage keyworddriven test scripts

This exercise focuses on developing and automating test scripts for a given web application using a GUI test automation framework. The exercise is structured into two main sections: test automation and test script debugging. The first part of the exercise provides guidance on creating documentation for a keyword library, generating initial test scripts, having AI validate these test scripts, and expanding the coverage with additional test scripts. The second part places an emphasis on debugging support, using system prompts to create an AI assistant that can check and correct test scripts.

This exercise combines traditional test automation with AI-assisted validation, demonstrating how fewshot prompting can be effectively used to create, maintain, and debug keyword-driven test scripts.

Hands-On Objective 2.2.3b (H2): Practice writing structured prompts for test report analysis in the context of regression testing

This exercise illustrates a methodical approach to analyzing regression test reports, utilizing structured prompts. The process begins with an analysis of the provided test results and a comparison with the test specification. It then progresses to the clustering of similar defects, the maintenance of a known anomalies list, and a cross-checking of findings. Each step is linked to the next one in a single LLM conversation.

The step-by-step approach demonstrates how structured prompts can be used to transform regression test results and test logs into actionable insights, thereby supporting effective test report analysis in the context of regression testing.

Show content Hide content
CTFL v4.0.1 Ch.1 – Fundamentals of Testing

1.2.3 – Errors, Defects, Failures, and Root Causes

FL-1.2.3: Distinguish between root cause, error, defect, and failure
Related exam questions
K2 Understand 59% match
Go to Syllabus p.17

Human beings make errors (mistakes), which produce defects (faults, bugs), which in turn may result in failures. Humans make errors for various reasons, such as time pressure, complexity of work products, processes, infrastructure or interactions, or simply because they are tired or lack adequate training.

Defects can be found in documentation, such as a requirements specification or a test script, in source code, or in a supporting work product such as a build file. Defects in work products produced earlier in the SDLC, if undetected, often lead to defective work products later in the lifecycle. If a defect in code is executed, the system may fail to do what it should do, or do something it shouldn’t, causing a failure. Some defects will always result in a failure if executed, while others will only result in a failure in specific circumstances, and some may never result in a failure.

Errors and defects are not the only cause of failures. Failures can also be caused by environmental conditions, such as when radiation or electromagnetic fields cause defects in firmware.

A root cause is a fundamental reason for the occurrence of a problem (e.g., a situation that leads to an error). Root causes are identified through root cause analysis, which is typically performed when a failure occurs or a defect is identified. It is believed that further similar failures or defects can be prevented or their frequency reduced by addressing the root cause, such as by removing it.

Show content Hide content
CTAL-TAE v2.0 Ch.6 – Test Automation Reporting and Metrics

6.1.3 – Explain How a Test Progress Report is Constructed and Published

TAE-6.1.3: Explain how a test progress report is constructed and published
Related exam questions
K2 Understand 58% match
Go to Syllabus p.41

The test logs give detailed information about the test steps, actions to take, and expected responses of a test case and/or test suite. However, the test logs alone cannot provide a good overview of the overall test results. For this, it is necessary to have test reporting functionality. After the execution of a test suite, a concise test progress report must be created and published. A report generator can be used for this.

Content of a test progress report

The test progress report must contain the test results, SUT information, and documentation of the test environment in which the tests were run in a format appropriate for each of the stakeholders.

It is necessary to know which tests have failed and the reasons for failure. To make troubleshooting easier, it is important to know the test execution history and who reported it (i.e., generally the person who created or last updated it). The person responsible needs to investigate the cause of the failure, report the defect related to it, follow-up on the fix to the defect and test that the fix has been correctly implemented.

Test reporting is also used to diagnose any failures of the TAF components.

Publishing the test reports

The test report should be published to all relevant stakeholders. It can be uploaded on a website, in the cloud or on the premises, sent to a mailing list or uploaded to another tool such as a test management tool. This helps ensure that reports will be reviewed and analyzed if individuals are subscribed to receive them by email or through chat messages posted by a chatbot.

An option is to identify problematic parts of the SUT, and keep a history of the test reports, so that statistics about test cases or test suites with frequent regressions can be gathered for trend analysis.

Stakeholders to report to include:

  • Management stakeholders
    • Typical roles: solution or enterprise architect, project/delivery manager, program manager, test manager, or test director
  • Operational stakeholders
  • Typical roles: product owner/manager, business representative, or business analyst
  • Technical stakeholders
    • Typical roles: team leader, scrum master, web administrator, database developer/administrator, test leader, TAE, tester, or developer

Test reports may vary in content or detail depending on the recipients. While technical stakeholders may be more interested in lower-level details, management will focus on trends, such as how many test cases were added since the last test run, changes in the pass-fail ratio and the reliability of the TAS and SUT. Operational stakeholders usually put more emphasis on product use related metrics.

Creation of dashboards

Modern reporting tools provide several reporting options through dashboards, colorful charts, detailed log collections and automated test log analysis. There are many available tools in the market to choose from.

These tools support data aggregation from sources such as pipeline execution test logs, project management tools, and code repositories. The visualization of data provided by these tools helps stakeholders see trends and make decisions accordingly. These trends can include defect clusters, increase/decrease in defect propagation to certain test environments, SUT performance degradation, and reliability of builds.

Artificial Intelligence/machine learning analysis of test logs

In recent years, some test automation tools include or are based on machine learning (ML) algorithms. Automated analysis of large amounts of data in test logs helps the TAE reduce time spent finding broken locators, analyzing the reason for test failures (i.e., is it a defect in the SUT or in the TAS?) and grouping common defects for test reporting (see the ISTQB CT-AI Syllabus).

Show content Hide content
CT-STE v1.0.1 Ch.8 – Reporting Security Test Results

8.2 – Identifying and Analyzing Vulnerabilities

It is important to note that the absence of a failed test suite does not mean that the system is without defects. Even passed test suites do not necessarily mean that the examined attack vector cannot be exploited. It simply states that with the test suites used it is not possible to be exploited by an analyzed attack vector.

If a security test fails, a potential vulnerability is identified. The test report should give all evidence needed to repeat the failed test case. A security test report might demonstrate many vulnerabilities. The following steps must be taken before any remedial action is taken:

  • Vulnerability demarcation: Usually a failed test represents a single failed test case and represents one vulnerability. However, several other test cases might exist which show the same vulnerability. For example, if an empty input parameter can be used to take over control of an application, maybe the same behavior can be achieved when using a 100MB file as an input parameter. During the vulnerability demarcation phase, different but similar tests are executed to demarcate the identified vulnerability. This is important for the subsequent risk assessment and must be supported by the STE.
  • Adjusting risk likelihood: This step is performed to double-check the risk likelihood of being able to identify a vulnerability in production. Usually, a security test is not performed on a production system. Even if the test environment is similar, it will never be completely identical to the production environment. In particular, some security controls might be explicitly disabled to allow a security test to be performed. If an SUT demonstrates a vulnerability, it might be obfuscated by other parameters put in place for production. If this is the case, the vulnerability still exists but cannot be directly exploited due to other parameters. This adjustment might change the risk level suggested by the security test and might therefore change the overall necessity to plan risk mitigation actions. The STE is tasked with considering this risk adjustment and taking corresponding actions.
  • Adjusting risk impact:

This step is performed to double-check the possible risk impact arising from the exposure to a vulnerability. Usually, the STE’s focus is on technical aspects, which makes the calculation of business impact difficult to estimate. Especially if the STE reuses impact assessments for identified vulnerabilities from an outside source (see chapter 4, CVSS), they can be imprecise for a specific context. If a vulnerability is identified, the business stakeholders refine the possible risk impact. The vulnerability may be estimated to have no impact from a business perspective (e.g., if the impacted component is seldom used), or it might be considered to have a high level of business criticality. This adjustment might change the risk level suggested by the STE and warrant an update to planned risk mitigation actions.

If all three of the above steps are applied, a clear view can be obtained of the identified vulnerability and its identified risk. The adjustment of risk likelihood and risk impact must consider some important parameters:

  • Close communication with business stakeholders, as they have the final say when discussing possible risk impact
  • Close communication with the operations team, as they have the final say when considering production parameters and how they differ from those used in security testing
  • Even if the existence of the vulnerability in the SUT is clear, stakeholders may try to actively influence the risk adjustment step (by adjusting likelihood or impact)

If the remaining risk level is considered to be too high to go into production or stay in production, a risk mitigation plan should be created. Management is responsible for deciding about the urgency of such risk mitigation plans. The decision can be between the following levels of urgency:

  • Stop operation immediately or stop further go live activities: If the risk level is considered to be too high and cannot be accepted, it can only be avoided by not running the SUT. The level of risk (e.g., high, medium or low) that influences this decision depends on the risk appetite, which is defined as “ the amount of risk that an organization is willing to accept to achieve its objectives ” [CARM22]
  • Continue running the system with intensive monitoring: If the system is too critical or the risk is too critical, the SUT might continue running but is intensively monitored
  • Add risk mitigation actions to the normal release plan:

If the risk can be handled or the system has many strict release constraints, the risk mitigation actions are analyzed and performed but not directly applied to the SUT. Instead, the patched components are added to the normal release cycle to ensure that the next planned release contains the required security patches.

The STE must ensure that confirmation tests and regression tests are performed for each risk mitigation action. The confirmation test should consider the evidence provided in the test report and should also use the lessons learned from the vulnerability demarcation step.

Show content Hide content
CTAL-TA v4.0 Ch.5 – Software Defect Prevention

5.2.1 – Using Models to Detect Defects

TA-5.2.1: Use a model of the test object to detect defects in a specification
Related exam questions
K3 Apply 57% match
Go to Syllabus p.50

Modeling provides abstractions of a system at a certain level of precision and detail. It helps stakeholders better understand the system being developed. As discussed below, modeling can support phase containment in at least three ways:

  • detecting defects in specifications
  • detecting defects in models
  • detecting defects in test objects by using model-based testing

Detecting defects in specifications . Specifications are often provided as informally written text. The TA can formally represent these specifications using models. When creating a model, the test conditions (e.g., requirements) are mapped to model elements and linked to them for traceability. Formalization and visualization of the test conditions in a model efficiently reveal defects such as incompleteness, inconsistencies, or ambiguities. The strength of modeling is that the specification is transformed while reviews check it in its original form. As a result, the TA also contributes to finding appropriate solutions for the defects detected.

Data-based models include domain models and combinatorial testing models. They allow the TA to detect domain defects such as overlapping partitions, gaps in the domain coverage, empty partitions, or missing or incorrect combinations of parameters.

Behavior-based models include CRUD matrices, state-based models, or scenario models. They allow the TA to detect incomplete or inconsistent entity lifecycles, missing or faulty state transitions, deadlocks, endless loops, ambiguous or inconsistent system behavior, or missing exception handling.

Rule-based models like decision tables or metamorphic relations allow the TA to detect defects in business rules, such as omissions, inconsistencies, ambiguities, redundancies, error-prone scenarios, or complex business logic.

Modeling can also detect defects such as inconsistencies in naming, data values (e.g., boundaries), and inputs or outputs. It can also identify missing, incomplete, ambiguous, or unnecessary information.

Detecting defects in models . If the test basis contains models, the TA can analyze these models to detect defects. This syllabus discusses defect detection in three typical models used in software testing: state transition diagrams (see Section 3.2.2 ), activity diagrams (see Section 3.2.3 ), and decision tables (see Section 3.3.1 ). Examples of defects in the models mentioned above include:

  • state transition diagrams – missing/wrong states, improper transitions, incorrect guard conditions or actions, redundant or unreachable states, and nondeterministic behavior
  • activity diagrams – missing, unreachable, or dead-end actions, incorrect order of actions, wrong, non-exclusive, or incomplete guard conditions in decision nodes, missing synchronization points, or improperly synchronized parallel flows that can lead to unintended behavior
  • decision tables – overlapping, inconsistent, or infeasible rules, incompleteness (e.g., missing combinations of conditions or missing actions for a given combination of conditions)

All models can also contain defects such as syntax errors, typos, duplicates, and inconsistent naming of model elements.

Detecting defects by using model-based testing (MBT) . In this test approach, an MBT tool designs and generates test cases and other test design testware based on an MBT model and test selection criteria defined by the TA. MBT is effective in finding anomalies in the specification because it allows for comprehensive coverage and systematic exploration of the expected behavior of the test object according to the MBT model. MBT models, especially the graphical ones, foster communication with stakeholders, creating a common perception and understanding of the test basis. More details on MBT can be found in the Model-Based Tester Syllabus (ISTQB-MBT, v1.1).

Show content Hide content
CTEL-TM v2.0 Ch.3 – Managing Across the Organization

3.1.2 – Value Communication and Stakeholder Engagement

ELTM-3.1.2: Analyze the quantitative and qualitative benefits of the test team using defined metrics and proxies to build a business case for a given stakeholder group
K4 Analyze 57% match
Go to Syllabus p.39

Test managers must be careful not to overstate the causal link between test activities and customer satisfaction. While high-quality testing reduces defects and improves system stability, both of which are prerequisites for positive user experiences, customer satisfaction ultimately depends on a broader set of factors, including correct requirements, market fit, usability, pricing, and support quality. Therefore, test teams should frame their contribution to customer satisfaction as enabling and reducing risk: they determine that the delivered product meets its specified quality characteristics, thereby preventing dissatisfaction caused by defects or poor performance. To measure this contribution, test teams use proxies such as EDRs, production failure frequency, MTTR, and system availability, (i.e., metrics that are empirically associated with customer satisfaction but do not claim exclusivity). This honest framing builds credibility with stakeholders and avoids the trap of promising what testing alone cannot deliver.

Value communication is an essential process that involves a thorough analysis of testing benefits within a specific organizational context. To be effective, this communication must be strategically presented to various stakeholders, considering their unique perspectives and decision-making criteria.

To communicate test value effectively, test managers must distinguish between three types of metrics, each serving a different purpose and audience:

  • Product metrics assess the quality of the delivered software, (e.g., EDRs, system availability, and post-release defects). These metrics are primarily of interest to technical stakeholders and development teams, as they reflect the test results of test activities.
  • Process metrics evaluate the efficiency and effectiveness of the test process itself, (e.g., MTTR, DDPs, test automation ROI, and feedback loop efficiency). These metrics help test managers and process owners identify improvement opportunities and demonstrate the value of test investments.
  • Project metrics track progress against the project plan and support predictability, including schedule adherence, scope clarity, budget compliance, and release confidence. These metrics are essential for communicating with project managers, product owners, and executives who need to make informed decisions about resource allocation and release readiness.

This categorization provides that the right metrics are communicated to the right stakeholders, avoiding information overload and enabling data-driven decision-making. The following sections elaborate on each category and provide guidance on their application in different organizational contexts. Beyond categorizing metrics by the audience, it is useful to distinguish leading from lagging indicators. Lagging indicators are outcome-based and measured after the fact, (e.g., EDR, post-release defects, and customer satisfaction). They confirm past performance but cannot be influenced retrospectively. Leading indicators are predictive and observed during the process, (e.g., DDP, coverage, feedback loop efficiency, learning rate, and psychological safety score). They signal likely future outcomes and let the test manager act before problems materialize. Effective value communication combines both: leading indicators to guide and lagging indicators to confirm.

The metrics mentioned above relate to different phases of the SDLC; their significance only arises from this positioning. Defect-related indicators can be categorized along the SDLC from prevention through detection to resolution:

SDLC Phase Defect concept / activity  Associated metric / proxy Cost-of-Quality Reference 
Requirements / Design Defect prevention (shift left) Reduction of early detected
defects
Prevention costs
Development / Test Defect detection Defect detection / discovery
rate 
Appraisal / internal failure 
Release / Production Escaped defects EDR, production failures External failure costs
Operation / Maintenance Defect resolution MTTR, system availability  External failure costs
Retrospective / Improvement  Recurrence prevention  Reduction in recurring defect
classes, time-to-learn 
Prevention (follow-up period) 

Table 1. Categorization of Defect-related Indicators

These phase-specific measurements become comparable and actionable only through end-to-end traceability across the SDLC: requirements, design work products, test cases, and defects are linked so that a defect can be followed from the requirement in which it originated to the failure in which it was resolved. This requires the underlying tools, (e.g., requirements management, version control, test management, defect management, and continuous integration/continuous delivery (CI/CD)), to be integrated. Full SDLC coverage is addressed in Section 3.5.

Operationalizing metrics using decision thresholds and triggers. Metrics only become actionable when they are linked to clear decision thresholds. Test managers must define, in collaboration with stakeholders, what constitutes good versus critical performance for each metric, and what actions to take when thresholds are breached.

  • EDR: A threshold of > 2% may trigger an RCA and process review; > 5% may halt release and require corrective action.
  • MTTR: A threshold of > 48 hours for critical defects may trigger escalation and test resource reallocation; > 72 hours may require a formal defect review.
  • Production failures: An increase of > 10% per quarter may trigger a quality audit and test process improvement initiative.
  • Coverage: Falling below 80% may trigger additional test design and execution; below 60% may require a release decision review and risk reassessment.
  • Psychological safety score: A score below 3.5/5 may trigger team workshops and leadership coaching; below 3.0/5 requires immediate intervention.

These thresholds should be:

  • Collaboratively defined with stakeholders to promote buy-in and realism
  • Context-dependent. Thresholds that apply to a safety-critical system differ from those for a lowrisk internal tool.
  • Reviewable and adjustable. Thresholds should be revisited periodically as the product and organization evolve.
  • Documented and communicated, so that all stakeholders understand the criteria for decisionmaking and the consequences of exceeding thresholds

By operationalizing metrics through clear thresholds, test managers transform data into actionable intelligence, enabling timely, well-founded decisions that balance quality, risk, and business value.

Quantitative benefit analysis. Quantitative benefits include cost avoidance (e.g., reduced production failures), efficiency gains (e.g., test automation ROI), time-to-market acceleration, and risk mitigation (e.g., quantified through security testing, compliance validation, and operational stability enhancement using established organizational risk assessment and classification criteria). Cost avoidance is demonstrated through defect prevention metrics, where the savings are realized by detecting and resolving defects before release, thereby preventing production failures (i.e., failures caused by defects that escaped into the production environment), customer support costs, and reputation damage. Efficiency gains are evidenced by test automation implementation, process optimization, and test resource utilization enhancements that show a positive ROI for testing (see Section 7.1.2). Time-to-market acceleration is measured through early defect detection, continuous integration support, and increased release confidence.

Qualitative benefit assessment. Beyond the customer-satisfaction contribution discussed above, qualitative benefits include improved system reliability and faster defect resolution, as well as regulatory compliance and audit readiness. The latter protect the internal legitimacy and effectiveness of the test organization and safeguard operational continuity.

Stakeholder-specific communication strategies. Communication should be tailored to each stakeholder group:

  • Executive communication emphasizes business impact, competitive positioning, and strategic risk management that align with organizational priorities and financial objectives.
  • Communication with technical stakeholders focuses on process efficiency, test tool effectiveness, and advancement in quality methodology and test practices, and adherence to budgets.
  • Communication with project managers emphasizes how test strategies improve test schedule predictability, test scope clarity, delivery confidence, and responsiveness, including speed of fixes, updates, and enhancements.
  • Customer-facing communications emphasize QC, reliability guarantees, and user experience enhancement.
  • Communication with AI strategists and data science teams focuses on model performance metrics, data quality and bias considerations, AI security and compliance risks, cost-benefit trade-offs of AI tooling, and the implications of technical debt introduced by AI components. It also addresses the limitations of test capacities when dealing with non-deterministic AI outputs and the need for new quality approaches such as model validation and continuous monitoring.

Benefit presentation techniques. Business cases are developed to connect test activities to business outcomes through real-world examples, case studies, and comparative analysis. Visual representations including dashboards, infographics, and trend analyses are created to support data-driven decision-making. Regular test reporting cycles provide consistent value demonstration while building stakeholders' confidence and understanding. Baseline measurements enable before-and-after comparisons that demonstrate test impact and continuous improvement.

Value proposition development. Test managers create comprehensive business cases to substantiate and prioritize test investments. These include cost-benefit analysis considering the CoQ (i.e., defect prevention, appraisal, internal failure, and external failure costs), risk assessment with quantification of potential failure costs, and competitive comparison that support test resource allocation decisions. Scenario modeling is used to demonstrate test value under different project conditions (e.g., higher coverage versus reduced time-to-market and avoided revenue loss through early defect detection) and organizational contexts.

Value communication also leverages predictive analytics, machine learning insights, and real-time test monitoring capabilities to increase credibility and support evidence-based stakeholder engagement in datadriven organizational cultures.

Business-quality alignment frame. To make quality measurable as a business enabler, test managers use lightweight alignment instruments that connect quality to business objectives and create a common basis for dialogue:

  • Quality value mapping / risk-based test alignment canvas: A one-page canvas that maps business risks, quality characteristics, test focus, and measurement criteria; it is maintained collaboratively in workshops and serves as a living compass for a risk-based test strategy.
  • Business impact scoring for quality metrics: test metrics are weighted by their business impact, distinguishing leading from lagging indicators, so that the most consequential quality signals receive the most attention.

These frameworks promote dialogue-based coordination and make quality a measurable business enabler.

Show content Hide content
CT-DAT v1.0 Ch.3 – Digital Accessibility Testing Activities

3.4.2 – Common Software Defects in Digital Accessibility Testing

DAT-3.4.2: Explain common software defects identified during digital accessibility testing
K2 Understand 57% match
Go to Syllabus p.36

The most frequent technical defects identified during digital accessibility evaluation include:

Table 7: Common Digital Accessibility Software Defects

Common Defect Description
Incorrect or unnecessary ARIA
[W3C ARIA]
ARIA can create barriers when roles, states, properties, or required interaction behavior are implemented incorrectly. Prefer
native semantic controls when they meet the need; when custom controls are required, implement and test the applicable ARIA
pattern and keyboard behavior.
Ignoring focus management When a modal closes or the user interface updates dynamically, keyboard focus is lost or reset to the beginning of the view,
causing a loss of context and disrupting keyboard navigation.
Dynamic content blind spots In a single page application, content updates without a page reload, but status changes (e.g., “ tem added to cart”) fail to be
announced programmatically to screen readers.
Vague alternative (alt) text Images have an alt attribute, but the text is not meaningful to the user, e.g., accepting “image123.jpg”, “picture”, or redundant text
like “ mage of logo” as valid.
Zoom and reflow failures At high zoom levels (e.g., 200% or 400%), content overlaps, disappears, or requires horizontal scrolling to be read.
Reliance on visual cues Visual cues (like displayed error messages or red borders) fail to be conveyed programmatically or via text for users who cannot
perceive color or see the interface visually.
Show content Hide content
CTEL-ITP v2011 Ch.4 – Analytical-based Improvement

4.2.3 – Use of Standard Anomaly Classifications

LO-4.2.3: Understand the use of standard anomaly classification for causal analysis
K2 Understand 56% match
Go to Syllabus p.31

Standards such as [IEEE 1044] allow a common classification of anomalies allowing an understanding of the project stages when faults are introduced, the project activities when faults are detected, the cost of rectifying the faults, the cost of failures, and the stage where the defect was raised versus where it was found (also known as Defect Leakage).

A common classification allows statistics about improvement areas to be analyzed across an organization. This classification must be introduced via training, tools and support so that anyone using the incident management system understands when to use the classifications and how to interpret them. This information may be used in the improvement of the development and test process to identify areas where improvement will be cost effective and to track the success of improvement initiatives. The defects to be analyzed will be recorded from all life cycle phases, including maintenance and operation.

Show content Hide content
CTEL-ITP v2011 Ch.4 – Analytical-based Improvement

4.2.4 – Selecting Defects for Causal Analysis

LO-4.2.4: Compare the causal analysis methods
K2 Understand 56% match
Go to Syllabus p.31

The application of methods described above may require the selection of specific defects for analysis from a potentially large collection. The following approaches can be taken in combination to selecting defects for analysis:

  • Pareto analysis (20% of defects considered representative of all)
  • Outliers in statistics (e.g., using values of statistical variation, such a 3-sigma, on a particular metric)
  • Project retrospectives
  • Use of defect severity categories
Show content Hide content
CTAL-TA v4.0 Ch.5 – Software Defect Prevention

5.1 – Defect Prevention Practices

The TA can contribute to defect prevention in several ways, using their domain knowledge, test expertise, and analytical skills. Examples include:

  • Participating in risk analysis – ensuring that identified risks are properly mitigated (e.g., selecting the most adequate test techniques).
  • Reviewing requirements, models, and specifications – allowing early detection of defects in the test basis preventing them from escaping into the code, thus significantly decreasing the cost of fixing them.
  • Participating in retrospectives – allowing the identification of potential improvements in test analysis, test design, test implementation, and test execution (e.g., using more effective test techniques, targeting testing of specific areas of risk, or improving test data and test environments to reduce both false-positive results and false-negative results) to better prevent defects from escaping.
  • Defect data collection and evaluation – supporting root cause analysis and process improvement by collecting detailed data on defects to enable their classification and statistical analysis and thus facilitate root cause analysis.
  • Participating in root cause analysis – preventing defects from reoccurrence by proposing corrective actions to address the identified root causes.

In addition to participating in defect prevention, the TA assesses (usually in consultation with the test manager) whether the proposed measures have resulted in the desired effect. Examples of metrics that help to evaluate the effectiveness of these measures include:

  • Defect removal efficiency (DRE) – measures the ratio of defects removed before the release and the total number of defects. The (unknown) denominator can be estimated or replaced by the total number of defects found up to a certain point in time (e.g., up to 6 months after the release). High DRE indicates fewer defects escape to production, which might imply better defect prevention practices. However, DRE does not distinguish between defects caught through prevention and detection.
  • Phase containment effectiveness (PCE) – measures the number of defects introduced and removed in the same phase relative to the total number of defects introduced in that phase. High PCE indicates that fewer defects escape to later phases.
  • Cost of quality – illustrates the relation between defect prevention, defect detection, and removal costs (see ISTQB-TM, v3.0, Section 3.2.1).
Show content Hide content
CTEL-ITP v2011 Ch.4 – Analytical-based Improvement

4.4.2.2 – Post-release Defect Rate

This indicator is defined as the number of defects found by customers during a certain period after release of the product per Kilo Lines Of Code (KLOC). If this rate decreases then the customers‟ perceived quality will increase.

Note: these defect metrics give information for the “manufacturing” view of quality as was discussed in Chapter 2.

Show content Hide content
CT-STE v1.0.1 Ch.8 – Reporting Security Test Results

8.1 – Security Test Reporting

Each security test ends in a test report [ISTQB Glossary ].Without test reporting, test lacks evidence that can be used to determine actions or decisions based on the test result.

The following standard information is important when reporting a failed security test case:

  • Used test environment: This usually includes specific IP-addresses, applied IP whitelisting, and used accounts/passwords.
  • Preconditions of the executed tests. This includes all preparation activities to be applied before the prepared test suite can be executed. This might include activities such as log-in-details, specific configuration file settings or specific perimeter configurations.
  • Used test data
  • Procedure of test execution
  • Expected results and actual results

A failed security test means a specific test has detected the violation of at least one security aspect from the CIA triad (see Chapter 1). A good test report includes a sufficient level of detail to enable the test to be repeated. Tools used to execute the tests might be named in the test report and screenshots might be included to support the test results with evidence.

In general, security test reports should be handled with high level of confidentiality. If this type of information is leaked outside the organization, it could dramatically reduce the organization’s reputation. Even worse, the information could be used to attack any systems which include this vulnerability.

The more failed tests a security test report contains, the more critical and sensitive the test report and its communication is. In general, every security test report must be communicated with care within the organization. This includes internal communications within the organization producing the SUT, as attackers might come from inside an organization (cf. [SwissCybInst20]). On the other hand, security test reports might be important for many people within an organization. This paradox directly influences the STE’s reporting activities and is usually resolved by creating different versions of the same test report, each of which contains different levels of detail. Each version of the test report should follow the “need to know” concept. This applies to those whose task it is to mitigate identified risks and therefore receive a complete test report or fragments of it based on the “need-to-know” concept.

The sensitivity of a security test report may be modified according to vulnerabilities identified. This is essential when ethical hackers identify a vulnerability in an SUT and wish to inform only the developer to give them the opportunity to mitigate this risk before the test report is made publicly available. This responsible disclosure is one of the characteristics of an ethical hacker, especially for white-hat hackers [Huneidy21]. Grey-hat hackers often use test report publication to increase pressure on an organization to work on patches [Huneidy21].

Show content Hide content
CTAL-TAE v2.0 Ch.6 – Test Automation Reporting and Metrics

6.1.2 – Analyze Data from the Test Automation Solution and the System Under Test to Better Understand Test Results

TAE-6.1.2: Analyze data from the test automation solution and the system under test to better understand results
Related exam questions
K4 Analyze 56% match
Go to Syllabus p.40

After test execution it is important to analyze the test results, to identify possible failure(s) both in the SUT and the TAS. For such an analysis the data collected from the TAS is primary and the data collected from the SUT is secondary.

  • Analyze the test environment data to support proper sizing of test automation (e.g., in the cloud)
    • Clusters and resources (e.g., CPU, and RAM)
    • Single versus multi browser (i.e., cross-browser) test execution
  • Compare test results of previous test executions
  • Determine how to use web logs to monitor software usage

Test execution failures need to be analyzed, as there are potential issues:

  1. Check if the same failure happened in the previous test executions. This might be a known defect either in the SUT or the TAS. The TAS can be built to log historical test results of the test cases, thus further helping the analysis.
  2. If the defect is not known, identify the test case and what it is testing. The test can be selfexplanatory, or the test case can be identified in the test management system, based on its ID logged with the test execution.
  3. Find in which test step of the test case the failure happened. The TAS logs this.
  4. Analyze the test log information about the state of the SUT and whether it matches the expected results using screenshots, API and network logs, or any log that shows the state of the SUT.
  5. If the state of the SUT is not what was expected, log a defect in the defect management system. Make sure to include all the necessary defect information and the logs that justify that it is a defect.

When there is a failure, it is possible for the actual result and the expected result of the SUT to match. In this case, most likely the TAS contains a defect which needs to be fixed, or an invisible mismatch is present.

Another situation that can occur is if the test environment is not available during the test run, or only partially available. In this case, all test cases can fail, either with the same defect or if parts of the system are down, with seemingly real failures. To identify the root cause of such defects, the SUT logs can be analyzed, which will show if there were any test environment outages at the time of the test run.

If the SUT implements audit logs for user interactions (i.e., UI sessions or API calls) it helps to analyze test results. There is usually a unique ID added to the interaction with the same ID for each subsequent call and integration in the system. In this way, knowing the unique ID of a request/interaction, the behavior of the system can be observed and traced back.

This unique ID is usually called a correlation ID or trace ID. This can be logged by the TAS to help analyze test results.

Show content Hide content
CT-UT v2.0 Ch.5 – Usability Testing

5.5.1 – Procedure for analyzing findings from a usability test

UT-5.5.1: Explain the procedure to apply for analyzing findings from a usability test
K2 Understand 56% match
Go to Syllabus p.40

Analysis of findings is the process that extracts findings from observations during usability test sessions.

The following steps are performed:

  1. During the usability test session, the scribe records usability observations, usually by writing them down. Usability observations reflect events that result in issues or have a positive impact on effectiveness, efficiency and satisfaction.
  2. After each usability test session (while everyone still remembers what happened), the scribe and the moderator meet and have a candid discussion about the observations taken from the usability test session.
  3. After all usability test sessions have been completed, the moderator and the scribe separately extract between twenty and thirty negative findings and between five and ten positive findings each from their observations. These findings reflect the observations that they consider most important.
  4. The moderator and the scribe meet again and have a candid discussion about their findings. The findings are merged into a common list consisting of between twenty and thirty negative defects and between five and ten positive findings.
  5. The moderator logs the defects in the common list on the organization’s usability defect tracking tool, which ideally is the same as the defect management tool.
  6. The moderator tracks the defects to resolution and reviews the implemented solution. If the implemented solution represents a risk, it should be subject to another usability test.
  7. If another usability test is needed based on the number of changes made to the system under test, that test should be scheduled.

Several points are of particular relevance to the analysis of findings:

  • Candid discussions between the moderator and the scribe (as mentioned in steps 2 and 4), are of essential importance to the success of the analysis. These discussions must be honest and based on observations rather than on personal opinions. Observers may be included in these candid discussions if they have any additional items that the scribe or moderator missed.
  • It is important that a usability test report is usable, so the number of reported findings must be limited (the values shown in point 3 above are rules of thumb).
  • It is a critical (and sometimes inconvenient) task for the moderator to prioritize the findings and report only the ones that are most important from a usability point of view.
  • Comparisons of the findings between the current usability test and previous usability test reports will help detect trends and frequently occurring usability defects.
Show content Hide content
CTEL-TM v2.0 Ch.6 – Test Monitoring and Test Control

6.2.3 – Preparing Test Progress Reports

ELTM-6.2.3: Select the proper level and media for providing test progress reports
K4 Analyze 55% match
Go to Syllabus p.78

Effective communication of test progress is essential for aligning stakeholders, managing expectations, and facilitating informed decision-making. The TME is responsible for selecting appropriate levels of abstraction and communication media to deliver impactful, timely, and targeted test progress reporting.

Selecting the proper level of the test progress report is vital because it communicates relevant information to the right stakeholders, in the right format, at the right time. Different stakeholders have different information requirements. Well-leveled test progress reports support stakeholders in understanding the current situation, identifying risks early, and making timely decisions. When test progress reports consistently deliver the right insights, stakeholders trust the test process and are more confident about the product quality. Three typical levels of test reporting can be distinguished:

  • Strategic is required by senior management and executives. It focuses on high-level KPIs, business risks, and general coverage trends in the context of project milestones. Test reports at this level are written in non-technical, outcome-driven language that aligns with business objectives.
  • Tactical is required by project managers, development leads, and test leaders. It focuses on progress against test plans, test resource usage, defect trends, risk/requirements coverage, test execution trends, and blocking defects. Test reports at this level employ a balanced mix of technical and management terminology.
  • Operational is required by testers, developers, and engineering. It focuses on daily test execution details, test case status, test environment readiness, and defect status. Test reports at this level use technical and detail-oriented language.

Once the test reporting levels are defined, the TME decides on the form in which the test reports will be delivered to stakeholders:

  • Static test reports (e.g., PDF files and spreadsheets) are suitable for formal documentation, audits, and executive summaries. They are archivable, shareable, versionable, and structured.
  • Dashboards are real-time visualization tools. They are suitable for tactical and operational test reporting, as they can provide detailed information in real time. They are up-to-date, often interactive, and feature drill-down capability, making them extremely useful for performing various types of analyses.
  • Presentations (e.g., slide decks) are suitable for stakeholders and milestone reviews. Such test reports are typically presented in a meeting, allowing the presenter to provide additional detailed information if needed. Presentations are customizable, contain focused messaging, and are a form of visual storytelling.
  • Emails and instant messaging are suitable for quick updates or urgent notifications. They are fast, informal, and very accessible. This form of test reporting is also common in Agile software development, which promotes lightweight documentation.
  • Meetings and verbal briefings are suitable for alignment, question and answer sessions, and complex defect explanation. They are interactive, clarify context, and allow substantial discussion.
  • Flipchart or whiteboard-based test reporting (e.g., burndown charts, Scrum boards, and Kanban boards) is typically used by Agile software development teams that are co-located or use toolsupported virtual equivalents. Test reports that use this form are low-tech, but provide immediate visibility, which promotes shared understanding and transparency. Such test reports engage collaboration by stimulating discussions and problem-solving, encouraging test team ownership of quality. They are easy to update in real time.

When selecting the right media for test reporting, the TME should consider different selection criteria, such as:

  • Type of test reporting (internal or external, (see Section 6.2.1)
  • Stakeholder’s role and information needs
  • Project phase and milestones
  • Criticality and sensitivity of information
  • Tool and technology availability
  • Regulatory and documentation requirements
  • Time constraints and frequency requirements

The higher the stakeholder's position in an organization, the more relevant general trending information may be than a detailed report. Charting and visually appealing graphs become increasingly important as the technical expertise of the consumer declines. Because project managers may have little time to devote to reviewing test reports, quick visuals, easily interpreted charts, and trending diagrams are handy for conveying a point quickly and accurately (see Section 6.2.2).

Show content Hide content
CT-DAT v1.0 Ch.3 – Digital Accessibility Testing Activities

3.4 – Risks and Defects in Digital Accessibility Testing

Digital accessibility testing is affected by project risks, and testers should identify and communicate those risks while working within the project's established risk-management responsibilities. Common software defects should also be recognized and reported.

  • Project risks (risks within the test process or strategy)
  • Common digital accessibility defects (frequent technical and structural code defects)
Show content Hide content
CTEL-TM v2.0 Ch.6 – Test Monitoring and Test Control

6.3.2 – Data-Based Test Process Analysis

ELTM-6.3.2: Analyze a given test condition based on available data
K4 Analyze 55% match
Go to Syllabus p.80

To be effective, assessing the current test process state and product quality based on metrics should always be data-driven. This section looks at each group of activities within the test process and discusses how metrics collected during these test activities should be interpreted.

Test planning . Adherence to the estimated test schedule and budget is critical to all test projects. Planned requirements coverage should align with the project's scope and business priorities. Effort metrics can be used to assess the feasibility of the test project quickly. Test environments and tools readiness are needed to execute the plan. Historical data enables the identification of high-risk or defectprone areas that require more rigorous testing. If risk levels or test resource constraints are too high, features or test objectives should be reprioritized.

Test monitoring and test control . Tracking deviations in test progress allows the TME to identify delays early and adjust test scope or test resources accordingly. Analyzing trends in DDPs helps to determine if testing is adequate or should be redirected. Monitoring the ratio of failed tests identifies quality concerns in specific areas. Effort variance data determines whether a reallocation of test resources is necessary. Evaluating actual coverage (e.g., requirements coverage) detects scope creep or under-tested features. The TME should evaluate how well telemetry, logging and monitoring data enable RCA and continuous improvement. The TME should also determine that the data is consistent with data reported by other teams (e.g., developers).

Test analysis and test design . Assessing the completeness of traceability determines if all requirements are testable and covered. Low numbers of test conditions per requirement may indicate underspecification or a lack of understanding of the requirement. Test design coverage metrics verify the thoroughness of the test conditions that have been derived. High ambiguity or inconsistency rates in requirements signal the need for further clarification or stakeholders' engagement. Defects found during the review cycles and during test design, if resolved, prevent those defects from escaping to the next phase. If a defect is identified but not resolved, additional work is incurred due to its existence. However, the time spent addressing a known defect is usually less than that spent detecting a previously unknown defect. Cost, in terms of money or time, is often associated with these defects. That cost is typically reported, usually in terms of cost savings, as part of test reporting. Defect trending is also essential at this phase. If numbers of defects are higher than expected in the review cycles, this may indicate a lower quality of the work products than was anticipated. If so, multiple review cycles may be necessary to achieve an acceptable level of quality before these work products progress to the next phase in the SDLC.

Test implementation and test execution . Readiness of test cases, test data, and test environments promotes timely test execution, proper test execution order, test team efficiency, and the avoidance of bottlenecks. Testing without 100% readiness may be an acceptable, even a preferred strategy. However, if the missing elements are required for the highest risk level items, it will force higher risk level testing toward the end of the test cycles and skew metrics, and that is generally not acceptable. Coverage must be closely monitored. If coverage is not proceeding according to plan, risks may appear where they were not planned. Passed/failed information determines if the expected passed/failed rate is being achieved. If a higher number of test cases is failing than was expected, additional test cycles may be required. If a higher number of test cases is passing than was expected, the TME needs to determine if testing is sufficient, where the extra time in the test schedule should be focused, and if plans should be adjusted for future testing. Test flakiness rate shows the level of test stability. Analyzing test automation coverage evaluates efficiency opportunities and test repeatability. Defect severity distribution can help with test prioritization and escalation strategies. Test effectiveness broken down by test types informs the TME how productive the various test types are in terms of defect detection, risk coverage, and tester skill usage.

Test completion . Achieved coverage allows for assessing if the exit criteria are met. Evaluating DD and severity trends determines residual risk and release readiness. Identifying open defects and their business impact supports making go/no-go decisions. Lessons learned can be used to refine future test planning, improve test estimates, and reduce escaped defects. Incorporating stakeholders' feedback validates the quality of the test process and test results.

Show content Hide content
CTFL v4.0.1 Ch.5 – Managing the Test Activities

5.3.2 – Purpose, Content and Audience for Test Reports

FL-5.3.2: Summarize the purposes, content, and audiences for test reports
Related exam questions
K2 Understand 55% match
Go to Syllabus p.54

Test reporting summarizes and communicates test information during and after testing. Test progress reports support the ongoing test control and must provide enough information to make modifications to the test schedule, resources, or test plan, when such changes are needed due to deviation from the plan or changed circumstances. Test completion reports summarize a specific test activity (e.g., test level, test cycle, iteration) and can give information for subsequent testing.

During test monitoring and test control, the test team generates test progress reports for stakeholders to keep them informed. Test progress reports are usually generated on a regular basis (e.g., daily, weekly, etc.) and include:

  • Testing period
  • Test progress (e.g., ahead or behind schedule), including any notable deviations
  • Impediments for testing, and their workarounds
  • Test metrics (see section 5.3.1 for examples)
  • New and changed risks within testing period
  • Testing planned for the next period

A test completion report is prepared during test completion, when a project, test level, or test type is complete and when, ideally, its exit criteria have been met. This report uses test progress reports and other data. Typical test completion reports include:

  • Test summary
  • Testing and product quality evaluation based on the original test plan (i.e., test objectives and exit criteria)
  • Deviations from the test plan (e.g., differences from the planned test schedule, duration, and effort).
  • Testing impediments and workarounds
  • Test metrics based on test progress reports
  • Unmitigated risks, defects not fixed
  • Lessons learned that are relevant to the testing

Different audiences require different information in the reports and influence the degree of formality and the frequency of test reporting. Test progress reporting to others in the same team is often frequent and informal, while test completion reporting follows a set template and occurs only once.

The ISO/IEC/IEEE 29119-3 standard includes templates and examples for test progress reports (called test status reports) and test completion reports.

Show content Hide content
CTEL-TM v2.0 Ch.6 – Test Monitoring and Test Control

6.1.2 – Evaluating the Test Measurement and Test Reporting Process

ELTM-6.1.2: Evaluate a test measurement and test reporting process

Evaluating test measurement and test reporting involves assessing whether the data collected and reported effectively supports decisions related to product release readiness, process improvements, resource management, and risk mitigation. The goal is to determine how well the process tracks progress toward defined test objectives and quality goals and whether it delivers actionable insights to stakeholders. This evaluation should identify strengths, gaps, inefficiencies, and opportunities for continuous improvement. Key aspects of this evaluation include:

Evaluating objectives and context alignment. The TME verifies whether the measurement objectives remain relevant and clearly aligned with current project needs and organizational goals. This includes evaluating:

  • Whether the measurement correctly supports informed decision making and analyses, such as release readiness, defect trends, quality assessment, and productivity
  • Whether the process accounts for contextual factors such as project constraints, the SDLC, technology used, regulatory requirements, and test resources
  • Whether test reporting objectives (e.g., risk management and test progress) and test scope (e.g., test levels and release cycles) are still appropriate and meaningful

Assessing stakeholder satisfaction and information utility. The TME evaluates whether stakeholders (e.g., product owners, test leaders, executives) are receiving the relevant and actionable information in a format they can use. This includes:

  • Determining if stakeholders are using test reports to make informed decisions
  • Assessing if information is presented at the right level of detail for each audience
  • Identifying unmet needs, redundant information, or information overload
  • Gathering direct feedback from stakeholders to refine test report content and delivery

Validating metric relevance. The TME evaluates whether the selected metrics are reliable, valid, and well defined. Reliability means that multiple measurements of an identical entity give the same or similar results (i.e., the standard deviation is low). Validity means that the metric accurately measures what it is expected to measure (i.e., it captures the real value of the attribute it is intended to describe). To have metrics that are both reliable and valid, there must be agreement on standard definitions for the entity and its attributes that are being measured (Westfall, 2009).

Each metric should be measured on an appropriate measurement scale. Four main measurement scales are:

  • Nominal assigns an item to a classification that categorizes the attribute of the entity.
  • Ordinal classifies the attribute of an entity by order, but there are no assumptions about the magnitude of the differences between categories. This allows the mathematical relational operators to be applied to measurement values (<, >, ≥, ≤, =, ≠).
  • Interval, where the exact distance between the scales is known. This allows the mathematical operations of addition and subtraction to be used.
  • Ratio provides the exact distance between the scales and there is an absolute zero point. This allows all mathematical operations to be used, including multiplication and division.

Basic metrics are measured directly. Derived metrics are calculated from one or more basic metrics, and their measurement scale depends on the scales of the basic metrics used and the mathematical operations applied. Hence, for a derived metric, the TME verifies that only mathematical operations that are valid for the lowest-scale metric involved are allowed.

To measure, the entity being measured must be determined. The TME checks if its attributes (i.e., measurable properties) are accurately described. Finally, a mapping system, called the measurement method, must be evaluated and accepted. In software testing, for many metrics, no standard mapping system has been widely accepted (e.g., defect severity can be measured as critical, major, or minor, or on a 4-point scale, 1, 2, 3, or 4).

Reviewing data sources and collection methods. The TME examines the accuracy, timeliness, and consistency of data sources and how data is collected. Key questions include:

  • Are current tools and sources (e.g., CI/CD pipelines and defect management systems) delivering trustworthy data in an efficient manner?
  • Is the frequency of data collection appropriate for decision timelines?
  • Are responsibilities for data entry, validation and oversight clearly assigned and executed?
  • Are there issues with missing, duplicated, incomplete, or incorrect data?

Evaluating report formats and visual communication. The TME assesses whether test reports and visualizations (see Section 6.2) clearly convey insights and are appropriate for the intended audience. The TME reviews test report formats and dashboards regarding clarity, readability, and consistency, and evaluates whether visualizations (e.g., charts and graphs) aid or hinder interpretation. The TME checks for standardization in terminology and layout and identifies any usability defects that may affect comprehension or decision-making.

Checking test reporting frequency and schedule. The TME evaluates the effectiveness of the test reporting schedule and test automation. Test reports should be delivered frequently enough to support timely decisions. They should be synchronized with development and test cycles. The test report generation should be automated where feasible (e.g., creating dashboards from test tools, as described in Section 6.2) to improve efficiency and reduce errors.

Evaluating baselines and targets. The TME reviews how actual measurements compare to defined baselines and targets. Baselines and realistic targets or thresholds should be set for each metric based on historical data or industry benchmarks. Targets should be specific, measurable, achievable, relevant, and time-bound (SMART). The TME determines if the defined variances or control limits are acceptable.

Assessing the measurement and test reporting procedures. The TME evaluates whether the measurement and test reporting procedures are well-documented, standardized, repeatable, followed consistently, and test results are validated appropriately. The measurement plan should reflect actual practices. Measurement and test reporting should be aligned with the test strategy and broader quality management framework. Responsibilities should be clearly defined and upheld.

Evaluating report distribution channels. The TME determines whether test reports are reaching the intended recipients securely, efficiently, and on time and appropriate distribution channels are being used. The TME makes sure that access controls are in place for sensitive or compliance-related information. The TME evaluates if there are delays or bottlenecks in delivery affecting decision-making.

Pilots/dry runs. The TME may conduct a dry run or limited pilot to verify the clarity and interpretability of metrics, data quality and reliability, and ease of collection and analysis. This tests the usefulness and accuracy of the measurement process. The TME gathers feedback which can be used to refine the approach.

Test monitoring, reviewing, and improving. The TME tracks whether metrics are supporting intended goals, identifies gaps, redundancies, or misinterpretations, and adjusts metrics, tools, or processes based on evolving needs and feedback. Regarding the test reporting process, the TME collects stakeholders’ feedback regularly, reviews whether test reports are aiding decision-making, and updates report formats or metrics as project or organizational needs evolve. The purpose is to continually improve the effectiveness of the measurement and test reporting process.

Show content Hide content
CTEL-ITP v2011 Ch.4 – Analytical-based Improvement

4.2.2 – Causal Analysis during an Inspection Process

LO-4.2.2: Understand causal analysis during an inspection process
K2 Understand 54% match
Go to Syllabus p.31

The inspection process is described in the ISTQB Foundation syllabus and expanded in the ISTQB Advanced syllabus. Using the software inspection process [Gilb & Graham] suggests a different approach to causal analysis.

The causal analysis meeting is a facilitated discussion which lasts two hours and follows a set timescale and format.

  • The selection of defects to be considered may be taken by the inspection leader or in the context of a project retrospective. Alternatively the issues to be discussed may identified in the first part of the analysis.
  • Part 1: Defect analysis (90 minutes) looks at specific defects and their specific causes
  • Part 2: Generic analysis (30 minutes) focuses on identifying broad trends in the defects. The team looks for trends, commonalities and what has recently gone well, improved or gone wrong. Defect taxonomies (introduced in the Advanced syllabus) can be a valuable support for this analysis. Generic analysis may also focus on particular defects found in dynamic testing to provide risk-based inputs for the future testing strategy.
  • The discussion must ensure that the number and severity of the defects being considered maximizes the return on the invested time.

In the defect analysis, each defect is categorized with:

  • A defect description - this is not the defect symptom but the defect itself
  • A cause category - e.g., communication, oversight, education, transcription error, process
  • A cause description - for the cause and any chain of cause events
  • The process step when the defect was created - this is not always where it was detected
  • Suggested actions to eliminate the cause - these must be specific and achievable
Show content Hide content
CTAL-TM v3.0 Ch.1 – Managing the Test Activities

1.6.5 – Tool Metrics

TM-1.6.5: Give examples for metric collection and evaluation by using tools
K2 Understand 54% match
Go to Syllabus p.45

Objective metrics from tools are designed and collected based on the needs of the test team and other stakeholders. Test tools mostly capture valuable real-time data and reduce data collection efforts. This data is used to manage the overall test effort and identify areas for optimization.

Different tools are focused on collecting different types of data. Examples of these include:

  • Test management tools can supply a variety of different metrics related to available test items, tests, planned tests as well as current and passed test execution status (e.g., passed, failed, skipped, blocked or planned)
  • Requirements management tools deliver traceability regarding requirements coverage by passed and failed test cases
  • Defect management tools can provide defect information such as status, severity, priority and defect density of test items. Other valuable data, such as the defect detection percentage, the test levels at which defects are introduced and detected defect lead time help to drive process improvement, but may not all be provided solely by the defect management tool.
  • Static analysis tools, among others, supply metrics related to code complexity
  • Performance testing tools can supply valuable information such as response times and failure rates under peak loads
  • Code coverage tools help to understand which parts of the test object have been exercised by testing
  • Although test tools can be used to collect metrics, they should also monitor themselves. In this context the quality of the test process can be measured (e.g., the number of defects found with and without tools and requirements coverage)
  • Test efficiency (e.g., duration of test execution and number of executed tests)

More details on the collection and usage of metrics can be found in Section 2.1 of this syllabus, Test Metrics.

Show content Hide content
CT-STA v1.0 Ch.6 – Interpreting Test Results and Defining Actions

6.2.1 – Analyzing security test reports to prioritize necessary actions

STA-6.2.1: Analyze a given security test report to prioritize necessary actions
K4 Analyze 54% match
Go to Syllabus p.46

Prioritization : This defines the order in which the vulnerabilities should be addressed. Prioritization can be directly linked to the severity of a vulnerability: For example, a vulnerability with a CVSS score of 10.0 (the highest score) must usually be fixed as soon as possible and thus warrants performing an immediate release update in production. An exemption might be a contextual adjustment (e.g. an exploitable system might not be connected with internet at all). By comparison, a vulnerability with a CVSS score of 1.0 can be fixed and deployed as part of the next planned releases. Note that business objectives may also influence prioritization levels. In general, the following summary of highlevel steps can serve as a guide for the prioritization:

STA TaskSTE TaskTask description
xDefine security test work packages.
xDefine stakeholder and audience-specific security test reports, formats,
and communication channels.
xExecute security test work packages and generate security test reports.
xxCreate a suggestion for prioritizing identified vulnerabilities.
xCommunicate the security test report and prioritization.
xPerform confirmation tests for the implemented fixes.

Severity: This is a major factor in deciding the prioritization, but should not be the only factor considered in that decision. Depending on the context and the domain, factors may apply that are equally as important, for example:

  • Some vulnerabilities may have a high severity but are design defects that require re-design and/or rewriting major parts of an application to mitigate them. In such cases, it may not be reasonable to assign them a high priority. Mitigations often involve multiple steps. A common approach is to first take high-priority measures to lower the risk and then implement a complete fix later with lower priority.
  • Low-severity vulnerabilities might be in critical functional areas and show signs of active exploitation. They may also represent a high risk of damage or loss if exploited. These factors can increase the priority of the vulnerability.
  • In cases where a number of vulnerabilities take a long time to resolve, it could be advisable to prioritize the defects that are deemed to be quick to resolve.
  • When vulnerabilities exist that have not been resolved for some time, it might be advisable to give them a higher priority because the likelihood that they are already known to attackers is higher. Such vulnerabilities have been proven to be present for a long time, in contrast to the vulnerabilities that were only recently introduced.
  • Sometimes multiple vulnerabilities can be fixed by making changes in the same or a similar area, or where the recommended actions overlap. In such cases, it is advisable to prioritize them closely together. Fixing them together is likely to be faster than changing the same area multiple times separately, as they can be considered as a whole and fixed in one change.
  • Vulnerabilities that are obviously visible but have a low severity could be prioritized higher if they give a bad impression of the application's security posture or lead to a compound risk. For example, exposing the version numbering dependencies used in test logs that are shown to the user could have a low severity if the versions are regularly updated. However, it could give the impression to attackers that the application is an easy target if it doesn’t follow basic security best practices, or the version number could be used to find potential vulnerabilities.
  • Even if a security test report already contains recommendations for necessary mitigation actions, they should not be followed blindly and considered as the only actions to perform. They may not cover aspects that go beyond mitigating the specific defect, and which could be equally important to consider improving overall security.

Risk levels : Actions taken to counteract vulnerabilities can be applied to the processes, organization, employees, or work products/deliverables. To define and propose appropriate actions for a vulnerability, it is first necessary to perform a risk analysis. After all, defining actions that involve a lot of effort to resolve a vulnerability whose risk is very low might not be appropriate, for example:

  • If a vulnerability cannot be mitigated by fixing the code, the risk level could be reduced by lowering the impact of the vulnerability being exploited (e.g. by not using the affected system any more for critical use cases).
  • If the vulnerability relates to potential unauthorized physical access to servers, the risk level could be transferred or shared by either outsourcing the server hosting to a different organization or by insuring against the damage should the event occur.
  • If secure storage of Personally Identifiable Information (PII) has a high-risk level, the risk can be avoided by not storing any PII in the first place. However, this is not always an option and may mean losing potential gains that could have resulted from accepting the risk.

Extent of actions: Actions may not be limited to the immediate treatment after identifying the risk level of a vulnerability, but may also extend beyond that. For example, it is not enough to perform security awareness training with employees once, it is necessary for new employees to get trained as well as retraining employees at regular intervals. In addition, if SQL injections are found regularly, for example, it is not enough to mitigate the vulnerabilities by fixing them as they are found. Measures should be taken to avoid SQL injection vulnerabilities are not introduced in the first place, for example, by regularly training developers in secure coding or establishing a code review process.

Combining actions: Actions may not directly target a single individual vulnerability. It could be more effective to define actions that consider similar or related vulnerabilities. If the majority of vulnerabilities are related to vulnerabilities in the code, considering each vulnerability on its own would only result in the vulnerabilities being addressed directly. Having a large number of such vulnerabilities may be an indicator of poor knowledge of secure coding practices by the developers, which makes training them a necessary additional action to support long-term improvements.

When determining appropriate actions, it is important to consider past actions taken against similar vulnerabilities. For example, if a vulnerability keeps re-occurring, this indicates that the actions taken are not effective enough or they were not executed properly. This makes it necessary to define different actions from those tried previously or to make changes to previously defined actions.

Scope : The scope of an action should be considered. For example, if a certain group of employees does not have any access to IT systems, it might not be beneficial to train them in security awareness at all. On the other hand, it might make sense to make the retraining interval of security awareness training shorter for employees with access to very critical infrastructure compared to employees with access to less critical infrastructure. Other actions, however, make more sense when performed globally, such as installing anti-virus software, as it is usually cheaper to have the same setup for every employee.

Show content Hide content
CTEL-TM v2.0 Ch.6 – Test Monitoring and Test Control

6.2 – Test Reporting

Section 6.2 - Test Reporting (Chapter 6: Test Monitoring and Test Control).
Show content Hide content
CT-STA v1.0 Ch.6 – Interpreting Test Results and Defining Actions

6.1.2 – Managing security test report follow-ups for vulnerability mitigation

STA-6.1.2: Explain management of security test report follow-ups to support timely vulnerability mitigation
K2 Understand 54% match
Go to Syllabus p.45

The STA is responsible for communicating security test reports and possible vulnerabilities in a form that enables all necessary mitigation actions (or other risk treatment methods) to be initiated.

The communication for non-emergency-findings starts with discussing the security vulnerabilities with relevant stakeholders once a reviewable security test report has been produced by the STE. The STA’s task is to explain the technical details to different stakeholders, including higher management. During this discussion, the STA shall formulate possible follow-up actions and recommendations. These may include temporarily shutting down the system, addressing the security vulnerabilities, or accepting the risks.

Follow-up actions depend on several factors:

  • Severity of the vulnerability. These are usually classified into specified categories such as low, medium, and high or are specified using the latest version of the Common Vulnerability Scoring System (CVSS) [CVSS] [ISTQB-STE].
  • Required security level and business criticality of the application.
  • Possibilities for changing code and the environment.

Apart from the technical details of the vulnerability itself, the severity may also depend on other factors, such as:

  • Location of the vulnerability: This could be in a functional component identified as having a high-risk level.
  • Defined security requirements: These were defined because they are considered to be important for securely operating the application.

If the vulnerability is linked to a known vulnerability, the publicly available details must be stated, including its severity and exploitability. This helps determine the severity of the vulnerability. Note that the severity of the vulnerability often does not match the vulnerability’s severity (e.g. if a system is not connected to Internet at all), but it provides a useful basis for localizing common severity levels and for further refinements (see [ISTQB-STE] and the difference between base CVSS and contextadjusted CVSS).

Show content Hide content
CT-QDO v1.0 Ch.3 – Automated and Manual Tasks in DevOps

3.1.3 – Quality Reporting for a CI/CD Pipeline

QDO-3.1.3: Implement uniform quality reporting practices in the CI/CD pipeline with information from both automated testing and manual testing
Related exam questions
K3 Apply 54% match
Go to Syllabus p.35

At any given time, it is important to understand the software quality level throughout the SDLC. This provides feedback to the DevOps team on code quality and to product management on release readiness. Integrating quality reporting into the CI/CD pipeline provides the latest development status. Each stage of the CI/CD pipeline creates information for quality reporting. Some quality data is generated automatically by the CI/CD pipeline (e.g., build success, test passed/failed status), while other data comes from manual testing (e.g., user acceptance testing, exploratory testing, or coverage in a test management tool). This information creates a comprehensive view of the software quality.

Quality reporting should occur consistently for each cycle of the CI/CD pipeline and thus be automated. Manual test results should be included in the quality reporting, typically via integration with a test management tool. The information is collected into metrics and combined into a quality report. The quality report can be delivered using various means (e.g., by a dashboard on a wiki page, in a shared tool, as an email or report based on a template, or as a combination of these) ( ISTQB® CTFL , v4.0).

Automated quality reporting requires the integration of several tools. These include various CI/CD pipeline tools’ logging, monitoring, and reporting functionalities (see Section 4.1.2 ). Task management and test management tools record the manual test results (e.g., the estimated quality level of software features or passed/failed test results). Production monitoring tools provide telemetry (see Section 4.2.7 ). Report generators should integrate with all these tools to create easy to understand reports for various stakeholders. Create a single source of truth by combining the metrics information from various sources into a single, accessible location (see Section 3.1.1 ).

The quality reporting implementation process has several steps:

  1. Determine the key performance indicators (KPIs), and related metrics to track, such as code quality, coverage, performance, build stability, build success rate, deployment frequency, failed deployment recovery time, and security vulnerabilities.
  2. Collect data using chosen metrics.
  3. Automate data gathering from various CI/CD pipeline stages.
  4. Clean and normalize the data for better consistency with data preparation tools (see Section 3.1.4 ).
  5. Store the collected data in a centralized database if the tools do not provide it.
  6. Visualize metrics using dashboards, such as charts, graphs, and tables that display the key metrics, including different sections for each stage of the CI/CD pipeline.
  7. Enable continuous monitoring in the dashboard for real-time updates.
  8. Ensure the dashboard accessibility for relevant stakeholders, such as developers, testers, product owners, scrum masters, and managers.

A quality report dashboard can enhance a CI/CD pipeline. Benefits include:

  • Tracking key metrics continuously for quick response and resolution
  • Analyzing CI/CD pipeline performance trends to identify bottlenecks and optimize the CI/CD pipeline efficiency
  • Reviewing security compliance status proactively to ensure the codebase meets security standards
  • Fostering a culture of transparency and collaboration through shared dashboards with stakeholders, aligning everyone on the project’s quality status and progress
  • Supporting retrospectives with data-driven insights to identify areas for improvement
Show content Hide content
CTAL-TM v3.0 Ch.2 – Managing the Product

2.3.4 – Defect Management Challenges in Hybrid Software Development

TM-2.3.4: Explain the challenges of defect management in hybrid software development
Related exam questions
K2 Understand 54% match
Go to Syllabus p.58

In practice, multiple teams often collaborate on the delivery of the system or system of systems. Examples include hybrid software development when a customer uses Agile software development and one of their suppliers uses a sequential development model, or when an organization using a sequential development model requires delivery of a subsystem from a team using Agile software development. Such a multi-team environment poses various challenges:

  • Alignment on defect attributes and tools to be used for defect management: In an ideal scenario all teams use one defect management tool. In practice it is common for each team to use a different defect management tool, especially when several supplier teams contribute to project delivery. In such cases it is good to establish synchronization between the defect management tools (preferably automatically).
  • Prioritization of defects: The product owner(s) should be included in defect management meetings and actively seek information about consequences and risks associated with defects. Defect management meetings should be held more often with Agile software development than in sequential development models to keep pace with the Agile team’s faster rate of product increment delivery. These meetings may, however, be shorter with Agile teams. Sometimes it is beneficial for a smaller group of defect management stakeholders to have the final word about prioritization of defects.
  • Alignment and transparency of the test plan for new development and defect fixes : All teams’ work should align to the same project plan irrespective of whether they are using Agile software development or sequential development models. All deliverables, including defect fixes, should be aligned with this project plan. Better alignment can be achieved by active participation of the members of all teams in the planning process (e.g., participation of sequential development model teams in Agile software development meetings where defects are discussed and prioritized). Transparency of development plans can be improved by sharing them between teams (e.g., via dashboards, or via the Product Backlog).
Show content Hide content
CTEL-TM v2.0 Ch.6 – Test Monitoring and Test Control

6.2.1 – Internal and External Test Reporting

ELTM-6.2.1: Distinguish between internal and external test reporting
K2 Understand 54% match
Go to Syllabus p.76

Internal test reports are circulated within the project team (e.g., testers, test managers, developers, Scrum Masters, product owners, DevOps engineers, and project managers). Their purpose is to manage day-today testing, identify and resolve defects quickly, track test progress and test resource usage, facilitate collaboration, enable test leader and test team coordination, and drive test team-level decision-making. Internal test reports are typically characterized by high detail, technical depth, raw data, and frequent updates (daily to weekly). They focus on information essential for the project team, such as test progress, defects, blocked tests, or coverage. The tone and style of internal test reports are informal to semi-formal, technical, and test team focused. While internal test reports may be less formal than external test reports, the quality of the data is just as critical, particularly when management decisions will be based on the information in these test reports. Typical formats of these test reports include spreadsheets, dashboards, internal emails, test team updates, or test reports from defect management tools.

External test reports are produced for external stakeholders. Their purpose is to provide transparency, demonstrate compliance, product quality, and test thoroughness, communicate risk and readiness to release, track contractual commitments, and provide business-level insights and accountability. External test reports are typically characterized by summarized, high-level insights tailored for business or oversight audiences. Their frequency is either milestone-based, or when a phase ends (e.g., test completion reports and regulatory reports). They focus on overall product quality, risk status, defect trends, and compliance metrics. The tone and style of external test reports are formal, business-focused, clear, and polished. Typical formats of these test reports include formal documentation, presentations, audit logs, or executive dashboards.

Both types of test reports should be aligned with consistent metrics and definitions to avoid confusion. Internal test reporting should be automated whenever possible to provide agility. In contrast, external test reporting should undergo a thorough review to determine accuracy, clarity, and an appropriate tone. In both types, traceability from raw data to high-level summaries should be confirmed for auditability.

Show content Hide content
CTEL-ITP v2011 Ch.4 – Analytical-based Improvement

4.4.3.3 – Early Defect Detection

As stated in the previous performance indicator, the efficiency of the defect removal process will increase if defects are found earlier in the process. Not only is this applicable for static versus dynamic testing but also for unit and integration versus system and acceptance testing. The performance indicator „Early Defect Detection‟ measures the total number of defects found during unit and integration testing (early dynamic testing) versus the total number of defects found during dynamic testing.

Show content Hide content
CTEL-TM v2.0 Ch.7 – Evaluating Effectiveness, Efficiency and Satisfaction and Improving Test Policy

7.1.2 – Using Metrics to Identify Opportunities for Test Process Improvement

ELTM-7.1.2: Analyze effectiveness, efficiency, and satisfaction metrics for the test process, identifying opportunities for improvement
K4 Analyze 54% match
Go to Syllabus p.86

Metrics should always be defined specifically for the test policy, test process, and test projects in place. This section provides a representative set of metrics that can be used.

Effectiveness metrics

Defect density ( DD ) quantifies the concentration of defects discovered relative to the functional or structural size of the test object:

where item 𝑋 is measured in units, (e.g., KLOC, function points, user story points, or requirements). High 𝐷𝐷 suggests the need for improvement in coding standards, using code reviews, refactoring, static analysis, strengthening requirements-to-design traceability, or early validation. Low 𝐷𝐷 may indicate a genuinely high product quality (e.g., after shifting effort to regression testing, test automation efficiency, or test suite optimization) or undetected defects (e.g., need to improve test design, coverage, risk-based testing, exploratory testing, or test environments).

𝐷𝐷 should not be reported in isolation but paired with defect severity to distinguish between high-impact architectural issues and low-priority cosmetic defects.

Test case effectiveness ( 𝑇𝐶𝐸 ) measures the test object through the lens of testing itself:

Low 𝑇𝐶𝐸 may indicate that test cases are overly “happy-path” or simply weak, suggesting to use boundary conditions, negative paths, or error scenarios and to perform test case reviews. Because a low value can also stem from high-quality software, always pair 𝑇𝐶𝐸 with coverage metrics to accurately assess whether the test results indicate a weak test suite or a mature product.

Escaped defect rate ( 𝐸𝐷𝑅 ) measures the percentage of defects that escaped the test process:

High 𝐸𝐷𝑅 may indicate poor shift-left practices, inadequate coverage, weak risk focus, environment mismatch, weak regression testing, or ineffective defect resolution. To stabilize 𝐸𝐷𝑅 , define a defect time window (e.g., the first 90 days post-release); otherwise, the metric fluctuates. While 𝐷𝐷𝑃 (ISTQB-TA) measures how many defects were caught, 𝐸𝐷𝑅 helps assess the efficiency of the removal process; together, they provide a holistic view of QC performance.

𝐸𝐷𝑅 should be reported with CoQ, as external failures usually cost more than internal ones.

Test result accuracy ( 𝑇𝑅𝐴 ) measures the reliability of the test results:

Low 𝑇𝑅𝐴 indicates vague, incomplete, or incorrect expected results, unclear pass/fail criteria, or ambiguous requirements, suggesting strengthening test oracles, clarifying requirements and acceptance criteria, using precise measurable assertions, and conducting test case reviews focused on expected results. Because false-negative results are often only identified retrospectively, calculating 𝑇𝑅𝐴 requires a standardized observation window, similar to 𝐸𝐷𝑅 . For more actionable insights, 𝑇𝑅𝐴 can be split into two distinct sub-metrics: pre-deployment accuracy that tracks false-positive results, and post-deployment accuracy that tracks false-negative results.

Efficiency metrics

Test execution efficiency ( 𝑇𝐸𝐸 ) quantifies the productivity of the test team:

Low 𝑇𝐸𝐸 may indicate inefficient test design, poor test environment or tooling support, skill or knowledge gaps, suboptimal test planning, a high amount of defect rework, test cases need refactoring, test environments need stabilizing, training on tools and domain knowledge is required, poor build and release stability, or the need for smoke testing before detailed testing.

Test automation efficiency ( 𝑇𝐴𝐸𝑓𝑓 ) is measured with respect to time ( 𝑇𝐴𝐸 ) or productivity ( 𝑇𝐴𝐸 ):  

Low 𝑇𝐴𝐸𝑓𝑓 may indicate poor test automation design, test automation framework defects, architectural inflexibility, high maintenance overhead, skill or knowledge gaps, test environment or tooling inefficiencies, test automation code needs refactoring, test environments need stabilizing, test automation best practices and coding standards need applying, or training on tools, languages and test frameworks is required.

Return on investment ( 𝑅𝑂𝐼 ) determines whether testing adds economic value.

The primary ROI driver in manual (M) testing is risk mitigation:

where estimated cost of avoided failures is calculated as the sum of the products of failure probabilities and failure impacts as cost for all defects found during manual testing. Low 𝑅𝑂𝐼 may indicate low-risk areas, over-testing, or test redundancy.

The primary ROI driver in automated (A) testing is labor substitution:

 where 𝑛 is the number of test runs.

The manual testing cost equivalent is the total cost to run the same automated tests manually once. The development and maintenance costs are those for the test automation.

Satisfaction metrics

Defect rejection rate ( 𝐷𝑅𝑅 ) measures the percentage of defects rejected by the development team:

A high 𝐷𝐷𝑅 may indicate ambiguous test basis, limited domain knowledge, or unstable test environments causing irreproducible failures. However, high 𝐷𝑅𝑅 can be acceptable when testing highly innovative features with frequently changing requirements. Providing context prevents 𝐷𝑅𝑅 from being used against the test team.

Customer satisfaction score ( 𝐶𝑆𝐴𝑇 ) is typically derived from surveys using standardized scales, such as a 5-point Likert scale (Likert, 1932). An example of 𝐶𝑆𝐴𝑇 on this scale is:

Additionally, the average score reflects the overall sentiment of all stakeholders, including those who are dissatisfied. An increasing average indicates that 𝑇𝑀𝐸 changes are effective, even if the top two response rates remain unchanged.

Net promoter score ( 𝑁𝑃𝑆 ) measures long-term loyalty and advocacy by asking how stakeholders are to recommend the test team or test services. Scores of 0–6 are detractors, 7-8 are passives (i.e., satisfied but unenthusiastic, not advocates, and vulnerable to competitors) and 9–10 are promoters; 𝑁𝑃𝑆 is the net result:

𝑁𝑃𝑆 = 𝑇𝑜𝑡𝑎𝑙 𝑃𝑟𝑜𝑚𝑜𝑡𝑒𝑟𝑠 − 𝑇𝑜𝑡𝑎𝑙 𝐷𝑒𝑡𝑟𝑎𝑐𝑡𝑜𝑟𝑠

Because the detractor range (0–6) is wider than promoters (9–10), 𝑁𝑃𝑆 is highly sensitive to stakeholders’ sentiment, serving as an early-warning system for test perception issues. Negative 𝑁𝑃𝑆 provides a red flag that testing may be perceived as a hurdle rather than a value-added activity. This metric allows the TME to proactively address perception issues before they impact cross-functional collaboration.

Report the average score with 𝑁𝑃𝑆 for context.

In large organizations, a high 𝑁𝑃𝑆 often justifies the internal cost of centralized test centers.

Show content Hide content
CTEL-TM v2.0 Ch.5 – Project Management Essentials

5.1.2 – Defect Prediction Models and Techniques

ELTM-5.1.2: Select a defect prediction model
K4 Analyze 53% match
Go to Syllabus p.64

Defect prediction forecasts where defects are likely to occur in a software component or system. It identifies defect-prone components before testing begins. Predicting defects enables a more effective allocation of test effort. It transforms test estimation from a flat effort model into a risk-driven strategy, focusing on where critical defects are most likely to occur. Critical components predicted to be defectprone get more coverage, AI-augmented testing, or manual exploratory testing effort, while non-critical components may get minimal testing. Defect prediction models can adjust test case volume, test automation or review test intensity, and test execution time depending on predicted defect counts. Predicted defect counts can also be a variable in regression or machine learning-based test effort prediction models.

The most used defect prediction models are binary classifications (i.e., defective versus non-defective component), multiclass classifications (e.g., low, medium, and high defect risk), and regression (predicting the number of defects per component).

Standard defect prediction models focus on files or components and are typically based on code metrics (e.g., size and complexity) and defect patterns. Just-in-time defect prediction models (Kamei 2013) evaluate each code change and predict if this specific change is likely to introduce a defect. Just-in-time models are typically based on change history (e.g., code churn, number of commits, change frequency, and volatility of changes) and test team characteristics (e.g., number of developers working on the file, and developers’ experience).

Statistical models rely on quantitative software metrics and use mathematical relationships to forecast defect proneness. This is useful when historical defect data is available and the relationship between variables and defects is known or follows a known distribution. Examples include:

  • Logistic regression predicts a binary outcome (i.e., defective/non-defective components) based on continuous and categorical predictors.
  • Linear regression predicts the number of defects as a continuous output.
  • Poisson regression predicts defect counts assuming a Poisson distribution (see example (Web1)).

Machine learning models learn complex patterns and nonlinear relationships from data and outperform statistical models, especially when the data is large in scale, or not easily modelled statistically. Examples include decision trees, random forests, support vector machines, and neural networks. These models typically need a large volume of labeled training data. Hence, they work well in large or long-lived projects with version control and defect management systems. Machine learning models may use nonhandcrafted metrics (e.g., tokenized code), which can be difficult or impossible to interpret.

Mutation analysis is based on fault seeding, a fault-based test technique, adapted as a defect prediction technique. It is useful in the absence of real data. A mutation is a small syntactic change made to the source code using a predefined mutation operator. Each mutant introduces a plausible defect that mimics a potential programming defect. A test suite kills a mutant if at least one of its tests yields different test results for this mutant compared to the original code. Mutation analysis assumes that the effectiveness of a test suite in detecting actual defects is equivalent to its effectiveness in killing mutants.

Let 𝑋 be the (unknown) total number of real defects,

  • 𝐷 – the number of detected defects,
  • 𝑀 – the number of generated mutants,
  • 𝐾 – the number of killed mutants.

We assume that 𝐷 / 𝑋 = 𝐾 / 𝑀 , which allows us to calculate 𝑋 = (𝐷⋅ 𝑀) / 𝐾 , the total number of real defects.

The difference 𝑋 − 𝐷 represents the estimated number of remaining defects in the code.

Mutation analysis is a heuristic estimate, and its accuracy depends on the “representative nature” of the mutation operators used.

The Rayleigh model (Kan, 2002) is a defect arrival model used to forecast the number of defects over time. It is useful when assumed that the defect discovery rate follows a Rayleigh distribution, (i.e., it quickly rises to a peak and then slowly tapers off. The maximum of defects is reached when approximately 40% of the total number of defects are detected.

Show content Hide content
CT-DAT v1.0 Ch.3 – Digital Accessibility Testing Activities

3.2.3 – Digital Accessibility Audits as a Method for Evaluating Conformance

DAT-3.2.3: Summarize the process of digital accessibility audits as a method for evaluating conformance
K2 Understand 53% match
Go to Syllabus p.32

A digital accessibility audit is a formal and comprehensive evaluation of a digital product (e.g., a website or mobile application) to determine its level of conformance with a specific set of digital accessibility standards, most commonly the WCAG. Unlike ongoing testing, an audit is a structured, point-in-time assessment that results in a formal report.

The primary purpose of an audit is to provide a documented, scoped assessment of conformance against specified accessibility requirements and to support prioritization and remediation. The strength of any conclusion depends on the defined scope, sample, methods, and evidence.

The process of performing a digital accessibility audit typically involves the following:

  • Scope definition. This is to clearly define what will be audited. This includes: `o` The specific website, application, solution, or platform.
    • The targeted digital accessibility standard and version (e.g., WCAG 2.2, or an earlier WCAG version where required by the applicable policy, regulation, contract, or standard).
    • The targeted conformance level (A, AA, or AAA).
  • Representative sample of pages/screens. It is based on business requirements and decisions. It can be that all pages/screens are candidates for audit. In some cases, it is impractical to audit every single page/screen of a large solution. Instead, a representative sample is selected. This sample should include:
  • The homepage and other high-traffic pages/screens.
    • Pages/Screens representing key user flows (e.g., login, search, checkout process, etc.).
    • Pages/Screens that contain a wide variety of content and components (e.g., forms, tables, multimedia, complex widgets, etc.).
    • Pages/Screens containing legal information and policies.
  • Conformance evaluation. The audit team performs a thorough evaluation of the selected sample using a combination of automated, manual, and assistive technology test activities (as described in Section 3.2.2). Every applicable success criterion from the target standard is assessed against the sample pages/screens.
  • Logging findings and prioritization of defects. Every identified defect is meticulously documented. Each finding should include:
    • A clear description of the defect and its location.
    • The applicable accessibility requirement or WCAG success criterion(s), when the finding represents non-conformance.
    • The impact of the defect on users with different types of disabilities.
    • Recommendations for remediation.
    • Defects are often prioritized by severity (e.g., critical, serious, moderate) to help development teams address the most impactful defects first.
  • Audit report generation. The final deliverable is a formal report that summarizes the findings. This report typically contains:
    • An executive summary stating the scope and the conformance conclusion supported by the evaluation.
    • The scope and methodology of the audit.
    • A detailed list of all identified defects and recommendations for fixing them.
    • A concluding statement describing the evaluated product, scope, limitations, and principal accessibility findings.
Show content Hide content
CTAL-TA v4.0 Ch.5 – Software Defect Prevention

5.3 – Mitigating the Recurrence of Defects

A TA can proactively help minimize the recurrence of defects into the software. This syllabus discusses two approaches related to mitigating the recurrence of defects: analyzing test results to improve test analysis and test design and supporting root cause analysis with defect classification.

Show content Hide content
CT-TAS v1.0 Ch.4 – Organizational Deployment and Release Strategies for Test Automation

4.1.2 – Identify Ways in Which Test Automation Helps Verify Reported Defects

TAS-4.1.2: Identify ways in which test automation helps verify reported defects according to requirements
Related exam questions
K2 Understand 53% match
Go to Syllabus p.30

Confirmation testing performed following a code fix can address a reported defect. A tester typically follows the test steps necessary to replicate the defect to verify that the defect no longer exists.

Defects have a way of reintroducing themselves into subsequent releases (e.g., this may indicate a configuration management or code repository management problem) and therefore confirmation tests are suitable candidates for test automation and can be added to the existing regression test suite.

An automated confirmation test typically has a narrow scope of functionality. Implementation can occur at any point once a defect is reported and the test steps needed to replicate it are understood.

Tracking automated confirmation tests allows for reporting the time and the number of test cycles expended in resolving defects.

By verifying fixes across multiple platforms, devices, browsers, and OS versions with test automation, the amount of time spent on testing is greatly reduced.

Show content Hide content
CTAL-TAE v2.0 Ch.6 – Test Automation Reporting and Metrics

6.1.1 – Apply Data Collection Methods from the Test Automation Solution and the System Under Test

TAE-6.1.1: Apply data collection methods from the test automation solution and the system under test
Related exam questions
K3 Apply 53% match
Go to Syllabus p.38

Data can be collected from the following sources:

  • SUT logs
    • Web/mobile UI
    • APIs
    • Applications
    • Web servers
    • Database servers
  • TAF logs to provide an audit trail
  • Build logs
  • Deployment logs
  • Production logs to monitoring data in production (see the ISTQB CT-PT Syllabus, section 2.3)
    • Performance monitoring in production to perform trend analysis
    • Performance efficiency test logs in a performance test environment (e.g., load, stress, and spike testing)
  • Screenshots and screen recordings (native to the automation tool or 3rd party)

Since a TAS has automated testware at its core, the automated testware can be enhanced to record information about its use. Testware enhancements made to the underlying testware can be used by all the higher-level automated test scripts. For example, enhancing the underlying testware to record the start and end time of test execution may well apply to all tests.

Features of test automation that support measurement and test report generation

The scripting languages of many test tools support measurement and reporting through facilities that can be used to record and log information before, during, and after test execution of individual tests, and entire test suites.

Test reporting on each of a series of test runs needs to have an analysis feature to consider the test results of the previous test runs so it can highlight trends, such as changes in the test success rate.

Test automation typically requires automating both the test execution and the test verification, the latter being achieved by comparing specific elements of the actual results with the expected results. This comparison is best done by a test tool using assertions. The level of information that is reported as a result of this comparison must be considered. It is important that the test status be determined correctly (i.e., passed or failed). In the case of failed status, more information about the cause of the failure will be required (e.g., screen shots).

Differences between actual results and expected results of a test are not always clear, and tool support can help greatly in defining comparisons that ignore differences that are expected, such as dates and times, while highlighting any unexpected differences.

Test logging

Test logs are a source that is frequently used to analyze potential defects within the TAS and the SUT. In the following section are examples of test logging, categorized by TAS and SUT.

TAS logging

The context determines whether the TAF or the test execution is responsible for logging information that should include the following:

  • Which test case is currently under test execution, including start and end time
  • The status of the test execution because, while failures can easily be identified in test logs, the TAF should also have this information and should report via a dashboard. The test execution status can be either passed, failed, or a TAS failure. Status can sometimes be inconclusive and it is important for an organization to define clear and consistent definitions for them. A TAS failure is applied to situations where the defect is not in the SUT.
  • Low-level details of the test log (e.g., logging significant test steps) including timing information
  • Dynamic information about the SUT (e.g., memory leaks) that the test case was able to identify with the help of third-party tools. Actual results and failures should be logged with the test suite that was executed when the failure was detected.
  • In the case of reliability testing or stress testing where numerous test cycles are performed, a counter should be logged to easily determine how many times test cases have been executed.
  • When test cases have random elements (e.g., random parameters or random test steps in state transition testing), the random number/choices should be logged.
  • All actions that a test case performs should be logged in such a way that the test logs, or parts of it, can be played back to re-run the test with the same test steps and the same timing. This is useful to reproduce an identified failure and to capture additional information. The test case action information can also be logged by the SUT for use when reproducing customer identified failures. If a customer runs a test suite, the test log information is captured and can then be replayed by the development team when troubleshooting a defect.
  • Screenshots can be saved during test execution for further use during root cause analysis.
  • Whenever a test suite initiates a failure, the TAS should make sure that all information needed to analyze the defect is available/stored, as well as any information regarding the continuation of testing, if applicable. The TAS should save any associated crash dumps and stack traces. Also, any test logs which can be overwritten (e.g., cyclic buffers are often used for test logs on the SUT) should be stored where they will be available for later analysis.
  • Use of color can help to distinguish different types of test log information (e.g., defects in red, and progress information in green).

SUT logging

Correlation of the test automation results with SUT logs to helps identify the root cause of defects in the SUT and the TAS.

  • When a defect is identified in the SUT, all necessary information needed to analyze the defect should be logged, including date and time stamps, source location of the defect, and error messages.
  • At the startup of a system, configuration information should be logged to a file, consisting of, for example, the different software/firmware versions, configuration of the SUT, and configuration of the operating system.
  • Using test automation, SUT logs can be easily searchable. A failure identified in the test log by the TAS should be easily identified in the test log of the SUT, and vice versa, with or without additional tools. Synchronizing various test logs with a time stamp facilitates correlation of what occurred when a failure is reported.

Integration with other third-party tools (e.g., spreadsheets, XML, documents, databases, and report tools)

When information from the execution of automated test cases is used in other tools for tracking and reporting (e.g., updating traceability information), it is possible to provide the information in a format that is suitable for third-party tools. This is often achieved through existing test tool functionality (e.g., export formats for test reporting) or by creating customized reporting that is output in a format consistent with other software.

Visualization of test results

Test results can be made visible using charts. Consider using colored icons such as traffic lights to indicate the overall status of the test execution/test automation so that decisions can be made based on reported information. Management is particularly interested in visual summaries to see the test results, which aides in decision making. If more information is needed, they can still drill down into the details.

Show content Hide content
CT-GT v2019 Ch.3 – Testing in the Gambling Industry

3.6.6 – Hardware integration defects

GTFL-3.6.6: Identify common areas where hardware integration defects are found
Related exam questions
K2 Understand 52% match
Go to Syllabus p.66

Hardware integration defects are often found in the following areas:

  • BNA
  • Hopper payouts
  • Edge lighting
  • Printers
  • Coin acceptors
  • Light tower functionality
  • Topper
  • LCD buttons
  • Cabinet door
  • Monitor and touch screens
  • Mechanical reels
  • Firmware
  • Drivers
Show content Hide content
CTEL-ITP v2011 Ch.6 – Process for Improvement

6.3.5 – Analysis of Results

LO-6.3.5: Recommend test process improvement actions on the basis of assessment results and the analysis performed
Related exam questions

If an analytical approach to improvement is being used (Chapter 4), the current situation may be analyzed by applying concepts such as:

  • Systems Thinking [Weinberg 92]
  • Tipping Points [Gladwell]

Systems Thinking helps analyze the relationships between different system (process) components and to represent those relationships as stable (“balancing”) loops or reinforcing loops. A reinforcing loop may have a negative effect (“vicious circle”) or a positive effect (“virtuous circle”).

Tipping Points help identify specific points in a system where a small, well-focused improvement may break a vicious circle and set off a chain reaction of further improvements.

When a model-based approach (Chapter 3) for test process improvement is followed, a comparison is made between the process maturity of the current situation and the desired objectives defined at the initializing phase.

Where appropriate benchmarks are available, these should be used in evaluating results. The following benchmarks may be used, where available:

  • Company-based, representing an entire organization
  • Industry-based, where possible relating to the same business segment
  • Project-based, where the comparison is made with a specific project that is regarded as meeting the desired objectives

Where key performance indicators have been established (see Section 4.4 and Section 6.2.2), these should be incorporated into the analysis. For example, if the Defect Density Percentage (DDP) has fallen below the required level, an analysis of faults found in production should be performed to evaluate their sources.

The result of the evaluation should provide sufficient information with which to define recommendations and support the planning process (see Sections 6.3.7 and 6.4 below).

Show content Hide content
CT-STE v1.0.1 Ch.8 – Reporting Security Test Results

8.3.2 – Avoiding a Vulnerability

Generally, avoiding the vulnerability is a very time consuming and expensive action. The following steps must be performed to avoid a vulnerability:

StepDescription
Locate the
vulnerability
• As the risk mitigation action must be taken at the level used to
implement the functionality (e.g., code, models, and
configurations) it might take time to identify the affected
component and within that component the affected area based
on an identified vulnerability at the system level
Understand the
vulnerability
• Before making a repair, a full understanding of the vulnerability
must be obtained by analysis of the vulnerability in the affected
area (e.g., code snippet)
Identify the risk
mitigation action
• An approach for risk mitigating must be developed.
• The mitigation action might consist of a completely new
algorithm, a new component, a small configuration change or
only some minor code adjustments (e.g., including a specific
exception behavior).
Execute the risk
mitigation action
• The identified risk mitigation action is applied.
Confirmation test• Perform a confirmation test on the system to test if the
vulnerability has been eliminated.
Regression test• Testing is a fundamental part of avoiding vulnerabilities and
should not only focus on the changed code. It is essential that
the complete regression test suite is executed to make sure that
the system is still running correctly, and the mitigation action has
had no unwanted side effects.
Deploy• Deploy the patched system.
• After the system is deployed there is usually some close
monitoring for a period to be sure that the system is running
well.
Show content Hide content
CTEL-ITP v2011 Ch.6 – Process for Improvement

6.3.7 – Recommending Improvement Actions

An assessment report must relate results to the specified test improvement objectives. The report must be delivered as soon as possible after completion of the assessment, possibly as a preliminary version followed later by a complete version.

As a minimum, the assessment report must include:

  • A management summary which refers to the vision statement (see Section 6.2.2.1) A statement of scope and objectives
  • Results of the analysis including
    • Positive aspects
    • Aspects which need improving
    • Open issues
  • A list of improvement recommendations

Where possible, recommendations should be thought of as improvement requirements (e.g., “provide tool support for a defect tracking system”) which may be implemented in a number of different ways (e.g., “use tool XYZ and provide training”). The implementation task is covered in Section 6.5.

An improvement recommendation should include the following information:

  • A unique identifier (for traceability)
  • The impact of the recommendation on one or more stated objectives (where possible using a scale to indicate the degree of fulfillment, such as “minimal”, “partial”, “full” or a percentage value)
  • Estimate of cost and benefits
  • Implementation timescale (e.g., short-term, medium-term, long-term)
  • Risks of implementation (e.g., high level of resistance expected in the change process, risks to achieving particular improvements)
  • Dependencies and assumptions (e.g., assumes that another recommendation is also implemented)

To assist in the planning and tracking of high-level recommendations they should, where possible, be broken down into small steps of improvement with tangible results.

Some process models, such as TPI Next, include specific improvement suggestions to help in the task of creating recommendations.

Show content Hide content
CTAL-TTA v4.0 Ch.4 – Quality Characteristics for Technical Testing

4.4.3 – Testing for Availability

Availability is typically specified in terms of the amount of time a system (or software) is available to users and other systems under normal operating conditions. Systems may have a low maturity, but still have a high availability. For instance, a phone network may fail to connect several calls (and thus have low maturity), but as long as the system recovers quickly and allows the next attempts to connect most users will be content. However, a single failure that caused a phone network outage for several hours would represent an unacceptable level of availability. Availability is often specified as part of an SLA and measured for operational systems, such as websites and software as a service (SaaS) applications. The availability of a system may be described as 99.999% (‘five nines’), in which case it should be unavailable no more than 5 minutes per year, alternatively system availability may be specified in terms of unavailability (e.g., the system shall not be down for more than 60 minutes per month).

Measuring availability prior to operation (e.g., as part of making the release decision) is often performed using the same tests used for measuring maturity; tests are based on an operational profile of expected use over a prolonged period and performed in a test environment as close to the operational environment as possible. Availability can be measured as MTTF/(MTTF + MTTR), where MTTF is the mean time to failure and MTTR is the mean time to repair (MTTR), which is often measured as part of maintainability testing. Where a system is high-reliability and incorporates recoverability (see section 4.4.5) then we can substitute mean time to recover for MTTR in the equation when the system takes some time to recover from a failure.

Show content Hide content
CTEL-ITP v2011 Ch.4 – Analytical-based Improvement

4.4.1 – Introduction

LO-4.4.1: Provide examples of the various categories of metrics and how they can be used in a test improvement context
K2 Understand 52% match
Go to Syllabus p.32

Measures, metrics and indicators form part of all improvement programs. This applies regardless of whether these improvements are carried out formally or informally. It is also regardless of whether the data are quantitative or qualitative, objective or subjective. The feelings of the people affected by the improvement are a valid measure of progress toward improvement.

Measures, metrics and indicators initially help to target areas and opportunities for improvement. They are required continuously in improvement initiatives in order to control the improvement process and to make sure that the changes have resulted in the desired improvements.

Measures, metrics and indicators can be collected at all stages of the software life cycle, including development, maintenance and use in production [Nance & Arthur 02]. They are also used for deriving other metrics and indicators. Note that for all items mentioned in Section 4.4.2 which relate to defects, it is important to make a distinction between the various priority and severity levels of the defects found. Specific measures, metrics and indicators may also be applied by test managers, in particular for the project level tasks of test estimation and for progress monitoring and control. Test process improvers will apply measures, metrics and indicators at the process level.

Show content Hide content
CTAL-TA v4.0 Ch.5 – Software Defect Prevention

5.2.2 – Applying Review Techniques

TA-5.2.2: Apply a review technique to a test basis to find defects
Related exam questions
K3 Apply 52% match
Go to Syllabus p.51

A well-defined test basis serves as the cornerstone for ensuring the quality of work products and the success of projects. Reviewing the test basis helps identify and address defects early, preventing defects from escaping to subsequent phases.

The TA can employ various review techniques during the individual review to identify defects in the test basis. Selecting the most appropriate review technique can enhance the efficiency and effectiveness of the review. This selection should consider factors such as reviewing goals, project objectives, available resources, test basis type, associated risks, business domain, and company culture.

Below, this syllabus discusses five review techniques commonly used by the TA.

Ad hoc reviewing is carried out by reviewers informally, without a structured process. Reviewers are provided with little or no guidance on how the task should be performed. Ad hoc reviewing needs little preparation and is highly dependent on the reviewers’ skills. During the review, reviewers read the test basis and document the anomalies as they encounter them. This technique, left unmanaged, can lead to a high volume of duplicate anomaly reports from multiple reviewers.

Checklist-based reviewing involves evaluating the test basis against a predefined checklist. Checklists remind reviewers to check specific points and can de-personalize the review. Checklists can be generic or specific to quality characteristics, test objectives, or test basis type. The TA tailors the checklist to the test basis type, risk level, or test condition. This ensures that the review will focus on the most relevant aspects of the test basis. Checklists should be regularly updated with previously missed defects. Keeping checklists current prevents overlooking newly identified anomalies. Checklists are not all-inclusive. Therefore, the TA is not limited to checking the listed items. This maximizes defect detection and allows the TA to capture anomalies that checklists may not explicitly cover.

Scenario-based reviewing involves simulating a process or activity to identify anomalies and refine the test basis. This review technique is most effective when the test basis has a scenario-based format, such as a use case or activity diagram. In such cases, reviewers can perform "dry runs" based on the expected usage of the work product. Scenarios provide valuable guidelines, but the reviewers are not constrained to documented scenarios and should also think beyond the scenarios to identify more anomalies.

Role-based reviewing involves assigning specific roles or responsibilities to reviewers. Typical roles are based on specific end-user types (e.g., experienced, inexperienced, senior, or child user) or specific roles in the organization (e.g., administrator or regular user). Each role can be described by a persona (i.e., the concrete but fictional character designed to represent the characteristics, needs, goals, and preferences of a particular group of users). By distributing responsibilities among reviewers based on their roles, rolebased reviews allow individuals to focus on specific aspects of the test basis, ensuring comprehensive coverage and, at the same time, avoiding duplication of anomalies.

Perspective-based reading involves reviewing the test basis from various perspectives or viewpoints (e.g., designer, tester, marketer, administrator, and end user). This leads to more in-depth individual reviewing with less duplication of anomalies among reviewers. In addition, perspective-based reading requires the reviewers to attempt to use the test basis under review to generate the work product they would derive from it. For example, a tester would attempt to generate draft acceptance tests based on requirements specifications to see if all the necessary information is included.

Show content Hide content
CTEL-TM v2.0 Ch.5 – Project Management Essentials

5.1 – Test Estimation and Defect Prediction

Section 5.1 - Test Estimation and Defect Prediction (Chapter 5: Project Management Essentials).
Show content Hide content
CT-AuT v2.1.1 Ch.2 – Standards for the Testing of Electric/Electronic (E/E) Systems

2.1.2.6 – Traceability in ASPICE

AuT-2.1.2.6: Explain the different traceability requirements of ASPICE from the testing perspective
Related exam questions
K2 Understand 52% match
Go to Syllabus p.23

As in the CTFL® Core Syllabus [ISTQB_FL_SYL], ASPICE additionally requires bidirectional traceability[11] . This allows the testers to:

  • Analyze impact, e.g., change impact analysis
  • Evaluate coverage
  • Track status

Moreover, this allows the tester(s) to ensure the consistency between the linked elements, textually and semantically.

ASPICE differentiates between vertical and horizontal traceability:

Vertically: ASPICE requires that stakeholder requirements be linked across all levels to the software units. In doing so, the link over all levels of development ensures consistency between the related work products.

Horizontally: ASPICE requires traceability and consistency, between the work results of the development and the corresponding test specifications and test results.

In addition, the base practice SUP.10.BP4 requires bidirectional traceability between change requests and work products affected by the change requests. Because change requests are often initiated by a defect, bidirectional traceability is established between change requests and the corresponding defect reports. Because of the occasionally large number of links, a consistent chain of tools can be helpful. This allows the tester to efficiently create and manage the dependencies.

11 In the following, the term traceability will always imply the bidirectional traceability.

Show content Hide content
CT-GT v2019 Ch.3 – Testing in the Gambling Industry

3.1.2 – Gambling and wagering functional testing defects

GTFL-3.1.2: Identify common areas where gambling and wagering defects are found
Related exam questions
K2 Understand 52% match
Go to Syllabus p.50

When performing gambling and wagering functional testing there are specific areas to target. Examples of these areas include the following:

  • Game rules
  • Digital graphics – hardware video components
  • Graphics – image components
  • Digital software
  • Accounting - money wagered on a game is correctly accounted for
  • Game or player history
  • Pay tables
  • Game play
  • Menus
  • Audio components
  • Peripherals
  • Help files
  • Overall game fun or player perspective

The types of defects that are often found in the above areas include, but are not limited to the following examples:

  • Documentation does not comply with the game specifications
  • Game rules are not followed
  • Graphics, both video and images, are not synchronized with the game and do not follow game specifications
  • Audio is not synchronized with the game and does not follow game specifications
  • Game software stops or does not work correctly
  • The player history, what the player put in the machine and their results are not accurate
  • Menus and help files are wrong versus the game specifications
  • When two buttons are hit simultaneously by accident the game does not handle the situation as expected
  • The game does not play well and provides pleasure to the player
  • Game does not pay the correct amounts of money as per the game specifications
  • There are loopholes in the game where someone can change the software or cheat
  • Legal specifications in each jurisdiction have not been met where the game will be played
    • The game does not handle the loss of power as expected
Show content Hide content
CTAL-TA v4.0 Ch.5 – Software Defect Prevention

5.2 – Supporting Phase Containment

The objective of phase containment is to detect and remove defects in the same phase of the SDLC in which they were introduced. In Agile software development, the shift left approach can be used similarly. This policy reduces the cost of quality. Because the test basis is an important input to test analysis and test design, the TA can best contribute to phase containment by evaluating the test basis quality. Early attention to test basis quality will minimize later effort and prevent the propagation of defects to subsequent phases. This syllabus discusses two common options for the TA to find defects in the test basis: modeling for testing purposes and reviewing the test basis using various review techniques ( ISO/IEC 20246 , 2017).

Show content Hide content
CT-AuT v2.1.1 Ch.2 – Standards for the Testing of Electric/Electronic (E/E) Systems

2.1.2.4 – Testware in ASPICE

AuT-2.1.2.4: Recall the requirements of ASPICE for testware
K1 Remember 51% match
Go to Syllabus p.22

ASPICE 4.0 does not impose requirements on testware. Instead, it establishes requirements concerning the documentation of specific information that arises during testing. For test activities, ASPICE requires the following information items (II):

  • 08-60: Verification measures (e.g., simulations, dynamic tests and static tests), typically documented in a test plan
  • 15-52: Verification results (e.g., documented verification logs, test reports and defect reports)

For each information item, ASPICE defines examples of information item characteristics (IIC) and content. They serve as an objective indicator for process execution.

ISO/IEC/IEEE 29119-3 can be used to further structure information items.

Show content Hide content
CT-GenAI v1.1 Ch.2 – Prompt Engineering for Effective Software Testing

2.2.1 – Test Analysis with Generative AI

GenAI-2.2.1: Apply generative AI to test analysis tasks
K3 Apply 51% match
Go to Syllabus p.24

GenAI can support test analysis tasks by generating and prioritizing test conditions, identifying defects in the test basis and providing coverage analysis. The input data includes requirements, user stories, technical specifications, GUI wireframes and other relevant information. The output consists of typical test analysis work products, such as prioritized test conditions (e.g., acceptance criteria).

Here are some typical test analysis tasks that can be supported by GenAI:

  • Identify potential defects in the test basis : GenAI can help analyze the test basis for inconsistencies, ambiguities, or incomplete information that could lead to defects. By comparing similar requirement patterns or applying knowledge from previous defect reports, the LLM can flag potential anomalies and suggest improvements.
  • Generating test conditions based on the test basis, for example on requirements/user stories: LLMs can analyze requirements and user stories to generate test conditions. Using natural language processing, they can interpret the meaning of requirements and break them down into measurable, testable statements. This can help translate requirements into specific test conditions.
  • Prioritize test conditions based on risk level: With information on the risk likelihood and risk impact of failure for each test condition, an LLM can help prioritize test effort. By considering aspects such as regulatory compliance, user-facing features (e.g., login functionality or payment processing), and historical defect data, the LLM can recommend priority levels.
  • Support coverage analysis: By mapping requirements and user stories to test conditions, an LLM can perform coverage analysis to determine whether all aspects of the test basis are covered. This is particularly useful for projects with complex requirements, where gaps in coverage can lead to escaped defects.
  • Suggest test techniques: GenAI can suggest relevant test techniques (e.g., boundary value analysis, equivalence partitioning) based on the type of requirement or user story being tested. This can help testers apply the most effective test techniques for specific test conditions.

The quality and relevance of inputs provided to the LLM in relation to the task to be completed directly impact the accuracy and precision of the output generated by the LLM.

Hands-On Objective 2.2.1a (H2): Practice creating structured multimodal prompts to generate acceptance criteria for a user story based on a GUI wireframe

This is an exercise to practice writing structured prompts using multimodal input (text and image). The goal is to generate high quality (i.e. well-formed, clear and complete) acceptance criteria from a user story and a GUI wireframe. Other text elements can be added to provide context, such as constraints on input fields or business rules to be applied to data processing.

The results obtained from the LLM are compared to assess the impact of different formulations of the structured prompt (role, context, instruction, textual and image input data, constraints, and output format) for a test analysis task.

This exercise provides practical experience in the importance of prompt structuring, the contribution of precise instructions, and the importance of both textual and image contextual data in obtaining accurate and relevant results from the LLM.

Hands-On Objective 2.2.1b (H2): Practice prompt chaining and human verification to progressively analyze a given user story and refine acceptance criteria

This is an exercise to practice prompt chaining to analyze a given user story and refine acceptance criteria, first by identifying ambiguities, then by evaluating testability, and finally by evaluating completeness. This exercise encourages a step-by-step approach, refining the analysis at each step to ensure that the acceptance criteria are well-formed and actionable to achieve the test objectives. At each step, the results provided by the LLM are manually verified and corrected, if necessary, either by adjusting the output or through a prompt chaining process with the LLM. In this way, the next stage uses a clean result from the previous stage to address another aspect of improving the acceptance criteria.

This exercise provides practical experience of the benefits of breaking down a complex task into subtasks, with human verification of the results of each stage.

Show content Hide content
CTFL v4.0.1 Ch.4 – Test Analysis and Design

4.4.1 – Error Guessing

FL-4.4.1: Explain error guessing
Related exam questions
K2 Understand 51% match
Go to Syllabus p.43

Error guessing is a test technique used to anticipate the occurrence of errors, defects, and failures, based on the tester’s knowledge, including:

  • How the application has worked in the past
  • The types of errors the developers tend to make and the types of defects that result from these errors
  • The types of failures that have occurred in other, similar applications

In general, errors, defects and failures may be related to: input (e.g., correct input not accepted, parameters wrong or missing), output (e.g., wrong format, wrong result), logic (e.g., missing cases, wrong operator), computation (e.g., incorrect operand, wrong computation), interfaces (e.g., parameter mismatch, incompatible types), or data (e.g., incorrect initialization, wrong type).

Fault attacks are a way to implement error guessing. This test technique requires the tester to create or acquire a list of possible errors, defects and failures, and to design tests that will identify defects associated with the errors, expose the defects, or cause the failures. These lists can be built based on experience, defect and failure data, or from common knowledge about why software fails.

See (Whittaker 2002, Whittaker 2003, Andrews 2006) for more information on error guessing and fault attacks.

Show content Hide content
CTEL-TM v2.0 Ch.7 – Evaluating Effectiveness, Efficiency and Satisfaction and Improving Test Policy

7.2 – Effectiveness, Efficiency and Satisfaction Metrics for the Test Policy

Test policy metrics quantitatively monitor how test activities support strategic test objectives, aligning test results with business goals. Their scope is organizational and strategic, targeting senior management, QA/QC leadership, stakeholders, and they are reviewed periodically. The TME is responsible for embedding these metrics into organizational test practices.

Metrics should measure effectiveness, efficiency, and satisfaction, and may extend existing project metrics to quantify process quality and risk. They can also be aggregated across related projects within the same organization or product line. As test objectives vary, suitable metrics differ and may cover testing, the SDLC, and project management.

Test policies effectiveness metrics

Requirements stability index ( 𝑅𝑆𝐼 ) measures the rate of change in baselined requirements. This index measures how volatile the test scope is that the test team must validate. It is a key predictor of test stability.

A low 𝑅𝑆𝐼 (e.g., below 80%) indicates wasted test design, as requirement changes make test cases obsolete. An effective test policy should require requirement reviews as entry criteria to stabilize 𝑅𝑆𝐼 before test design begins.

Test objective stability index ( 𝑇𝑂𝑆𝐼 ) measures whether the strategic test objectives (TO) as defined in the test policy remain consistent throughout the test project lifecycle.

A low 𝑇𝑂𝑆𝐼 indicates a mid-stream shift in the test mission (e.g., from risk mitigation to date-driven release). While high stability can indicate a mature organization with a robust test policy, 𝑇𝑂𝑆𝐼 must be interpreted with caution. Because not all test objective changes carry equal weight, a low score should trigger a qualitative review to distinguish between reactive management and necessary strategic adaptation. Rather than aiming for a perfect 100%, 𝑇𝑂𝑆𝐼 should be used to provide visibility into the impact of shifting priorities on the overall test strategy.

Test policy compliance rate ( 𝑇𝑃𝐶𝑅 ) measures how well test projects or test teams comply with mandatory test policy rules, such as peer reviews, coverage, and tool usage.

𝑇𝑃𝐶𝑅 can be calculated per activity, project, or test team. A low 𝑇𝑃𝐶𝑅 may indicate an overly bureaucratic or misaligned test policy (e.g., forcing sequential development models on Agile teams). The TME can use 𝑇𝑃𝐶𝑅 to identify test teams needing coaching rather than auditing. A high 𝑇𝑃𝐶𝑅 should correlate with lower 𝐸𝐷𝑅 ; if not, the test policy is flawed.

Assessment or audit pass rate ( 𝐴𝐴𝑃𝑅 ) measures the results of formal evaluations, such as TMMi (TMMi, 2022), ISO 29119, or regulatory audits, against defined standards.

While 𝑇𝑃𝐶𝑅 measures rule adherence, 𝐴𝐴𝑃𝑅 measures their effectiveness. In regulated sectors, a high 𝐴𝐴𝑃𝑅 reduces the cost of non-compliance, such as fines, product recalls, and legal actions.

Test policy efficiency metrics

Reduction rate of test cycle time ( 𝑅𝑅𝑇𝐶 ) measures how efficiently a test policy changes, (e.g., CI/CD pipeline integration, automated regression testing, or shift left), reduces test cycle time (TCT), from build readiness to final test report.

Where 𝐵𝑎𝑠𝑒𝑙𝑖𝑛𝑒 𝑇𝐶𝑇 is the average test time before a test policy change, 𝐶𝑢𝑟𝑟𝑒𝑛𝑡 𝑇𝐶𝑇 is after improvements are made. The TME considers more than test cycle times, which can be inflated by delays (e.g., for test environments or test data). If the test policy ignores SLAs for test environments, 𝑅𝑅𝑇𝐶 stays low despite improvements like increased test automation. Reductions in time are only efficient if 𝐷𝐸𝑅 remains stable, as faster testing at the expense of quality is risky.

Reuse rate of testware ( 𝑅𝑅𝑇𝑊 ) measures how effectively a test policy enforces modularity and standardization, such as using a central test data repository and keyword-driven test automation.

𝑅𝑅𝑇𝑊 can be calculated at the test, project, or test team level and measures reuse of testware (e.g., test cases, test data, test scripts and test tools) used in new contexts with minimal changes (e.g., >80% identical). The TME distinguishes cloning from true reuse (i.e., referencing a single source) which reduces maintenance. An effective test policy should provide a common testware library. Low 𝑅𝑅𝑇𝑊 (e.g., below 30-40%) indicates repeated reinvention.

Defect delay index ( 𝐷𝐷𝐼 ) measures defect decay, (i.e., the period from defect introduction to defect removal) since costs and risks rise exponentially the longer a defect remains unresolved.

Phases should be numbered, e.g., requirements = 1, design = 2, coding = 3, component testing = 4, component integration testing = 5, system testing = 6, acceptance testing = 7, production = 8.

𝐷𝐷𝐼 = 0 means defects are contained in the phase where they originated. 𝐷𝐷𝐼 > 0 signals higher risk and cost as defects escaped quality gates. A high 𝐷𝐷𝐼 reflects technical debt, leading to costly hotfixes and downtime.

Test overhead ratio ( 𝑇𝑂𝑅 ): measures a test policy’s “bureaucracy” by comparing effort spent on test process management versus test analysis, design, and execution.

Management efforts (numerator) include meetings, reports, and audits. Technical efforts (denominator) include test activities of the given test process, such as defining entry and exit criteria, designing the physical test environment infrastructure, executing regression tests, and archiving testware. A healthy 𝑇𝑂𝑅 is 0.1-0.2. Above 0.25, the test policy is too heavy, with testers spending more time on test process overhead rather than technical efforts (i.e., actual testing), common in highly regulated industries.

Test policy satisfaction metrics

Feedback-to-test policy ratio ( 𝐹𝑇𝑃𝑅 ): measures test policy engagement and evolution, showing whether it is a living document or static shelfware.

Feedback on the test policy includes improvement suggestions, exception requests, and issues, often submitted as change requests. A zero 𝐹𝑇𝑃𝑅 usually signals the policy is ignored, while a high 𝐹𝑇𝑃𝑅 suggests it is too vague or restrictive. The TME uses 𝐹𝑇𝑃𝑅 to trigger test policy reviews and updates, such as adjusting test automation mandates for CI/CD pipelines.

Test governance maturity score ( 𝑇𝐺𝑀𝑆 ). This composite score, from internal assessment or test frameworks like TMMi, measures the organization’s test policy maturity.

where is a rating for 𝑖 -th governance domain (e.g., test policy and test strategy, test organization, infrastructure, operational processes, and people) and denotes its importance. 𝑇𝐺𝑀𝑆 is calculated using a structured rating system and weighted priorities, based on institutionalization and evidence-based performance in each governance domain. For example, scale can be 1–5, as in TMMi (1 = initial, 5 = optimization).

Weights can be adjusted to align with organizational goals (e.g., shifting from infrastructure to operational processes to prioritize DevOps). No domain should be weighted 0%, as gaps, especially in infrastructure or people domain, can cause failures. 𝑇𝐺𝑀𝑆 highlights maturity gaps to justify test process improvements to executives.

Summary

To design test policy metrics, the TME can use the GQM approach (Basili, 1984). Metrics should be tied to follow-up actions like improving communication, test policy, or feedback loops. Metrics must be evaluated contextually, not in isolation, using three comparative lenses:

  • Industry benchmarks such as (PractiTest, 2025), (Sogeti, 2024). (Jones, 2011) or (van Veenendaal, 2022) – e.g., to have EDR <10%, DDP of 85–95%, test policy compliance rate >90% in regulated sectors
  • Similar projects to compare metrics before and after test policy implementation or across projects with similar size, stack, or test team composition
  • Product families to make comparisons across shared architectures, product variants, or test teams using the same test policy to assess reuse and defect detection

The TME should normalize metrics to account for test team size, release cycle, product complexity, and test automation maturity. The TME should track trends via dashboards or time series charts to monitor adoption, fatigue, and correlations between effectiveness and satisfaction, and assess test policy maturity using models like TMMi or TPI NEXT® (Sogeti, 2009).

Show content Hide content
CTFL v4.0.1 Ch.1 – Fundamentals of Testing

1.1.2 – Testing and Debugging

FL-1.1.2: Differentiate testing from debugging
Related exam questions
K2 Understand 50% match
Go to Syllabus p.16

Testing and debugging are separate activities. Testing can trigger failures that are caused by defects in the software (dynamic testing) or can directly find defects in the test object (static testing).

When dynamic testing (see chapter 4) triggers a failure, debugging is concerned with finding causes of this failure (defects), analyzing these causes, and eliminating them. The typical debugging process in this case involves:

  • Reproduction of a failure
  • Diagnosis (finding the defect)
  • Fixing the defect

Subsequent confirmation testing checks whether the fixes resolved the problem. Preferably, confirmation testing is done by the same person who performed the initial test. Subsequent regression testing can also be performed, to check whether the fixes are causing failures in other parts of the test object (see section 2.2.3 for more information on confirmation testing and regression testing).

When static testing identifies a defect, debugging is concerned with removing it. There is no need for reproduction or diagnosis, since static testing directly finds defects, and cannot cause failures (see chapter 3).

Show content Hide content
CT-TAS v1.0 Ch.5 – Test Automation Impact Analysis

5.4.1 – Analyze Test Report Data to Inform Decision Making

TAS-5.4.1: Analyze test report data to inform decision making
Related exam questions
K2 Understand 50% match
Go to Syllabus p.43

The format and the content of a test automation report may vary depending on the stakeholders receiving it. It can be created for management, operational or technical stakeholders. Additional information can be found in the CTAL-TAE Syllabus, section 6.1.3.

Upon receiving a test automation report, it can be incorporated into a broader test report, or it can be consolidated, and then escalated within the organizational structure.

Different stakeholders find different values in such a test automation report. A strategic approach is to identify key metrics that are important for the stakeholders involved by emphasizing these important metrics.

Data collected with automation can help:

  • Identify trends and perform root cause analysis
  • Shift test automation efforts to maintenance
  • Shift test automation efforts to improvements and further development of a TAF
  • Add capabilities to the overall TAS
  • Increase shift left and shift right approaches to testing
  • Expand the functional coverage of test automation in future sprints
  • Focus more on defect clusters
  • Advise developers of areas to improve the code
  • Advise on the overall SDLC processes
  • Change the content and format of future test automation reports

Based on the information described above, the TAEs in collaboration with other stakeholders can identify gaps and certain improvement points in the existing test automation coverage and test results.

Show content Hide content
Export to PDF
Layout
Content
Exam Questions
Results to Include 0 / 0
Filters
Share Your Feedback

How is your experience with TBoK?