STEs need to understand the nuances of different lines of defense so that appropriate tests can be designed to verify and validate their effectiveness.

How to Test System Hardening

Testing the effectiveness of system hardening can be accomplished in a variety of ways. System hardening restricts the access of the system to the right roles, opens only the needed services, and monitors application updates. Therefore, to test the effectiveness of system hardening, tests should be designed to detect whether the system hardening measures are working, applied in the right places, and in the right ways. It is also relevant to test for system hardening protections that are too restrictive and might be excessive compared with the security risks.

Some system hardening tests may be based on a review or an audit, while others may be based on the ability of certain user groups to perform certain actions, or to access certain data.

How to Test Firewalls

Due to the number of protocols, their different options and the complexity of the networks to be protected, it is difficult to configure a firewall efficiently and consistently. Tests for firewall effectiveness should include:

• Performing an audit to check the firewall configuration
• Port scanning to verify if the security policy is well implemented
• Using malformed network packets and network fuzz testing to exploit unexpected behavior
• Fragmentation attacks to bypass filtering features with the objective of carrying out an attack behind the firewall
• Targeting the web application firewall by encoding and compressing data or obfuscating it to hide the malicious information that represents the attack. Web application firewall evaluation criteria [WAFEC] can be used to test the effectiveness of a web application firewall.

How to Perform Intrusion Detection

Scenario-based detection is based on a known scenario or “signature”. It is easy to bypass because only known attacks are detected. Tests could include the following evasion techniques:

• Character encoding or modification of data (e.g., adding white space and end of line indicators)
• Internet protocol (IP) fragmentation, transmission control protocol (TCP) segmentation
• Encryption or obfuscation
• URL encoding

Behavior-based detection is based on a model of the system behavior and generates a large number of false-positive results and false-negative results. A false-negative result is any security alert that should have been reported but was not. False-negative results can occur when a new attack is developed that an intrusion detection system (IDS) is not aware of, or perhaps a rule might be written in such a way as to detect some attacks but miss those not specified in the model.

The accuracy of this detection method should be maintained. It is possible for an attacker to deviate from normal IDS behavior, which results in a new specification for intrusive behavior. Complementary tests should use malicious traffic to add new intrusive specifications to be considered as authorized traffic.

How to Perform Malware Scanning

Developers of malware use different techniques to protect their code against reverse-engineering and detection by anti-malware software. Some of these techniques include:

• Exploiting system library functions used by anti-malware
• String obfuscation to disable the understanding of malicious code behavior
• Code permutation
• Insertion of unused code
• Dynamic loading of functions and libraries (e.g., to limit the analysis of the malicious code)
• Automatic update of applications

From the functional suitability testing perspective, signature based anti-malware tools could be used to test the effectiveness of anti-malware without developing real malicious pieces of code. Other types of malicious files must be tested regarding the type of applications

The testing of behavior-based anti-malware is difficult because there is no clear understanding and definition of what malicious behavior is. Ideas for testing can benefit from techniques used by developers of malware:

• Unsigned execution files trying to use system calls to make system changes
• Trying to launch unusual processes with granted rights
• Attempting to copy execution files in unauthorized locations
• Trying to call unusual system APIs

An important consideration when implementing new anti-malware (either based on signature or behavior) or upgrading existing anti-malware is to test the implementation on a representative platform before deploying it to the entire organization.

Testing Data Obfuscation

Strict configuration control between the obfuscated data and keys used for the obfuscation is needed to ensure the correct versions of keys are used. Otherwise, the data cannot be understandable for use.

Since private data could be involved in some tests, data obfuscation may be used for testing purposes to render production data used in a system test environment anonymously. Sensitive data, such as user information used by a health information system, must not be divulged to testers. Tests could include brute force or dictionary attacks to attempt to get plain data from obfuscated data.

Tests to verify obfuscation of code could include:

• Reverse-engineering of code
• Brute force attacks, because some obfuscation mechanisms are vulnerable

Test results allow the TA to identify failures, but they also provide feedback to the TA to help improve defect detection effectiveness. Some commonly used techniques to analyze test results are described below.

Predicted versus actual defect cluster analysis . A few components usually contain most defects (see ISTQB-CTFL, v4.0.1, Section 1.3). After testing, the TA can identify the actual defect-prone areas and compare predicted versus actual defect clusters. In the case of discrepancies, more rigorous testing may be applied to areas where more defects were found than expected. When determining the clusters, measurable criteria should ensure clarity and consistency (e.g., defect density and defect severity). Among these criteria, the severity of the defects should play a significant role. Small clusters of critical or major defects are usually more important (i.e., need more rigorous testing) than larger clusters of minor or cosmetic defects.

Defect detection percentage (DDP) analysis . DDP is one of the most important test effectiveness measures for a test level. When calculating DDP, the number of escaped defects should be limited to those that the test level under consideration could have detected. Clear boundaries for defect counting, such as temporal limits (e.g., defects found within a defined time frame after release) and exclusion criteria (e.g., defects in third-party components or specific customer environments), should be established to ensure consistency. A low DDP for a given test level indicates a high percentage of escaped defects, meaning defect detection is ineffective. In such a case, the TA should analyze the reasons and propose measures to improve it, making the test level more focused and rigorous. The DDP is best divided by severity levels, as the priority of reducing escaped defects usually depends on their severity.

Structural coverage analysis assesses the extent to which tests have exercised specific areas of the test object. Identifying low-coverage areas allows the TA to target test efforts in those areas. Increasing their coverage helps to discover new, previously escaped defects. Structural coverage such as statement coverage, branch coverage, or neuron coverage is usually measured with test tools. When selecting the additional areas to be covered, their risk levels should be considered.

Test gap analysis assesses the extent to which tests have exercised recent code changes. This allows the TA to focus additional test effort in areas that are especially error prone (i.e., new changes that have not been tested at all) instead of targeting all areas that have low coverage (e.g., code that has not changed in a long time and has been tested for previous releases).

Defect arrival pattern analysis . The number or density of defects found in successive phases of a project (e.g., iterations) can be compared with patterns that describe the theoretical distribution of these values over time. A classic example of a defect arrival pattern is the Rayleigh model (Elsayed, 2021). It has a single peak and is skewed to the right, showing that the expected number of defects found first increases in time and, after reaching its maximum value, drops down slowly towards zero. Based on the analysis of such a pattern, it is possible to infer the strength of existing test cases and the potential for their improvement. For example, suppose defect detection remains at a constant, low level when the pattern suggests it should be increasing. In that case, it may mean that the existing tests are too weak and unable to detect additional defects.

The analytical methods described above use various defect-related metrics, such as the number of defects or DDP. These metrics are calculated based on the test results (e.g., number of tests passed/failed), defect reports, and structural metrics (e.g., code coverage or defect density). Note, however, that these metrics are not always as easy to read from the test results as they may seem. For example, the number of failed tests is not necessarily the same as the number of defects detected by these tests. To calculate the actual number of defects detected, the results of the debugging process must be carefully analyzed because the relation between test results and defects can be many-to-many. Several tests may detect the same defect or one test may detect several defects. Moreover, the severity of the defects may differ from the criticality of the tests, as a critical test case may fail due to a cosmetic defect.

Scenario-based detection is easy to bypass because only known attacks are detected. Tests could include the following evasion techniques:

• Character encoding or modification of data (e.g., adding white space, end of lines, etc.)
• IP fragmentation, TCP segmentation
• Encryption, obfuscation
• URL encoding

Behavior-based detection generates a large number of false positive results and false negative results. A false negative result is any alert that should have reported but wasn’t. False negatives can occur when a new attack is developed that an IDS is not aware of, or perhaps a rule might be written in such a way as to detect some attacks but miss others. Also, the accuracy of this detection method should be considered. It is possible for an attacker to deviate the IDS behavior from its normal behavior resulting in a new specification containing intrusive behavior. Thus, this new traffic is not considered as anomalous. Complementary tests should use malicious traffic in order to add new intrusive specifications considered as authorized traffic.

Some inputs can be used to define a set of tests for IDS, such as “Profile Protection Intrusion Detection System” [PP-IDS] and “Web Application Firewall Evaluation Criteria” [WAFEC].

When a test script fails or passes unexpectedly, root cause analysis must be performed. This will include inspecting test logs, performance data, setup, and teardown of the test script.

It is also helpful to execute a few isolated tests. Intermittent failures are more difficult to analyze. The defect can be in the test case, the SUT, the TAF, the hardware or the network. Monitoring system resources may yield clues for the root cause. Test log file analysis of the test case, the SUT and the TAF can help identify the root cause of the defect. Debugging may also be necessary. To aid in identifying the root cause may require support from a test analyst, business analyst, developer, or system engineer.

Verify if all the assertions are in place. Missing assertions may result in inconclusive test results.

After test execution it is important to analyze the test results, to identify possible failure(s) both in the SUT and the TAS. For such an analysis the data collected from the TAS is primary and the data collected from the SUT is secondary.

• Analyze the test environment data to support proper sizing of test automation (e.g., in the cloud)
  • Clusters and resources (e.g., CPU, and RAM)
  • Single versus multi browser (i.e., cross-browser) test execution
• Compare test results of previous test executions
• Determine how to use web logs to monitor software usage

Test execution failures need to be analyzed, as there are potential issues:

1. Check if the same failure happened in the previous test executions. This might be a known defect either in the SUT or the TAS. The TAS can be built to log historical test results of the test cases, thus further helping the analysis.
2. If the defect is not known, identify the test case and what it is testing. The test can be selfexplanatory, or the test case can be identified in the test management system, based on its ID logged with the test execution.
3. Find in which test step of the test case the failure happened. The TAS logs this.
4. Analyze the test log information about the state of the SUT and whether it matches the expected results using screenshots, API and network logs, or any log that shows the state of the SUT.
5. If the state of the SUT is not what was expected, log a defect in the defect management system. Make sure to include all the necessary defect information and the logs that justify that it is a defect.

When there is a failure, it is possible for the actual result and the expected result of the SUT to match. In this case, most likely the TAS contains a defect which needs to be fixed, or an invisible mismatch is present.

Another situation that can occur is if the test environment is not available during the test run, or only partially available. In this case, all test cases can fail, either with the same defect or if parts of the system are down, with seemingly real failures. To identify the root cause of such defects, the SUT logs can be analyzed, which will show if there were any test environment outages at the time of the test run.

If the SUT implements audit logs for user interactions (i.e., UI sessions or API calls) it helps to analyze test results. There is usually a unique ID added to the interaction with the same ID for each subsequent call and integration in the system. In this way, knowing the unique ID of a request/interaction, the behavior of the system can be observed and traced back.

This unique ID is usually called a correlation ID or trace ID. This can be logged by the TAS to help analyze test results.

Each phase of the SDLC should include activities to detect and remove potential defects. For example, static testing techniques (i.e. reviews and static analysis) can be used on design specifications, requirements specifications and code prior to delivering those work products into subsequent activities. The earlier each defect is detected and removed, the lower the overall cost of quality for the product. Cost of quality is minimized when each defect is removed within the same phase in which it was introduced (i.e., when the software process achieves perfect phase containment).

During static testing we search for defects. During dynamic testing the presence of a defect is revealed when it causes a failure, which results in a discrepancy between the actual results and the expected results of a test (i.e., an anomaly). In some cases, a false-negative result occurs when the tester does not observe the anomaly. When an anomaly is observed, further investigation should be performed. This investigation usually starts with completing a defect report in accordance with the defined test and defect management process. A failed test does not always result in the creation of a defect report. (For example, in test-driven development, where component tests, usually automated, are used as a form of executable design specification). Until the development of the component is complete, some or all of the tests must initially fail. Until the development of the component is complete, some or all of the tests must initially fail. Therefore, the result of such a test is not necessarily caused by a defect and is typically not tracked via a defect report.

A defect report progresses along a workflow (for simplicity and consistency with most defect management tools we will further use the term “defect workflow”) and moves through a sequence of defect states. In most of these states, one person owns the defect report and is responsible for carrying out a task (e.g., analysis, defect removal, or confirmation test). The following diagram represents a simple defect workflow:

Figure 2: A Simple Defect Workflow

A simple defect workflow may cover the following defect states:

• OPEN (may be called NEW): The initial state when the defect report is created.
• IN PROGRESS: The team is working on the defect report analysis and/or fix.
• REJECTED: A defect report is rejected by the person who processed it (usually a developer or an analyst). There may be many reasons for rejection (e.g., invalid information, incorrect test, duplicate defect report) and this information is added to the defect report.
• RESOLVED (may be called FIXED, READY FOR RETEST): A tester runs a confirmation test often following the steps to reproduce the failure from the defect report itself to determine whether the fix has indeed resolved the defect.
• CLOSED: The defect report has reached its terminal state, and no further work is intended to be done. The tester transitions the defect report to this state either after a successful confirmation test or to acknowledge rejection of the defect report.

A simple defect workflow is used in many organizations and is extended by the use of other defect states relevant for a given context (e.g., RE-OPENED, ACCEPTED, CLARIFICATION, or DEFERRED).

The defect workflow may vary in different organizations in terms of different names of the defect states, rules for transitions among defect states and roles responsible for tasks in given defect states. Often the defect workflow is more simple in Agile software development than in sequential development models. The defect workflow should be adapted to a given context. When designing the defect workflow, it is advisable to respect several good practices:

• If possible, the defect workflow should be defined organization-wide to provide unified defect management across all projects
• Duplicate and false-positive defects should be represented by a separate state or a combination of the REJECTED status with choosing the reason for rejection. They may be helpful in further defect analyses with the aim of improving the test process.

• It is recommended to use only one terminal state (e.g., CLOSED). The transition to this state often requires choosing a reason for closure, useful for process assessment and process improvement activities.
• The names of states in the defect workflow should be the same as for analogous states used for other entities (e.g., user stories and test tasks) to simplify working with them.
• Consecutive defect states should belong to different responsible roles. If two or more consecutive states belong to the same responsible role, there should be a good reason (e.g., to measure the time spent in a defect state).
• Each defect state, except for the terminal state, should have more than one outgoing transition to allow the responsible role a decision regarding the next step. Exceptions of this rule should be justified (e.g., to monitor time spent on a given activity).
• The set of attributes required to be entered when performing a state transition should be limited to those which give substantial value to defect management.