Once the problem areas have been identified by the audit, risk must be evaluated and an improvement plan put in place. The auditor’s report may include recommendations as well as other areas of risk. From this point, risk identification, assessment and mitigation activities can be planned.
Risk identification is the process of documenting a risk or an area of risk. In the context of IT security, the risks are security-related. Risk assessment is the activity that assigns a value to the identified risks. It is important to understand that traditional IT risk assessment models are not sufficient to address IT security risks. Any security risk assessment model or approach should be specifically oriented to IT security risk profiles.
Security risks often are measured in terms of risk exposure. Risk exposure is computed by multiplying the potential impact or loss by the likelihood of that loss occurring. For example, if one customer’s account information is compromised, what would be the impact? What if that customer had $100 million of assets on deposit?
The likelihood of occurrence can be determined by applying a security risk assessment model such as that found in NIST Publication 800-30, Guide for Conducting Risk Assessments [NIST 800-30]. Another excellent guide to performing security risk assessments is the OWASP Risk Rating Methodology [OWASP2]. The following information is abstracted from [NIST 800-30].
Risk models define the risk factors to be assessed and the relationships among those factors. Risk factors are characteristics used in risk models as inputs to determining levels of risk in risk assessments. Risk factors are also used extensively in risk communications to highlight what strongly affects the levels of risk in particular situations, circumstances, or contexts.
Typical risk factors include threat, vulnerability, impact, likelihood, and predisposing condition. Risk factors can be decomposed into more detailed characteristics (e.g., threats decomposed into threat sources and threat events). These definitions are important for organizations to document prior to conducting risk assessments because the assessments rely upon well-defined attributes of threats, vulnerabilities, impact, and other risk factors to effectively determine risk.
Threats
A threat is any circumstance or event with the potential to adversely impact organizational operations and assets, individuals, other organizations, or a country through an information system via unauthorized access, destruction, disclosure, or modification of information, and/or denial of service.
Threat events are caused by threat sources. A threat source is characterized as:
In general, types of threat sources include:
Various taxonomies of threat sources have been developed. Some taxonomies of threat sources use the type of adverse impacts as an organizing principle. Multiple threat sources can initiate or cause the same threat event—for example, a provisioning server can be taken offline by a denial-of-service attack, a deliberate act by a malicious system administrator, an administrative error, a hardware fault, or a power failure.
Vulnerabilities and Predisposing Conditions
A vulnerability is a weakness in an information system, system security procedures, internal controls, or implementation that could be exploited by a threat source.
Most information system vulnerabilities can be associated with security controls that either have not been applied (either intentionally or unintentionally), or have been applied but retain some weakness. However, it is also important to allow for the possibility of emergent vulnerabilities that can arise naturally over time as organizational missions/business functions evolve, environments of operation change, new technologies proliferate, and new threats emerge. In the context of such changes, existing security controls may become inadequate and may need to be reassessed for effectiveness. The tendency for security controls to potentially degrade in effectiveness over time reinforces the need to maintain risk assessments during the entire software lifecycle and also the importance of continuous monitoring programs to obtain ongoing situational awareness of the organizational security posture.
Vulnerabilities are not identified only within information systems. Viewing information systems in a broader context, vulnerabilities can be found in organizational governance structures (e.g., the lack of effective risk management strategies and adequate risk framing, poor intra-agency communications, inconsistent decisions about relative priorities of missions/business functions, or misalignment of enterprise architecture to support mission/business activities).
Vulnerabilities can also be found in external relationships (e.g., dependencies on particular energy sources, supply chains, information technologies, and telecommunications providers), mission/business processes (e.g., poorly defined processes or processes that are not risk-aware), and enterprise/information security architectures (e.g., poor architectural decisions resulting in lack of diversity or resiliency in organizational information systems).
Impact
The level of impact from a threat event is the magnitude of harm that can be expected to result from the consequences of unauthorized disclosure of information, unauthorized modification of information, unauthorized destruction of information, or loss of information or information system availability. Such harm can be experienced by a variety of organizational and non-organizational stakeholders including:
The following information should be explicitly documented by an organization:
Organizations may explicitly define how established priorities and values guide the identification of highvalue assets and the potential adverse impacts to organizational stakeholders. If such information is not defined, priorities and values related to identifying targets of threat sources and associated organizational impacts can typically be derived from strategic planning and policies. For example, security categorization levels indicate the organizational impacts of compromising different types of information.
Likelihood
The likelihood of occurrence addresses the probability (or possibility) that the threat event will result in an adverse impact, regardless of the magnitude of harm that can be expected. This is a weighted risk factor based on an analysis of the probability that a given threat is capable of exploiting a given vulnerability (or set of vulnerabilities). The likelihood risk factor combines an estimate of the likelihood that the threat event will be initiated with an estimate of the likelihood of impact (i.e., the likelihood that the threat event will result in adverse impacts).
For adversarial threats, an assessment of likelihood of occurrence is typically based on:
For other than adversarial threat events, the likelihood of occurrence is estimated using historical evidence, empirical data, or other factors. Note that the likelihood that a threat event will be initiated or will occur is assessed with respect to a specific time frame (e.g., the next six months, the next year, or the period until a specified milestone is reached).
If a threat event is almost certain to be initiated or occur in the (specified or implicit) time frame, the risk assessment may take into consideration the estimated frequency of the event. The likelihood of threat occurrence can also be based on the state of the organization (including, for example, its core mission/business processes, enterprise architecture, information security architecture, information systems, and environments in which those systems operate. Predisposing conditions and the presence and effectiveness of deployed security controls to protect against unauthorized/undesirable behavior, detect and limit damage, and/or maintain or restore mission/business capabilities should also be taken into consideration.
Determining the Level of Security Risk
The likelihood of occurrence assessment and the impact assessment can be combined to calculate an overall severity for the risk. Specific assessment scores may be used as the basis of the completing the risk matrix. In other cases, estimates (low, medium or high) may be used.
The scoring for the risk matrix can be based on a scale of 0 – 9, where the numeric values are determined by specific criteria. For example, risk likelihood criteria could be evaluated for data privacy as:
0 - <3 (Low) Private data is not stored on local devices and is encrypted when stored on secure media. 3 - <6 (Medium) Private data may reside on devices such as notebook computers, but is encrypted. 6 - 9 (High) It is not known exactly if private data resides on local devices. Encryption cannot be assured.
Likewise, risk impact criteria could be assessed on the same 0 – 9 scale based on specific criteria. For example:
However the tester arrives at the likelihood and impact estimates, the estimates can be combined into a final severity rating for the risk item. If there is good business impact information, that should be used instead of the technical impact information. If there is no information about the business, then technical impact is the next best thing.
Below is a sample view of a risk matrix that can be used to determine the severity of individual risks.
| Overall Risk Severity | ||||
| Risk Impact | High | Medium | High | Critical |
| Medium | Medium | Medium | High | |
| Low | Low | Low | Medium | |
| Low | Medium | High | ||
| Risk Likelihood | ||||
In the example matrix above, if the likelihood is medium and the impact is high, the overall severity is high.
In addition, the risk assessment report should identify if the risk is ongoing. Continual risks indicate an increased likelihood that a loss will occur.
The severity of a risk determines the relative importance of mitigating the risk. The higher the risk severity, the more immediate the requirement for the response. The level of detail provided in any particular risk assessment is consistent with the purpose of the risk assessment and the type of inputs needed to support follow-on likelihood and impact determinations.
The recommendations from the ”establishing” phase are prioritized according to a list of criteria, each of which may be weighted according to the need for the improvement and the stakeholders involved.
At a minimum the following criteria should be considered:
In software development, testing is the most important quality risk mitigation activity, and makes it possible to reduce the likelihood of failures. Other possible risk mitigation measures include a contingency plan (e.g., by providing workarounds), risk transfer to a third party (e.g., the vendor of a component), or risk acceptance.
In test planning , the time and effort associated with developing and executing a test should be proportional to the risk level: Testing for higher risk levels should start early and use more rigorous test techniques, while testing for lower risk levels may start later and should use less rigorous test techniques. To best mitigate the overall risk through testing, the test manager should analyze the following contextual factors and select an appropriate test approach:
In addition, the risk level should influence quality control decisions such as the use of reviews of work products like test cases, the level of independence of testing from development, and the extent of regression testing performed.
During test monitoring and test control , risk-based testing allows reporting on the test progress in terms of the residual risk level at any point in time. This supports the development team and stakeholders in monitoring and controlling software development, including making release decisions, based on the residual risk level. This requires reporting test results in terms of risks in a way stakeholders can understand.
During test implementation , test prioritization is based on the risk levels. During test execution, this ensures early coverage of the most critical areas and mitigation of the highest level risks.
Whether risk-based testing proceeds depth-first, breadth-first, or combined, the time allocated for testing may be consumed without all planned tests being run. At this point, risk-based testing facilitates the provision of a justified recommendation to management whether to extend testing or to accept the remaining risk.
The most established standard in IT security is the series of ISO 27000. The [ISO 27001] standard is internationally accepted and entitled “Information technology — Security techniques — Information security management systems — Overview and vocabulary”. The focus of this standard is on information security management, i.e., to identify risks, to evaluate them and to manage them through information security controls. All these activities are combined in an information security management system (ISMS) which is the overall core of the ISO 27000 standard. The standard is broad in scope and focuses on the general way in which an organization should assess their risks, contrast them with their specific needs and deal with the most relevant risks. The core standard can be applied to all organizations.
The ISO 27000 series consists of more than 40 individual standards, which can be classified into the following:
The most used standards that apply to the context of security testing and cover the relevant test objects and test conditions which the STE should consider, are the following:
The most important part of ISO 27001 for an STE is presented in this appendix. It lists security controls for different aspects such as access control, disaster recovery and network security. Each of these controls, if applied in a specific context, are important inputs for an STE, as it is their task to measure the effectiveness of a security control. [Cald11]
De facto Standards for Security Testing
There are many de facto standards which can be leveraged by an STE. One of the most important series of de facto standards stems from the MITRE corporation, even though its main business is not to generate standards. MITRE is a private, not-for-profit organization which provides engineering and technical guidance for the federal US government. The most important sponsors of MITRE are the Department of Defense, the Federal Aviation Administration and the Department of Homeland Security [MITRE21].
For the area of security testing, MITRE hosts and maintains the following well-known standards that provide added value for the STE:
Common Attack Pattern Enumeration and Classification (CAPEC™)
CAPEC™ provides a publicly available catalogue of common attack patterns. The idea is to get a better understanding of how attackers exploit weaknesses in applications and other cyber-enabled capabilities. Attack patterns are based on software design patterns for attackers. Two typical entry attack patterns are SQL injection (CAPEC-66) and relative path traversal (CAPEC-139) [CAPEC21].
CAPEC provides different views on its data sets. The most relevant ones are:
Each test object that an STE tests should be located within this catalogue. Often CAPEC is the starting point to get an initial overview of possible attacks that might be relevant for a given system.
The following MITRE standards are used for further refinement for effective security testing:
Common Weakness Enumeration (CWE)
CWE is a list of software/hardware weaknesses. Usually, each common attack pattern has one or more weaknesses that are usable for leveraging a CAPAC attack pattern [CWE21]. CWE uses the concept of views, the most used of which are:
Each common weakness is an effective starting point for an STE to test whether the underlying attack pattern can be exploited.
Open Web Application Security Project (OWASP)
It is important to realize that CWE and OWASP [OWASP21] overlap and they both list common weaknesses. OWASP is well known for publishing its OWASP Top 10 ranking.
Common Weakness Scoring System (CWSS)
The more common a weakness becomes, the more important it gets to have a prioritization scheme in place. CWSS provides a mechanism for prioritizing weaknesses in a consistent, flexible, open manner [CWSS21]. Prioritization is calculated by three sets of metrics:
By using specific, predefined weights, all of these metrics can be aggregated into one overall CWSS value for one specific weakness. CWSS can handle unknown metrics by default values or by defining/focusing on an individual metric subset. In addition, many metrics of the Base Finding Metric Group can automatically be calculated by a static analysis tool.
Common Vulnerability Scoring System (CVSS)
A similar prioritization mechanism to CWSS is CVSS [CVSS21], which follows a similar approach, but assumes an existing, deployed vulnerability (see CVE below). Both, CVSS and CWSS are scoring systems for computer security: CVSS is a reactive approach because vulnerabilities already exist before ranking. CWSS is a proactive approach, as you are working with software before releasing it into production. Both approaches are often used together, even if they are not fully compatible (cf. [SecJour21]).
Common Vulnerabilities and Exposures (CVE)
CVE is a database of publicly disclosed information about security issues [CVE21]. A CVE number uniquely identifies a particular vulnerability from the list. CVE helps because it provides a standardized identifier for a given vulnerability within a specific system. If a system is affected by a specific CVE, this vulnerability is a specific instance of a common weakness (CWE) that can be used to do a specific attack (CAPEC). New entries within the CVE repository usually originate from the daily work of STEs. If they identify a new vulnerability unknown to CVE, they can publish it at CVE to engage the security community to identify counter measures.
Best Practices for Security Testing
Best practices only need to achieve a low formal criterion to be considered as best practices. Many best practices may fail after some time if they don’t help. Some will stop being used because of missing publication/marketing, but a few might improve their maturity on their way to being considered for a standard.
One typical mature best practice that is still used today is the STRIDE model which was invented by Microsoft [Micro09]. STRIDE allows for systematic threat modeling from an attacker perspective. The term itself is an acronym for six threat categories, which classifies potential threats: s poofing, t ampering, r epudiation, i nformation disclosure, d enial of service and e levation of privilege.
Generally, STRIDE is used to support developers to consider threats during design and to close identified gaps. The STE can use the same approach to focus on testing.
A usability finding is a result from a usability evaluation that identifies some important issue, problem, or opportunity.
Positive usability findings are important for the following reasons:
A usability test report should contain a section that describes the most important findings from the usability test and associated recommendations for improvement of the software product.
The description of each finding should include the following items:
Classification and Severity Rating of Findings
Severity classifications and ratings are assigned to usability problem to indicate the type of the finding, its impact and criticality on the user experience, and the consequences.
The moderator and the note-taker rate usability problems from the test participants' point of view. Sometimes, the severity ratings are allocated in cooperation with a domain expert.
Typical classifications are:
| Classification | Description |
|---|---|
| Usability problem | Each usability problem must have a severity rating as described in the following note. |
| Positive finding | Works well. This approach can be recommended. |
|
Good idea |
A suggestion from a test participant that could lead to a significant improvement of the user experience. |
|
Functional problem |
Defect |
Typical severity ratings of usability problems are:
| Severityratings | Description |
|---|---|
|
Minor |
Minor dissatisfaction, noticeable delays, or superficial difficulties. |
|
Major |
Substantial delays; or moderate dissatisfaction |
|
Critical |
Test participants gave up. Showstopper, substantial dissatisfaction or minor financial damage to user. |
|
Catastrophic |
Existential threat. Potentially life-threatening, bodily harm or substantial financial damage. |
Important parameters that influence severity ratings are: