NIST SP 800-55v1: Measurement Guide for Information Security: Volume 1 — Identifying and Selecting Measures

NIST Special Publication 800 NIST SP 800-55v1

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures Katherine Schroeder Hung Trinh Victoria Yan Pillitteri

This publication is available free of charge from: https://doi.org/10.6028/NIST.SP.800-55v1

NIST Special Publication 800 NIST SP 800-55v1

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures Katherine Schroeder Hung Trinh Victoria Yan Pillitteri

Computer Security Division Information Technology Laboratory

This publication is available free of charge from: https://doi.org/10.6028/NIST.SP.800-55v1

December 2024

U.S. Department of Commerce Gina M. Raimondo, Secretary National Institute of Standards and Technology Laurie E. Locascio, NIST Director and Under Secretary of Commerce for Standards and Technology

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Certain equipment, instruments, software, or materials, commercial or non-commercial, are identified in this paper in order to specify the experimental procedure adequately. Such identification does not imply recommendation or endorsement of any product or service by NIST, nor does it imply that the materials or equipment identified are necessarily the best available for the purpose. There may be references in this publication to other publications currently under development by NIST in accordance with its assigned statutory responsibilities. The information in this publication, including concepts and methodologies, may be used by federal agencies even before the completion of such companion publications. Thus, until each publication is completed, current requirements, guidelines, and procedures, where they exist, remain operative. For planning and transition purposes, federal agencies may wish to closely follow the development of these new publications by NIST. Organizations are encouraged to review all draft publications during public comment periods and provide feedback to NIST. Many NIST cybersecurity publications, other than the ones noted above, are available at https://csrc.nist.gov/publications. Authority This publication has been developed by NIST in accordance with its statutory responsibilities under the Federal Information Security Modernization Act (FISMA) of 2014, 44 U.S.C. § 3551 et seq., Public Law (P.L.) 113-283. NIST is responsible for developing information security standards and guidelines, including minimum requirements for federal information systems, but such standards and guidelines shall not apply to national security systems without the express approval of appropriate federal officials exercising policy authority over such systems. This guideline is consistent with the requirements of the Office of Management and Budget (OMB) Circular A-130. Nothing in this publication should be taken to contradict the standards and guidelines made mandatory and binding on federal agencies by the Secretary of Commerce under statutory authority. Nor should these guidelines be interpreted as altering or superseding the existing authorities of the Secretary of Commerce, Director of the OMB, or any other federal official. This publication may be used by nongovernmental organizations on a voluntary basis and is not subject to copyright in the United States. Attribution would, however, be appreciated by NIST. NIST Technical Series Policies Copyright, Use, and Licensing Statements NIST Technical Series Publication Identifier Syntax Publication History Approved by the NIST Editorial Review Board on 2024-11-04 Supersedes NIST SP 800-55 Rev. 1 (July 2008) https://doi.org/10.6028/NIST.SP.800-55r1 How to Cite this NIST Technical Series Publication: Schroeder K, Trinh H, Pillitteri VY (2024) Measurement Guide for Information Security: Volume 1 — Identifying and Selecting Measures. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-55v1. https://doi.org/10.6028/NIST.SP.800-55v1 Author ORCID iDs Katherine Schroeder: 0000-0002-4129-9243 Hung Trinh: 0000-0002-3323-0836 Victoria Yan Pillitteri: 0000-0002-7446-7506

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Contact Information cyber-measures@list.nist.gov National Institute of Standards and Technology Attn: Computer Security Division, Information Technology Laboratory 100 Bureau Drive (Mail Stop 8930) Gaithersburg, MD 20899-8930 Additional Information Additional information about this publication is available at https://csrc.nist.gov/pubs/sp/800/55/v1/final, including related content, potential updates, and document history. All comments are subject to release under the Freedom of Information Act (FOIA).

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Abstract This document provides guidance on how an organization can develop information security measures to identify the adequacy of in-place security policies, procedures, and controls. It explains the measures prioritization process and how to evaluate measures. Keywords assessment; information security; measurement; measures; metrics; performance; qualitative; quantitative; reports; security controls. Reports on Computer Systems Technology The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the Nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analyses to advance the development and productive use of information technology. ITL’s responsibilities include the development of management, administrative, technical, and physical standards and guidelines for the cost-effective security and privacy of other than national security-related information in federal information systems. The Special Publication 800-series reports on ITL’s research, guidelines, and outreach efforts in information system security, and its collaborative activities with industry, government, and academic organizations. Audience This guide is written primarily for users with responsibilities or interest in information security measurement and assessment. Government and industry can use the concepts, processes, and candidate measures presented in this guide.

i

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Patent Disclosure Notice NOTICE: ITL has requested that holders of patent claims whose use may be required for compliance with the guidance or requirements of this publication disclose such patent claims to ITL. However, holders of patents are not obligated to respond to ITL calls for patents and ITL has not undertaken a patent search in order to identify which, if any, patents may apply to this publication. As of the date of publication and following call(s) for the identification of patent claims whose use may be required for compliance with the guidance or requirements of this publication, no such patent claims have been identified to ITL. No representation is made or implied by ITL that licenses are not required to avoid patent infringement in the use of this publication.

ii

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Table of Contents 1. Introduction ...................................................................................................................................1

2. Fundamentals ................................................................................................................................4

3. Measurement Considerations .......................................................................................................12 3.1.1. Measures Documentation .......................................................................................................... 12 3.1.2. Measurement Reporting ............................................................................................................ 14 3.1.3. Data Management...................................................................................................................... 14 3.1.4. Data Quality ................................................................................................................................ 15 3.1.5. Uncertainty and Errors ............................................................................................................... 15

3.3.1. Implementation Measures ......................................................................................................... 17 3.3.2. Effectiveness Measures .............................................................................................................. 17 3.3.3. Efficiency Measures .................................................................................................................... 18 3.3.4. Impact Measures ........................................................................................................................ 18 4. Selecting and Prioritizing Measures ..............................................................................................19

4.2.1. Comparing Measures and Assessment Results .......................................................................... 20 4.3.1. Likelihood and Impact Modeling ................................................................................................ 21 4.3.2. Weighing Scale ........................................................................................................................... 22 References.......................................................................................................................................24 Appendix A. Glossary .......................................................................................................................26 Appendix B. Data Analysis Dictionary ...............................................................................................29 iii

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Appendix C. Modeling Impact and Likelihood ...................................................................................32

Appendix D. Change Log ...................................................................................................................34

List of Tables Table 1. Stevens Scale of Measurement ..............................................................................................3 Table 2. Data analysis examples .........................................................................................................6 Table 3. Data cleaning methods for reducing uncertainty ..................................................................16 Table 4. Examples of measures and types of qualitative and semi-quantitative assessment results .... 20

List of Figures Fig. 1. Notional process for the definition, collection, and analysis of metrics ....................................10

iv

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

1. Introduction Information security measurement enables organizations to describe and quantify information security, allocate finite resources, and make informed and data-driven decisions for improved outcomes. However, organizations first need to know what policies, procedures, and controls they have in place at any given time; whether those policies and procedures are having the desired results; and how the organization and its risks are impacted. By developing and monitoring measurements that evaluate what an organization has in place for information security risk management and how well those efforts are working, an organization can better address their goals and direct resources. 1.1. Purpose and Scope NIST Special Publication (SP) 800-55v1 (Volume 1) is a flexible guide for developing and selecting information security measures at the organization, mission/business, and system levels to identify the success of in-place policies, procedures, and controls. 1 This document expands on previous NIST work on information security measures and measurements by focusing on quantitative assessments2 and addressing organizational and program maturity. The SP 800-55v2 [23] provides a methodology for implementing an information security measurement program. Additionally, while many of the principles of information security measurement may apply to privacy, privacy is out of scope for this document. 1.2. Relationship to Other NIST Publications This document is intended to provide considerations for measuring the information security program activities described in other NIST publications, including: •

SP 800-137A, Assessing Information Security Continuous Monitoring Programs [14]

•

The NIST Cybersecurity Framework (CSF) 2.0 [1]

•

SP 800-30r1 (Revision 1), Guide for Conducting Risk Assessments [9]

•

SP 800-37r2, Risk Management Framework for Information Security Systems and Organizations: A System Life Cycle Approach for Security and Privacy [10]

•

SP 800-161r1, Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations [17]

•

NIST Engineering Statistics Handbook [18]

•

NIST Internal Report (IR) 8286, Identifying and Estimating Cybersecurity Risk for Enterprise Risk Management (ERM) [4]

This document uses the term controls to broadly describe identified countermeasures for managing information security risks. It is intended to be framework- and standard-agnostic and can also apply to other existing models or frameworks. 2 SP 800-55 uses the terms quantitative assessment and measurement synonymously. Refer to Sec. 1.4 , Document Terminology, for additional information. 1

1

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

1.3. Document Organization The remaining sections of this document discuss the following: •

Section 2, Fundamentals

•

Section 3, Measurement Considerations

•

Section 4, Selecting and Prioritizing Measures

•

Appendix A, Glossary

•

Appendix B, Data Analysis Dictionary

•

Appendix C, Modeling Impact and Likelihood

•

Appendix D, Change Log

1.4. Document Terminology In the context of this document, the following terms are defined as follows: •

Assessment: The action of evaluating, estimating, or judging against defined criteria. Different types of assessment (i.e., qualitative, quantitative, and semi-quantitative) are used to assess risk. Some types of assessment yield measures.

•

Assessment result: The output or outcome of an assessment.

•

Information security3: The protection of information and systems from unauthorized access, use, disclosure, disruption, modification, or destruction to provide confidentiality, integrity, and availability. [2]

•

Measurement: The process of obtaining quantitative values using quantitative methods.

•

Measures: Quantifiable and objective values that result from measurement.

•

Metrics: Measures and assessment results designed to track progress, facilitate decision-making, and improve performance with respect to a set target.

•

Qualitative assessment: The use of a set of methods, principles, or rules for assessing risk based on nonnumerical categories or levels. [9]

•

Quantitative assessment: The use of a set of methods, principles, or rules for assessing risks based on the use of numbers where the meanings and proportionality of values are maintained inside and outside of the context of the assessment. [9]

•

Semi-quantitative assessment: The use of a set of methods, principles, or rules for assessing risk based on bins, scales, or representative numbers whose values and meanings are not maintained in other contexts. [9]

3 The term “cybersecurity” can be used interchangeably with “information security.”

2

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

This document discusses concepts that are similar to the Stevens Scale of Measurement, as shown in Table 1, but takes a different view on what is and is not a measurement. For the purposes of this document, a nominal scale is considered a form of data gathering, and an ordinal scale is considered a ranking system. Both interval and ratio scales use variables that represent true numbers and can be used in a quantitative assessment, so they are considered measurement [19].

Table 1. Stevens Scale of Measurement Definition

Example

Nominal

A nominal scale only looks at classification or identification. Nominal scales are used in surveys and in dealings with either nonnumeric variables or numbers that do not have an assigned value. The data collected from a nominal scale can be used for counting, mode, or correlation contingency matrices.

Examples include demographic information, such as what county someone lives in, blood type, and marital status.

Ordinal

An ordinal scale is similar to a nominal scale in that it primarily uses non-numeric values or numbers that are meant to show ranking. Related statistics include medians and percentiles.

Examples of ordinal measurements include income level, Likert scales (strongly disagree to strongly agree), and rankings.

Interval

An interval scale is used when measuring variables with equal intervals between values. When using an interval scale, there is no true zero. Interval statistics include mean, standard deviation, and rank-order correlation.

Examples of the use of interval scales are temperature or time scales. Interval data allows for quantitative analysis, such as descriptive statistics like frequency, averages, position, and dispersion.

Ratio

Ratio scales allow for the categorization and ranking of data, similar to an interval scale, but with a true zero and no negative values. Ratio scales allow for numbers to be used for addition, subtraction, multiplication, and division.

Examples of ratio measurements deal with true zeros, such as ruler measurements, age, money, and number of occurrences.

Scale Level

3

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

2. Fundamentals The terms assessment and measurement are often used interchangeably in the information security field, as both aid in risk management and security posture analysis. This document provides a lexicon for key terminology and an overview of foundational concepts for measuring and assessing information security risk and clarifies the distinction between assessment and measurement. As described in Sec. 1.4, assessment refers to the process of evaluating, estimating, or judging against defined criteria, and measurement is the process of obtaining quantitative values. Hence, assessment is a broader concept that also includes measurement. Organizations perform multiple kinds of assessment when evaluating information security risk, such as risk assessments, program assessments, and control assessments. Risk assessments are used to identify the risks that an organization faces and can support decision-making [9]. Program-level assessments are used for decision-making about the strategies, policies, procedures, and operations that determine the security posture of an information security program. In control assessments, organizations evaluate whether specific controls are performing the way they were intended and achieving the desired results. Both program assessments and control assessments are in and of themselves a form of risk assessment and provide a different lens for viewing information security risk. SP 800-55 is intentionally agnostic on specific risk assessment models. Many of these models may help identify areas of threat, likelihood, vulnerability, and impact that require further assessment. 4 2.1. Measurement and Quantitative Assessment Measures are numerically expressed data that are gathered through the process of measurement.5 Measures can be derived from any operations or systems that can be measured with numbers. Quantitative assessments judge measures data against a set criteria or target and can be used to analyze information security risks using frequency, rates, financial impacts, and other numeric indicators. Using quantitative assessments requires a knowledge of measurement techniques and data analysis processes. One challenge of measurement is using the right measures and quantity of measures to perform useful analysis. A single measure alone may not provide sufficient data to make risk-based decisions, but organizations may also have restraints on resources that prevent them from employing and analyzing every potential measure. An organization finds the number of measures and depth of analysis that work best for their needs. The ability to measure information security risks relies on data availability. Methods for collecting information security data may include experimentation, observation, or sampling. The NIST Engineering Statistics Handbook [18] offers detailed information on choosing a sampling scheme, including the following methods:

4 For additional information about risk assessment models, see [9].

5 As described in Sec. 1.4, measures and quantitative assessment results can be used synonymously, as can the terms measurement and

quantitative assessment.

4

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

•

Experimentation is a systematic approach to testing new ideas, methods, or activities that applies principles and techniques at the data collection stage to ensure the generation of valid, defensible, and supportable conclusions. A recognizable use of experimentation to collect information security data is a phishing test, which is a form of internal security testing where organizations send fake phishing emails to determine which users respond to it. The rates of success are then judged against set criteria.

•

Observational data refers to capturing data through the observation of an activity or behavior without the direct involvement of the subject. Observational data is often gathered as part of routine information security operations, such as log management tools that are used to collect and analyze network activities. Data from these logs is observational and can be used for further analysis.

•

Sampling is the process of taking samples of something for the purpose of analysis. Sampling may be used when continuous observation and passive data collection are not an option or when random, stratified, or systematic sampling may be preferred. Random sampling is a method of sampling in which each sample has an equal chance of selection in hopes of gathering an unbiased representation. Stratified sampling is the process of segmenting a population across levels of some factors to minimize variability within those segments (e.g., taking a sample from a terminal in each department of an organization). Stratified sampling may help an assessment target organizational units without being overwhelmed by noisy data collection but can provide biased results. Systematic sampling involves taking samples at a regular interval (e.g., once an hour or from every tenth user). Systematic sampling is useful for identifying macro-trends but may not provide enough conclusive measurements for the entire population if an underlying pattern is present.

Once the data from measurement is procured, the outputs of quantitative analysis can be used in a quantitative assessment to determine whether the organization is meeting its information security goals and support risk-based decision-making. Data analysis methods 6 are largely based on the type of questions that the organization is asking about their information security risks, program, and controls. The NIST Engineering Statistics Handbook [18] identifies three popular approaches to data analysis: 1. Classical — In the classical data analysis approach, data collection is directly followed by modeling, and the analysis, estimation, and testing that come after focus on the parameters of that model. Classical data analysis includes deterministic and probabilistic models, such as regression and the analysis of variance (ANOVA). 2. Exploratory — Exploratory data analysis begins by inferring what model would be appropriate before trying different analytic models. Identifying patterns in the data may give insight as to what models would produce the most useful information. Some common exploratory data analysis graphical techniques include standard deviation plots and histograms.

6 Appendix C provides additional examples of quantitative data analysis methods.

5

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

3. Bayesian — Bayesian/predictive methodology consists of formally combining both the prior distribution of the parameters and the collected data to jointly make inferences and/or test assumptions about the model of parameters. Bayesian methods can be used for expected range setting and predictive models. Table 2 shows examples of quantitative analysis across risk assessment, program-level assessment, and control-level assessment. Table 2. Data analysis examples Type of Assessment

Approach

Example

Risk Assessment

Classical (Value at Risk [VaR])

Risk Assessment

Bayesian/predictive

Program-Level Assessment

Classical (Mean)

Program-Level Assessment

Exploratory Data Analysis (Scatter Plot)

Program-Level Assessment

Bayesian/predictive

6

An organization conducting a risk assessment will likely consider their value at risk if they were to suffer an adverse information security event. The organization may look at potential losses from downtime, the cost of repairing the environment, or reputational damage. The Bayesian/predictive method looks at prior distribution, collected data, and set parameters to make inferences about future outcomes. Using data from SP 800-53 control RA-3(4), Predictive Cyber Analytics, as part of a risk assessment, the inferences found through the Bayesian/predictive method allow organizations to make risk-based decisions based on the likelihood of future events. At the program level, an organization may choose to identify the mean time it takes to complete an action. For example, using SP 800-53 control PM22, Personally Identifiable Information Quality Management, the mean time to correct or delete inaccurate or outdated personally identifiable information is measured. The organization may also consider the variance in that data from year to year or see whether certain individuals are addressing that personally identifiable information at different rates. An organization may want to use a scatter plot as part of a program-level assessment to reveal relationships or associations between two variables. Using data collected as part of SP 80053 control PM-31, Continuous Monitoring Strategy, one can examine linear relationships shown in a scatter plot of historical data. The scatter plot can reveal outliers or information about typical uses of a system. The Bayesian/predictive method can be used to influence programmatic decisions around continuous improvement. For example, using SP 800-53 control PM-6, Measures of Performance, and the Bayesian/predictive method on prior historical data, an organization can determine what future data may look like. This information

NIST SP 800-55v1 December 2024 Type of Assessment

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures Approach

Control Assessment

Classical (Linear Regression)

Control Assessment

Exploratory Data Analysis

Control Assessment

Bayesian/predictive

Example on future outcomes can be used to set the expected results of information security performance. At the control level, an organization may have implemented continuous monitoring (i.e., SP 80053, control CA-7) of a specific system-level metric. The data provided by the continuous monitoring of a system can be used in linear regression to learn what “normal” looks like for that system, which in turn allows the organization to identify deviations from that “normal.” This is a foundational piece of the information security measurement and assessment process. At the control level, a multi-factor/comparative box plot could be used to compare the key characteristics or unusual data in a data set monitoring a control. The Bayesian/predictive method may be used to make decisions about the frequency of equipment maintenance using SP 800-53 control MA-6(2), Timely Maintenance | Predictive Maintenance, and historical data about organizational equipment.

Organizations that are early in the process of assessing their information security risks, program, or systems may rely heavily on qualitative assessments that present nonnumerical information in place of measurement. These nonnumerical methods can help show context, examine labels, and look at behavior. A prominent example of qualitative assessment featured in many information security measurement programs is the risk matrix — a table that uses colored rating scales to show the impact and likelihood of various risks. As organizations gain the ability to record and track information security data, they can move toward the increased precision and reduced bias of quantitative assessments.

2.2. Types of Assessment There are three types of assessment: 1. Qualitative assessments are subjective and interpretive, using nonnumerical values or categories, such as high, medium, and low or heat maps. 2. Semi-quantitative assessments use numbers, but those numbers do not maintain their value outside of the assessment context. This is commonly seen in models that use number rankings to show a level of organizational integration. While the assessment

7

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

may say that the organization is at “level 3,” that “3” represents a set of qualities rather than a numerical value. 3. Quantitative assessments use data and statistics to obtain objective, precise results, and the numbers retain their values outside of the context. For example, 98 % of authorized accounts belong to current employees, and 2 % belong to former employees. Here, the values “98 %” and “2 %” stay the same regardless of the context. Since for the purpose of SP 800-55 measurement is the process of obtaining quantifiable values using quantifiable assessment methods, measures are quantitative assessment results. Quantitative assessments (i.e., measurements) can provide objective data that allows for tracking and shows changes. However, they can be difficult to produce in early stages of measurement since they require more data and resources than nonnumerical and categorified qualitative assessments. In contrast, simple nonnumerical and categorified qualitative assessments may be more commonly used and easier to conduct, but their results can also be subjective and require everyone to have an equal understanding of the scale used. It is important for organizations to consider their motivations for measuring information security risks before determining whether a quantitative or qualitative assessment is appropriate. For example, an organization motivated primarily by compliance with an industry certification or international standard has different measurement needs than an organization motivated by cost reduction. An organization could have multiple, competing motivations that drive the identification and selection of measures. Some organizational motivations may benefit from quantitative assessments, such as trying to determine whether the organization is patching known vulnerabilities in an acceptable amount of time. Knowing the mean time to remediate a vulnerability provides more precise insight into patching efficiency than simply knowing the number of vulnerabilities patched in a year. Because the question of mean time to remediate a vulnerability deals with attainable non-zero numbers, a measurement can be taken, and a mathematically derived answer can be given. When real and attainable numbers based on gathered data can be found and analyzed, a quantitative assessment may be the appropriate action. If there are proposed questions that do not have measurable numbers attached to them but still need to be addressed, a qualitative assessment may be the best option. Commonly used qualitative methods include color scales that represent risk levels or number scales that show rankings. For the purposes of this document, qualitative and semi-quantitative assessments are not considered measurements, and the values produced by these types of assessments are not considered measures. Most organizations will use a mixture of quantitative, semi-quantitative, and qualitative assessments. Ultimately, some or all of the assessment results will be used to determine success.

8

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

2.3. Benefits of Using Measures Developing and establishing measurements to capture and provide meaningful data at all levels of an organization requires careful consideration. Meaningful measures take organizational information security goals and objectives into account and are obtainable, repeatable, and feasible to measure. Information security measurement enables organizations to quantify improvements or gaps in securing systems and demonstrate quantifiable progress in accomplishing strategic goals and objectives (i.e., security posture). Well-designed measurements can provide information on the implementation, effectiveness, efficiency, and business impacts of controls, such as the results of information security activities, events (e.g., incident data, revenue lost to cyber attacks), and information security investments. Measurement provides data that can enable an organization to examine the impacts of implementing information security programs, specific controls, and associated policies and procedures. Such data is integral when making risk-based decisions, weighing performance against designated metrics, and demonstrating compliance. Measurement can also increase accountability and strengthen governance by providing data that can facilitating the identification of the personnel responsible for controls implemented within specific organizational components or systems and support an environment that allows for continuous analysis and improvement. 2.4. Metrics In addition to measurements, organizations also utilize metrics to track progress, facilitate decision-making, and improve performance. Information gained from measurements may be used to identify and define new metrics. Metrics can be applied at the system level, program level, 7 and organization level. System-level metrics, such as the frequency of third-party access to a system or the number of communication ports open, can facilitate tactical decision-making and support program-level metrics. Program-level metrics, such as the number of security incidents in a year or the cost per incident, may be helpful when making organizational strategic decisions. Both system- and program-level metrics can also support risk managementinformed decision-making. Metrics are designed to track progress, facilitate decision-making, and improve performance with respect to a set target. Metrics leverage measures to provide insight into how well an organization is performing at the program or system level and whether the organization is reducing their information security risk. As with measures, the characteristics of meaningful metrics include the value being objective, accurate, precise, tied to a fixed reference or point in time, replicable, and comparable to previous measurements. Metrics are set with organizational goals in mind and drive subsequent assessments whose results then inform the

7 SP 800-39, Managing Information Security Risk: Organization, Mission, and Information System View, includes a model of multi-level risk management for the integration of risk management across the organization. In this model, three levels are identified to address risk: (i) the organization level, (ii) the mission/business process level, and (iii) and the system level. For the purposes of this document, the program level can be synonymous with the mission/business process-level and/or the organization level.

9

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

metrics going forward. Figure 1 shows a notional process for the definition, collection, and analysis of metrics.

Fig. 1. Notional process for the definition, collection, and analysis of metrics

Knowing why measurements are being taken can help when selecting which potential measurements or metrics to focus on. The purpose and output of measurements must be unambiguous and easily understood, and the chosen metrics tell a meaningful story about organization-, program-, or system-level information security. For example, when evaluating cybersecurity awareness training, consider completion rates and the results of review quizzes instead of marking participation as “low, medium, or high.”

10

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

When initially selecting quantitative metrics, it is natural to want to adopt any available data points, perhaps with the intent to use them to narrow down focus on critical elements. Poorly selected quantitative metrics can undermine the overall quality of reporting and erode confidence in the work product. When considering a quantitative metric, consider whether the data point directly illustrates the performance of a valid control or tracks the presence of a material risk to the organization’s objective. If not, it may be best to exclude the data item in question. With quantitative metrics - initially at least - less is often more. By keeping metrics consistent over time, organizations can evaluate long-term trends and expected ranges. A new metric may provide important insights, but tracking the measurements related to metrics over a continuous period (e.g., quarter to quarter, year to year) will give more information about the success of organization-, program-, and system-level information security plans, policies, procedures, and goals. Some metrics may be gathered because of external guidance or regulations. Key risk indicators (KRIs) and key performance indicators (KPIs) are examples of metrics, though not all metrics fall into these categories. Organizations may find that a wide variety of metrics fit their needs. For example, appropriate measures at the organization level may include the cost per security incident as part of the budget allocation process, whereas measurements at the system level may include the frequency of virus scans across individual systems.

11

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

3. Measurement Considerations Because measurement can involve large amounts of data, having a plan for data handling is critical to ensuring that factors such as documentation, data management, data quality, and uncertainty are all considered. An organized and repeatable process that allows for the consistent assessment of collected data provides much-needed context for measurements. Information security measurements can be scoped to a variety of environments and needs. Assets, controls, vulnerabilities, and security incidents can all be measured. Measures can be applied to organizational units, sites, processes, and other constructs. Since the high volume of measurements may be too numerous to track, organizations may want to aggregate data to help track security trends across an organization. Organizations will carefully define the scope of measures based on specific stakeholder needs, strategic goals and objectives, operating environments, risk priorities, and resources. As an organization develops measurements, they may want to consider options to make the data collection process easier, such as automation. Information security measures can be applied at the system level to provide quantifiable data regarding the implementation, effectiveness, efficiency, and impact of required or desired security controls. System-level measures can be used to determine the system’s security posture, demonstrate compliance with organizational requirements, and identify areas of improvement. Measurements can be used to monitor organizational information security activities at the program and organization levels. These measurements may be derived by aggregating multiple system-level measures or developed by using the entire enterprise as the scope. Organizationlevel measurements require that the processes on which the measures depend are consistent and repeatable and ensure the availability of data across the organization. Perfectly measuring information security is challenging due to the gap between mathematical models and practical implementations [21]. Instead, experimenting as possible with relative metrics, models, and approaches over time is the best way to identify the most effective performance indicators. 3.1. Organizational Measures Considerations 3.1.1. Measures Documentation Organizations document their measures in a standard format to ensure the repeatability of measures development, collection, and reporting activities. By keeping a consistent record of what is being measured, where the data comes from, what formulas and calculations are being used, and who interacts with the data, it becomes easier to trace data and ensure continuity of the process.

12

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Organizations can tailor their standard format to their unique environments and requirements based on internal practices and procedures. However, the following fields offer a common starting point: •

Unique ID: A unique identifier for tracking and sorting. The unique identifier can use an organization-specific naming convention or directly reference another source.

•

Goal: Statement of strategic and/or information security goals to guide control implementation for system-level control measures and higher-level measures. These goals are usually articulated in strategic and performance plans. When possible, include both the organization-level goal and the specific information security goal extracted from organization documentation, or identify an information security goal that would contribute to the accomplishment of the selected strategic goal.

•

Scope: Definition of what is considered in and out of scope. The scope helps explain aggregated risks and distinguish the total risk from the risks currently being measured.

•

Measure: Statement of measurement. Use a numeric statement that begins with the words “percentage,” “number,” “frequency,” “average,” or other similar term.

•

Type: Statement of whether this is a record of implementation or a measure of effectiveness, efficiency, or impact.

•

Formula: Calculation that results in a numeric expression of a measure. The organization may also note the information gathered in an implementation survey.

•

Target: A range or a designated upper or lower bound. A threshold for a satisfactory rating for the measure (e.g., a milestone completion or statistical measure) can be expressed in percentages, time, currency, or other unit of measurement. The target may be tied to a required completion time frame. It may also be useful to select and record final and interim targets to track progress toward a stated goal.

•

Implementation evidence: Evidence used to compute the measure, validate that the activity is performed, and identify probable causes of unsatisfactory results for a specific measure. o For manual data collection, identify questions and data elements that would provide the data inputs necessary to calculate the measure’s formula, qualify the measure for acceptance, and validate the information provided. o For automated data collection, identify data elements that would be required for the formula, qualify the measure for acceptance, and validate the information provided.

•

Time-based reference: o When the measure was taken. This supports the repeatability of measures and helps identify expired or invalid data. o

How often the data is collected, analyzed, and reported. Select the frequency of data collection based on a rate of change that is being evaluated. Select the

13

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

frequency of data reporting based on external reporting requirements and internal customer preferences. •

Responsible parties: Key stakeholders, such as: o Information owner — Identify the organizational component and the individual who owns the required information. o Information collector — Identify the organizational component and the individual responsible for collecting the data. 8 o Information customer — Identify the organizational component and the individual who will review the data.

•

Data source: Location of the data to be used in calculating the measure, including databases, tracking tools, logs, organizations, and specific roles within the organization that can provide the required information.

•

Reporting format: Indication of how the measure will be reported, such as a pie chart, line chart, bar graph, or other format. It may also be beneficial to include a sample.

3.1.2. Measurement Reporting If a measure is reported to stakeholders without proper context, there can be unintended consequences or indecision. Therefore, a report could include elements such as: •

Any risk indicated by the measure

•

Whether the findings fit in the organization’s risk appetite

•

What prioritization, action, or decision might need to be taken

•

Any compliance issues that may be associated with the finding

•

The impact on being audit-ready

These examples are not meant to provide a complete list of elements to be included in measurement reporting but highlight how measurement reporting can support organizational decision-making by providing context around measurements and findings. More information about measurement reporting and communication about measurements can be found in SP 800-55v2, Measurement Guide for Information Security: Volume 2 — Program. 3.1.3. Data Management Although substantial amounts of information security data may be collected, not all data will be useful for the information security measurement program. Any data collected specifically for

8 When possible, the information collector will be a different individual or even a representative of a different organizational unit than the

information owner to avoid the possibility of a conflict of interest and ensure separation of duties, though this may not be feasible for smaller organizations.

14

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

information security measures are as non-intrusive as possible and of maximum usefulness to ensure that available resources are primarily used to correct problems rather than collect data. Information security data repositories represent a significant collection of operational and vulnerability data. Due to the sensitivity of this data, they are protected in accordance with applicable laws, regulations, policies, and procedures. 3.1.4. Data Quality Data collection methods and the data repositories used for measures data collection and reporting (either directly or as data sources) are clearly defined to ascertain the quality and validity of the data. This also helps ensure that testing is repeatable and can show changes over time. Data validity is suspect if the primary data source is an incident-reporting database that only stores information reported by a few organizational elements or if reporting processes between organizations are inconsistent. The importance of standardizing reporting processes cannot be overemphasized. When organizations are developing and implementing processes that may serve as inputs into an information security measurement program, ensuring that data gathering and reporting are clearly defined helps facilitate valid data collection. Having a validation process in place to check the integrity, accuracy, and structure of the data provides a way to address potential errors before any analysis is done. By setting a standard process to validate data, an organization can have a repeatable way to look at the data and ensure its quality. 3.1.5. Uncertainty and Errors Even when measurements are intended to be precise and accurate, random and systemic errors can still occur. While there is no guaranteed way to measure uncertainty in all measurements, statistical information calculated from the data (e.g., standard deviation, standard error of mean, and confidence intervals) can provide more insight. Uncertainty can be reduced by using data cleaning methods, such as validation, normalization, transformation, and imputation, as shown in Table 3.

15

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures Table 3. Data cleaning methods for reducing uncertainty

Data Cleaning Method Data Validation Normalization Transformation Imputation

Definition The process of determining that collected data is acceptable according to a predefined set of tests [15] The conversion of information into consistent representations and categorizations [4] The conversion of data from one state or format into another state or format The replacement of unknown, unmeasured, or missing data with a particular value. The simplest form of imputation is to replace all missing values with the average of that variable [18].

In addition to making the data itself more useable, data analysis methods can address uncertainty within the data. Organizations often make quantitative projections using statistical methods, such as regression, time series analysis, and machine learning methods. When looking at projections, it is helpful to consider that future events and other unknown factors can cause unforeseen changes. 3.2. Characteristics of Measures The reliability and effectiveness of security measures require their adherence to a set of defined characteristics. These characteristics act as a foundation for the development, implementation, and assessment of information security measures: •

Accuracy: Collecting and analyzing accurate data that aligns closely with the security objectives and requirements specified within the assessment’s scope and reflects the intended security objectives under evaluation.

•

Numeric precision: Using precise and objective data that naturally uses real numeric values. This facilitates consistent analysis and comparison and enables stakeholders to make informed decisions regarding security posture.

•

Correctness: Data collection processes and methodologies that adhere to predetermined specifications and standards to provide accuracy and reliability. The correctness of data collection procedures ensures the consistency and validity of results.

•

Consistency: Security measurements that remain independent of the individuals or entities conducting them. Consistency in measurement methodologies and criteria fosters reliability across different evaluators and environments and enhances the trustworthiness of assessment outcomes.

•

Time-based reference: Establishing a fixed reference point for data collection and analysis. Time-based measurements provide context and facilitate the identification of evolving security threats and vulnerabilities.

•

Replicability: Security measurements that are repeatable under identical conditions and yield consistent results across multiple assessments. Replicability ensures the reliability 16

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

and validity of measurement outcomes and supports longitudinal analysis and benchmarking efforts. •

Unit-based standardization: Data that is expressed using standardized units and formats. Unit-based standardization facilitates interoperability and comparability across diverse datasets and evaluation contexts.

These characteristics collectively define the foundation of robust security measurement frameworks. Adherence to these principles ensures the reliability, consistency, and replicability of security measure developed to safeguard assets and information. 3.3. Types of Measures This document separates measures/assessment results into four types: 1. Implementation 2. Effectiveness 3. Efficiency 4. Impact 3.3.1. Implementation Measures Implementation measures demonstrate the progress of specific controls. Monitoring implementation may include assessment results, such as a tally of known systems or a binary “yes/no” about which systems have up-to-date patches. 9 Implementation measures look at quantitative outputs and are usually demonstrated in percentages. Examples of implementation measures related to information security programs include the percentage of systems with approved system security plans and the percentage of systems with password policies that are configured as required. Implementation measures can also examine systemlevel areas, such as the percentage of servers in a system with a standard configuration. By gathering this data, an organization can understand how its goals are being implemented and what tasks still need to be accomplished. Organizations never fully retire implementation measures because they are a record of what exists and what needs improvement. However, once implementation measures are completed, the emphasis and resources of the measurement program broaden to include effectiveness, efficiency, and impact measures. 3.3.2. Effectiveness Measures Effectiveness measures evaluate how well implementation processes and controls are working and whether they are meeting desired outcomes. An effectiveness assessment can either concentrate on the evidence and results of a quantitative analysis of measures or be applied in a qualitative “yes/no” paradigm. Effectiveness measures may require multiple data points that 9 Records of these essential implementation assessment results are foundational to information security measurement and are addressed in SP 800-55v2.

17

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

quantify the degree to which information controls are implemented and their effects on the organization’s information security posture. 3.3.3. Efficiency Measures Efficiency measures examine the timeliness of controls by determining the speed at which they give useful feedback and how quickly those issues are addressed. An efficiency assessment concentrates on the evidence and results of quantitative measures analysis. Effectiveness and efficiency together are often referred to as Program Results. 3.3.4. Impact Measures Impact measures articulate the impact of information security on an organization’s unique mission, goals, and objectives by quantifying the following: •

Cost savings produced by the information security program

•

Costs incurred from addressing information security events

•

The cost-effectiveness of any given processes and controls

•

Business value gained or lost

•

Regulatory fines

•

Contractual penalties

•

The degree of public trust gained or maintained by the information security program

•

Other mission-related impacts of information security

These measures combine the results of control implementation with a variety of information about resources. They can provide the most direct insight into the value of information security to the organization and are sought by executives. While impact measures are largely organization-specific, a general example would be the percentage of the organization’s budget devoted to information security controls. After implementation measures are gathered, further analysis — not dissimilar to normalization — is required before the results can be expressed to a non-technical audience.

18

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

4. Selecting and Prioritizing Measures Developing and selecting information security measures consists of four major activities: 1. Identifying and defining the current information security program 2. Developing, testing, and validating specific measures to gauge the implementation, program results, and impacts of security controls to ensure that they adequately safeguard the relevant attributes of the assets being protected 3. Prioritizing measures based on organizational needs 4. Evaluating collected measures data 4.1. Identification and Definition This document focuses on the development of measures related to information security risk management, which is part of a larger implementation process of information security measurement.10 The identification and definition of the existing information security program are important to the development of measures. Identification and definition include: •

Establishing and analyzing assets: An accounting of the assets in place and their potential impacts on the organization

•

Stakeholders and interests: Identifying relevant stakeholders and their interests in information security measurement

•

Goals and objectives: Identifying and documenting security goals and objectives that will guide control implementation and ensuring that they can be reliably and objectively translated from technical data sources to meaningful business intelligence

•

Information security policies, guidelines, and procedures review: Examining existing organization-specific policies, guidelines, and procedures related to information security

•

Information security implementation review: Reviewing any existing measures and data repositories that can be used to derive measures data

4.2. Developing, Testing, and Validating Measures Knowing what controls are implemented in an organization is foundational to quantitative assessment. The system- and program-level controls that need to be tracked must be understood before an organization can evaluate what kinds of measurements to take or the process of prioritizing potential measures. This creates a structure for determining what measurements need to be taken and what metrics are used for evaluation. When developing measures, an organization needs to know the attributes of the asset it is trying to protect. For example, there is no organizational gain in trying to measure the 10 Refer to SP 800-55v2 for more information.

19

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

confidentiality of a Domain Name System (DNS) registry when the critical characteristics are its integrity and availability. As an organization cannot take and evaluate infinite measures, the most successful measures consider stated priorities that inform the organization’s information security strategy. For example, if the organization considers the resource availability of a public website to be of critical importance, then selecting “the average number of port scans per day” as a measure might yield an accurate and informative result but not one that is relevant to the stated goals of the organization. 4.2.1. Comparing Measures and Assessment Results Qualitative and semi-quantitative assessments may also be useful or even necessary to assess implementation, effectiveness, efficiency, and impact, as shown in Table 4. Table 4. Examples of measures and types of qualitative and semi-quantitative assessment results Examples of Qualitative or SemiQuantitative Assessment Results Determine whether identified controls are in place.

Examples of Measures and Assessments The percentage of systems with up-to-date patches (i.e., implementation of a specific control or capability)

Effectiveness: Examine how well controls are working.

Use a color-coded risk matrix to demonstrate the potential risks involved with improperly configured access controls.

A chart that shows the changes of percentage of information security incidents caused by improperly configured access controls over a 5-year period

Efficiency: Examine the timeliness of controls.

Use a 1–5 scale to determine whether the organization is at an acceptable level of responsiveness in case of an information security incident.

Data that compares the mean time of response to information security incidents versus the cost of the incident

Impact: Examine the impact of information security on an organization’s mission.

Rank risks on a color-coded scale to evaluate financial impacts to an organization.

Data on the known costs of breaches to industry peers

Assessment Types Implementation: Examine the progress of specific controls.

After implementation measures are gathered, further analysis, not dissimilar to data normalization, is required before the results can be expressed to a non-technical audience. Successful effectiveness and efficiency assessments of individual phenomenon may require additional analysis using multiple measures. For example, when looking for “delayed vulnerability remediations,” simply checking the mean time to incident response is unlikely to provide all of the information necessary to determine the causes of an issue.

20

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

4.3. Prioritizing Measures After implementation measures are in place, organizations prioritize which efficiency, effectiveness, and impact measures to implement. Prioritization can be driven by a variety of factors, including an organization’s risk management strategy, mission and business objectives, the availability of data collection for target assets, the value of the asset, the cost of the controls required to prevent adverse outcomes, the potential availability of imperfect or partial controls, information from risk assessments, policies, and legal, regulatory, or other requirements. 4.3.1. Likelihood and Impact Modeling Likelihood and impact modeling are meant to work in tandem as part of a larger risk assessment process.11 Simply knowing either the likelihood or the potential impact of an event is not enough information to determine the importance of a potential measure to an organization. Identifying existing data for use in likelihood and impact modeling typically involves working with stakeholders from across the organizational structure. When possible, data from existing risk assessments can be utilized to reduce redundancy and enable decision-making (e.g., using existing modeled data to help decide what measurements to prioritize). Organizations may also have useful data from audits, interviews, surveys, or studies and external data on likelihood and impact. Annual reports can provide information on threat landscapes and the financial impacts of information security incidents that can be used to create models. A wide range of event likelihood models can be used to assess the likelihood of adverse events when determining which systems and controls to measure. Organizations can also compare impact models with event probability models (e.g., expected loss and statistical analysis of historical market trends) to determine their measurement priorities. Controls or systems with higher likelihoods of incident or higher potential impacts if affected could then be prioritized when organizations decide how to allocate measurement resources. Where possible, leverage existing event likelihood and impact models (e.g., risk registers12) to avoid a duplication of efforts. More information on quantitative likelihood and impact models can be found in Appendix C. In addition to using historical information for likelihood and impact modeling, current trends may provide useful datapoints when prioritizing and selecting measures. Staying updated on current threats allows for more effective continuous measurement and assessment. At the same time, recency bias 13 about current events often influences choices when determining courses of action and resource allocation. Outliers and unexampled events may occur over time. An organization can prepare for these issues using horizon scanning, stress tests, and system resilience. 14 11

More information on risk assessments can be found in SP 800-30, Guide for Conducting Risk Assessments.

12 More information on risk registers can be found in [4].

13 Recency bias is the tendency to favor recent events or experiences over historical ones. 14 More information on cyber resiliency can be found in SP 800-160v2.

21

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

4.3.2. Weighing Scale Information gained from modeling likelihood and impact can be combined with knowledge about organizational goals and existing controls to create a customized weighing scale to prioritize potential measures. Using a weighing scale with set parameters ensures consistency when prioritizing and selecting measures, even those that are unrelated to information security. Measures that are ultimately selected are useful for: •

Identifying causes of unsatisfactory performance

•

Pinpointing areas for improvement

•

Facilitating consistent policy implementation

•

Redefining goals and objectives

•

Modifying security policies

4.4. Evaluating Methods for Supporting Continuous Improvement After an organization selects its measures, the collected data is evaluated. Evaluation may look different depending on the types of measures being analyzed. Quantitative data analysis methods, like those in Sec. 2.3, can be used to evaluate measures. For implementation measures, evaluation may be as simple as comparing the percentage of controls implemented with the goal percentage of implementation. Effectiveness, efficiency, and impact measures will likely be more complicated to evaluate. Both effectiveness and efficiency measures often begin by establishing average data output and evaluating acceptable ranges against output going forward. For example, an organization may want to know if the volume of data being transferred on the network has an anomaly. To monitor for changes, the average volume of data transferred is established. An organization may also set an acceptable range based on a standard deviation from this average. This may mean looking for outliers in the data or monitoring for changes over time. Evaluating impact measures will likely include outcomes outside of information security, such as financial outcomes or even public perception. Various indicators and inputs can be useful to track the effectiveness and efficiency of an information security program by monitoring performance and security over time, such as: •

False positive rate: The proportion of positive reports that were incorrectly identified

•

Key performance indicators: A measure of progress toward intended results

•

Key risk indicators: A metric used to measure risk

•

Leading indicators: A predictive metric that tracks events or behaviors that precede incidents

•

Lagging indicator: A metric that tracks the outcome of events or trends

22

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

•

Mean time to detect: A metric that tracks the average amount of time that a problem exists before it is found

•

Mean time to recovery: A metric that tracks the average amount of time it takes to recover from a product or system failure

•

Mean time between failures: A predictive metric that tracks the average time between system breakdowns

•

Mean time to repair: A predictive metric that tracks the average time it takes to repair a system

Access to average outputs, acceptable ranges, and long-term data makes effectiveness and efficiency measures more accurate and beneficial by enabling organizations to track changes over time. Even if processes are not yet consistent, average outputs and acceptable ranges help organizations set metrics. Some metrics are directly related to established averages, while others are set by other sources, and established ranges may not have any effect on organizational goals. While inconsistent processes will not provide meaningful data, measurements may still be used to establish average outputs and acceptable ranges for future analysis. Data analysis for finding average outputs and acceptable ranges will typically include historical data and a forecast of what that trend may continue to look like in the future if all variables stay the same. Some measures also have the potential to give misleading information. Inputs such as phishing test success rates or the number of known vulnerabilities depend heavily on the quality of work behind them. A poorly designed phishing test might show a better success rate while giving less information about the preparedness of the workforce to recognize a well-designed phishing email. This does not mean that organizations need to avoid these measures altogether, but numbers alone may not always show the whole story.

23

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

References [1]

[2]

[3]

[4]

[5] [6] [7] [8] [9] [10]

[11]

[12]

National Institute of Standards and Technology (2024) The NIST Cybersecurity Framework (CSF) 2.0. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Cybersecurity White Paper (CSWP) NIST CSWP 29. https://doi.org/10.6028/NIST.CSWP.29 National Institute of Standards and Technology (2006) Minimum Security Requirements for Federal Information and Information Systems. (U.S. Department of Commerce, Washington, DC), Federal Information Processing Standards Publication (FIPS) 200. https://doi.org/10.6028/NIST.FIPS.200 Bowen P, Kissel RL (2007) Program Review for Information Security Management Assistance (PRISMA). (National Institute of Standards and Technology, Gaithersburg, MD), NIST Interagency or Internal Report (IR) NIST IR 7358. https://doi.org/10.6028/NIST.IR.7358 Stine KM, Quinn SD, Witte GA, Gardner RK (2020) Integrating Cybersecurity and Enterprise Risk Management (ERM). (National Institute of Standards and Technology, Gaithersburg, MD), NIST Interagency or Internal Report (IR) NIST IR 8286. https://doi.org/10.6028/NIST.IR.8286 Taylor BN (2011) The current SI seen from the perspective of the proposed new SI. Journal of Research of the National Institute of Standards and Technology 116(6):797. https://doi.org/10.6028/jres.116.022 Software Quality Group (2021) Metrics and Measures. (National Institute of Standards and Technology, Gaithersburg, MD). Available at https://www.nist.gov/itl/ssd/softwarequality-group/metrics-and-measures Thomas D (2019). Monte Carlo Tool. (National Institute of Standards and Technology, Gaithersburg, MD). Available at https://www.nist.gov/servicesresources/software/monte-carlo-tool ASTM International (2018) ASTM C1012/C1012M-18a – Standard Test Method for Length Change of Hydraulic-Cement Mortars Exposed to a Sulfate Solution (ASTM International, West Conshohocken, PA). https://doi.org/10.1520/C1012_C1012M-18A Joint Task Force Transformation Initiative (2012) Guide for Conducting Risk Assessments. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) 800-30, Rev. 1. https://doi.org/10.6028/NIST.SP.800-30r1 Joint Task Force (2018) Risk Management Framework for Information Systems and Organizations: A System Life Cycle Approach for Security and Privacy. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-37, Rev. 2. https://doi.org/10.6028/NIST.SP.800-37r2 Chew E, Swanson MA, Stine KM, Bartol N, Brown A, Robinson W (2008) Performance Measurement Guide for Information Security. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-55, Rev. 1. https://doi.org/10.6028/NIST.SP.800-55r1 Grance T, Nolan T, Burke K, Dudley R, White G, Good T (2006) Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities. (National Institute of Standards and

24

NIST SP 800-55v1 December 2024

[13] [14]

[15]

[16] [17]

[18] [19] [20]

[21] [22]

[23]

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-84. https://doi.org/10.6028/NIST.SP.800-84 Kent K, Souppaya M (2006) Guide to Security Log Management. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-92. https://doi.org/10.6028/NIST.SP.800-92 Dempsey KL, Pillitteri VY, Baer C, Niemeyer R, Rudman R, Urban S (2020) Assessing Information Security Continuous Monitoring (ISCM) Programs: Developing an ISCM Program Assessment. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-137A. https://doi.org/10.6028/NIST.SP.800-137A Barker E, Smid M, Branstad D (2015) A Profile for U.S. Federal Cryptographic Key Management Systems. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-152. https://doi.org/10.6028/NIST.SP.800-152 Ross R, Winstead M, McEvilley, M (2022) Engineering Trustworthy Secure Systems. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-160v1r1. https://doi.org/10.6028/NIST.SP.800-160v1r1 Boyens JM, Smith AM, Bartol N, Winkler K, Holbrook A, Fallon M (2022) Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-161r1-upd1, Includes updates as of November 1, 2024. https://doi.org/10.6028/NIST.SP.800-161r1-upd1 Heckert N, Filliben J, Croarkin C, Hembree B, Guthrie W, Tobias P, Prinz J (2012) NIST/SEMATECH e-Handbook of Statistical Methods. (National Institute of Standards and Technology, Gaithersburg, MD). https://doi.org/10.18434/M32189 Stevens SS (1946) On the Scales of Measurement. Science 103(2684):677-680. http://www.jstor.org/stable/1671815 Turan MS, Barker E, Kelsey J, McKay KA, Baish ML, Boyle M (2018) Recommendation for the Entropy Sources Used for Random Bit Generation. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-90B. https://doi.org/10.6028/NIST.SP.800-90B Stolfo S, Bellovin S, Evans D (2011) Measuring Security. IEEE Security & Privacy 9(3):6065. https://doi.org/10.1109/MSP.2011.56 Ross R, Pillitteri V, Graubart R, Bodeau D, McQuaid R (2021) Developing Cyber-Resilient Systems: A Systems Security Engineering Approach. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-160v2r1. https://doi.org/10.6028/NIST.SP.800-160v2r1 Schroeder K, Trinh H, Pillitteri VY (2024) Measurement Guide for Information Security: Volume 2 — Developing an Information Security Measurement Program. (National Institute of Standards and Technology, Gaithersburg, MD), NIST Special Publication (SP) NIST SP 800-55v2. https://doi.org/10.6028/NIST.SP.800-55v2

25

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Appendix A. Glossary assessment The action of evaluating, estimating, or judging against defined criteria. Different types of assessment (i.e., qualitative, quantitative, and semi-quantitative) are used to assess risk. Some types of assessment yield results. assessment results The output or outcome of an assessment. Bayesian methodology Statistical approach to data analysis based on Bayes’ theorem where uncertainty is quantified by combining existing information with new information to create forecast models. [18, adapted] classical data analysis A data analysis technique where data collection is followed by the imposition of a model, and the analysis, estimation, and testing that follow focus on the parameters of that model. [18, adapted] data validation The process of determining that data or a process for collecting data is acceptable according to a predefined set of tests and the results of those tests. [15] experimentation A systematic approach to the process of testing new ideas, methods, or activities that applies principles and techniques at the data collection stage to ensure the generation of valid, defensible, and supportable conclusions. exploratory data analysis A data analysis technique where data collection is immediately followed by analysis with the goal of inferring what model would be appropriate. [18, adapted] false positive An erroneous acceptance of the hypothesis that a statistically significant event has been observed. [20] imputation The replacement of unknown, unmeasured, or missing data with a particular value. The simplest form of imputation is to replace all missing values with the average of that variable. [18, adapted] information security The protection of information and systems from unauthorized access, use, disclosure, disruption, modification, or destruction to provide confidentiality, integrity, and availability. [2] interval scale From the Stevens Scale of Measurement, a quantitative measurement scale using variables with equal values and no true zero, such as time and temperature. [19, adapted] key performance indicator A metric of progress toward intended results. key risk indicator A metric used to measure risk. lagging indicator A metric that tracks the outcome of events or trends. leading indicator A predictive metric that tracks events or behaviors that precede incidents.

26

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

machine learning The development and use of computer systems that adapt and learn from data with the goal of improving accuracy. mean The sum of the data points divided by the number of data points. Commonly referred to as the average. [18, adapted] mean time to detect A metric that tracks the average amount of time that a problem exists before it is found. mean time to recovery A metric that tracks the average amount of time that it takes to recover from a product or system failure. measurement The process of obtaining quantitative values using quantitative methods. measures Quantifiable and objective values that result from measurement. median The value of the point that has half the data smaller than that point and half the data larger than that point. [18] metrics Measures and assessment results designed to track progress, facilitate decision-making, and improve performance with respect to a set target. mode The value of the random sample that occurs with the greatest frequency. This value is not necessarily unique. [18] Monte Carlo analysis A probabilistic sensitivity analysis used to account for uncertainty. [7] nominal scale From the Stevens Scale of Measurement, a scale that labels named variables into classifications. [19, adapted] normalization The conversion of information into consistent representations and categorization. [4] observational data Data captured through the observation of an activity or behavior without the direct involvement of the subject. ordinal scale From the Stevens Scale of Measurement, a scale that orders and ranks data without establishing a degree of variation between ranks. [19, adapted] outliers An observation that lies an abnormal distance from other values in a random sample from a population. [18] qualitative assessment The use of a set of methods, principles, or rules for assessing risk based on nonnumerical categories or levels. [6] quantitative assessment The use of a set of methods, principles, or rules for assessing risk based on numbers where the meanings and proportionality of values are maintained inside and outside of the context of the assessment. [6]

27

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

random sampling A method of sampling where each sample has an equal chance of selection in hopes of gathering an unbiased representation. [18, adapted] ratio scale From the Stevens Scale of Measurement, a quantitative measurement scale with a true zero using variables that can be compared to find differences or intervals. [19] regression A statistical technique used to predict the value of a variable based on the relationship between explanatory variables. sampling The process of taking samples of something for the purpose of analysis. semi-quantitative assessment The use of a set of methods, principles, or rules for assessing risk based on bins, scales, or representative numbers whose values and meanings are not maintained in other contexts. [9] stratified sampling The process of segmenting a population across levels of some factors to minimize variability within those segments. [18] systematic stratified sampling A method of sampling where samples are taken at a regular interval. [18, adapted] time series analysis The analysis of an ordered sequence of values of a variable at equally spaced time intervals. [18, adapted] transformation The conversion of one state or format into another state or format.

28

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Appendix B. Data Analysis Dictionary The following information is found in the NIST Engineering Statistics Handbook [18]. B.1. Bayesian Methodology Bayesian or predictive methodology consists of formally combining the prior distribution on the parameters and the collected data to jointly make inferences and/or test assumptions about the model of parameters. •

•

Bayes Formula

Law of Probability

P(A|B) =

𝑃𝑃(𝐴𝐴, 𝐵𝐵) 𝑃𝑃(𝐴𝐴) × 𝑃𝑃(𝐵𝐵|𝐴𝐴) = 𝑃𝑃(𝐵𝐵) 𝑃𝑃(𝐵𝐵) 𝑛𝑛

P(B) = � 𝑃𝑃(𝑃𝑃|𝐴𝐴𝑖𝑖 )𝑃𝑃(𝐴𝐴𝑖𝑖 ) B.2. Classical Data Analysis

𝑖𝑖=1

Classical data analysis is when data collection is followed by a model, and the subsequent analysis, estimation, and testing focus on the parameters of that model. Classical data analysis includes deterministic and probabilistic models, such as regression and ANOVA. Some of the more common relevant classical quantitative models include: Location •

Measures of Location (mean, median, and mode)

•

Confidence Limits for Mean and One Sample t-Test

•

Two Sample t-Test for Equal Means

•

One Factor Analysis of Variance

•

Multi-Factor Analysis of Variance

Scale (or variability or spread) •

Measures of Scale

•

Bartlett’s Test

•

Chi-Square Test

•

F-Test

•

Levene Test

29

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Skewness and Kurtosis •

Measures of Skewness and Kurtosis

Randomness •

Autocorrelation

•

Runs Test

Distributional Measures •

Anderson-Darling Test

•

Chi-Square Goodness of Fit Test

•

Kolmogorov-Smirnov Test

Outliers •

Detection of Outliers

•

Grubbs Test

•

Tietjen-Moore Test

•

Generalized Extreme Deviate Test

2-Level Factorial Designs •

Yates Algorithm

B.3. Exploratory Data Analysis Exploratory data analysis emphasizes graphical techniques and inferring different analytic models in order to determine what model would be appropriate. Some common exploratory data analysis graphical techniques include: Univariate •

Run Sequence Plot

•

Lag Plot

•

Histogram

•

Normal Probability Plot

•

4-Plot

•

PPCC Plot

•

Weibull Plot

•

Probability Plot

𝑦𝑦 = 𝑐𝑐 + 𝑒𝑒

30

NIST SP 800-55v1 December 2024

•

Box-Cox Linearity Plot

•

Bootstrap Plot

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Time Series 𝑦𝑦 = 𝑓𝑓(𝑡𝑡) + 𝑒𝑒

•

Run Sequence Plot

•

Spectral Plot

•

Autocorrelation Plot

•

Complex Demodulation Amplitude Plot

•

Complex Demodulation Phase Plot

•

Decomposition

1 Factor •

Scatter Plot

•

Box Plot

•

Bihistogram

•

Quantile Plot

•

Mean Plot

•

Standard Deviation Plot

𝑦𝑦 = 𝑓𝑓(𝑥𝑥) + 𝑒𝑒

Multi-Factor/Comparative •

Block Plot

Multi-Factor/Screen

𝑦𝑦 = 𝑓𝑓(𝑥𝑥𝑥𝑥, 𝑥𝑥1, 𝑥𝑥2, … , 𝑥𝑥𝑥𝑥) + 𝑒𝑒 𝑦𝑦 = 𝑓𝑓(𝑥𝑥1, 𝑥𝑥2, 𝑥𝑥3, … , 𝑥𝑥𝑥𝑥) + 𝑒𝑒

•

DOE Scatter Plot

•

DOE Mean Plot

•

DOE Standard Deviation Plot

•

Contour Plot

31

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Appendix C. Modeling Impact and Likelihood This appendix is intended to provide a high-level overview of complex statistical concepts. The successful application of these concepts will require further training and understanding on the part of practitioners. C.1. Bayesian Methodology Bayes’ formula expresses the conditional probability of event A given event B written as P(A|B). It can be calculated using Bayes’ Rule: P(A|B) =

𝑃𝑃(𝐴𝐴, 𝐵𝐵) 𝑃𝑃(𝐴𝐴) × 𝑃𝑃(𝐵𝐵|𝐴𝐴) = 𝑃𝑃(𝐵𝐵) 𝑃𝑃(𝐵𝐵)

Bayesian or predictive methodology is applied when there is previous knowledge of the conditions associated with an event. It can provide conditional probability estimates quickly and without using significant resources. Because Bayesian methodology relies on prior information, it is important to note that the use of either inaccurate or a different selection of prior information may lead to results that do not provide significant insight. C.2. Monte Carlo Methodology The Monte Carlo method is a multiple probability simulation used to predict possible outcomes of an uncertain event. The Monte Carlo method uses randomly generated outcomes within a set range, and the frequencies of different outcomes generated form a normal distribution. The Monte Carlo method allows for repeated modeling and can be performed using spreadsheet editors or programing languages for statistical computing. When using the Monte Carlo method, it is important to note that these simulations show an estimated probability and not an inevitable outcome. C.3. Time Series Analysis Time series analysis shows the level, trend, seasonality, or noise within a series of data points in a time series. Time series data is often found when monitoring a process over a period. Time series analysis considers the potential for an internal structure, such as trends or seasonal variations to data. Time series regression models are primarily used for forecasting. Time series decomposition exhibits patterns within time series data and can be useful when setting the expected range or use of processes or systems.

32

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

C.4. Value at Risk Value at risk (VaR) is a statistical analysis technique that builds a model that measures the risk of loss, primarily using a probability density function. The three key elements of building a VaR model are a fixed time period, a specific level of loss in value, and a confidence interval. Calculating VaR can be helpful when making decisions about investments and resources. Like all predictive models, VaR relies heavily on the quality of inputs and cannot effectively estimate all scenarios.

33

NIST SP 800-55v1 December 2024

Measurement Guide for Information Security Volume 1 — Identifying and Selecting Measures

Appendix D. Change Log In December 2024, the following changes were made to this Special Publication: •

Separated document into two volumes. Volume 1 focuses on identifying and selecting measures, and Volume 2 focuses on developing a measurement program.

•

The information originally found in Sec. 2, Roles and Responsibilities, has been updated and can be found in SP 800-55v2.

•

The information originally found in Sec. 4, Legislative and Strategic Drivers, has been removed.

•

The process ordinally found in Sec. 5, Measures Development Process, has been updated and can be found in SP 800-55v2, Developing a Measurement Program.

•

A new Sec. 1.4, Document Terminology, explores terminology that is relevant to the measurement and analysis of information security.

•

A new Sec. 2, Fundamentals, has subsections that explore types of assessment and metrics.

•

Section 3 now focuses on measurement considerations (formerly Sec. 3.4) and types of measures (formerly Sec. 3.3).

•

Section 3.1, Organizational Measures Considerations, has new information about measures documentation and reporting, data quality, and uncertainty.

•

A new Sec. 3.2 describes the characteristics of successful measures.

•

In Sec. 3.3, efficiency measures and effectiveness measures are now listed as two separate kinds of measures, where they were formerly grouped as program results.

•

Section 4 focuses on selecting and prioritizing measures. It is expanded from the former Sec. 5.5 and now includes information about developing, testing, and validating measures (Sec. 4.2); comparing measures and assessment results (Sec. 4.2.1); prioritizing measures (Sec. 4.3) using likelihood and impact modeling (Sec. 4.3.1) and a weighing scale (Sec. 4.3.2); and evaluating methods for supporting continuous improvement (Sec. 4.4).

•

Appendix A, Candidate Measures, was removed.

•

Appendix B, Data Analysis Dictionary, was added.

•

Appendix C, Modeling Impact and Likelihood, was added.

34

Related documents

• NIST SP 800-55v2: Measurement Guide for Information Security: Volume 2 — Developing an Information Security Measurement Program#3017
• NIST SP 800-59: Guideline for Identifying an Information System as a National Security System#3026
• NIST SP 800-171Ar3: Assessing Security Requirements for Controlled Unclassified Information#2919
• On the Necessity of Establishing a National Cybersecurity Testbed#1178859
• NIST SP 800-60v1r1: Guide for Mapping Types of Information and Information Systems to Security Categories#3027