Governance of Information Security Metrics Do Not Necessarily Improve Security

¶ … Security Metrics

Governance of Information Security: Why Metrics Do Not Necessarily Improve Security

The objective of this study is to examine the concept that the use of various Metrics has tended to improve security however, Metrics alone may not necessarily improve security. This study will focus on two well-known metrics.

The work of Barabanov, Kowalski and Yngstrom (2011) states that the greatest driver for information security development in the majority of organizations "is the recently amplified regulatory environment, demanding greater transparency and accountability. However, organizations are also driven by internal factors, such as the needs to better justify and prioritize security investments, ensure good alignment between securities and the overall organizational mission, goals, and objectives, and fine-tune effectiveness and efficiency of the security programs." (p.1)

It is reported that a survey conducted by Frost and Sullivan demonstrated "that the degree of interest in security metrics among many companies (sample consisted of over 80) was high and increasing (Ayoub, 2006); while, in a global survey sponsored by ISACA, dependable metrics were perceived to be one of the critical elements of information security program success by many security professionals and executives, though, they were also deemed difficult to acquire (O'Bryan, 2006)." (Barabanov, Kowalski and Yngstrom, 2011, p.2)

In addition, it is reported that the focus on governance includes a "need for proper measurement and reporting on all the echelons within the organization, starting at the highest level. Another survey instigated by ISACA showed that organizations that are missing an information security governance project had identified metrics and reporting as the areas in their information security programs where the lack of quality was most noticeable." (Barabanov, Kowalski and Yngstrom, 2011, p.2) Barabanov, Kowalski and Yngstrom report that the correlation reported in their study highlights the requirement of recognizing "that measurement and reporting are connected with management on all organizational levels." (Barabanov, Kowalski and Yngstrom, 2011, p.2)

I. Defining Metrics

There is reported to be a great deal of ambiguity in relation to the precise definition of the term metric or 'security metric' according to Barabanov, Kowalski and Yngstrom (2011) since the terms "security metric and measure tend to be used interchangeably." (p.3) Definitions that have been proposed are stated to include those as follows:

(1) measure - A variable to which a value is assigned as the result of measurement where measurement is defined as the process of obtaining information about the effectiveness of Information Security Management Systems (ISMS) and controls using a measurement method, a measurement function, an analytical model, and decision criteria (ISO/IEC, 2009a).

(2) (IS) Measures - the results of data collection, analysis, and reporting, which are based on, and monitor the accomplishment of, IS goals and objectives by means of quantification (Chew et al., 2008).

(3) Metric - a consistent standard for measurement, the primary goal of which is to quantify data in order to facilitate insight (Jaquith, 2007)

(4) Metric - a proposed measure or unit of measure that is designed to facilitate decision making and improve performance and accountability through collection, analysis, and reporting of relevant data (Herrmann, 2007).

(5) Metrics - broad category of tools used by decision makers to evaluate data. A metric is a system of related measures that facilitates the quantification of some particular characteristic. In simpler terms, a metric is a measurement that is compared to a scale or benchmark to produce a meaningful result (McIntyre et al., 2007).

(6) Security Metrics - the standard measurement of computer security (Rosenblatt,2008).Although the specifics of the different definitions are subject to some variation, certain common characteristics generally emerge. (Barabanov, Kowalski and Yngstrom, 2011, p.20)

Primarily, metrics and measures are "considered to be measurement standards that that facilitate decision making by quantifying relevant data, where measurement refers to the process by which they are obtained. " (Barabanov, Kowalski and Yngstrom, 2011, p.20)

Stoddard, et al. (2005) reports that the term metrics "…describes a broad category of tools used by decision makers to evaluate data in many different areas of an organization. In its simplest form, a metric is a measurement that is compared to a scale or benchmark to produce a meaningful result." (p.3)

II. Characteristics of Good Metrics

The characteristics of good metrics is reported to include the following:

(1) Metrics should measure and communicate things that are relevant in the specific context for which they are intended, and be meaningful (in both the content and the presentation) to the expected target audience.

(2) The value of metrics should obviously not exceed their cost. Measures should be cheap/easy enough to obtain so that potential inefficiencies of data collection do not pull the resources needed for subsequent stages of measurement or in other parts and functions of the organization.

(3) The timeliness and frequency of measurement has to be appropriate for the rate of change of the targets of measurement so that the latency of metrics does not defeat their purpose. It should also be possible to track changes over time.

(4) Good metrics should ideally be objective and quantifiable. This implies that they have to be derived from precise and reliable numeric values (and not qualitative assessments, which have potential for bias), and likewise be expressed by using readily understood and unambiguous units of measure; and (5) Metrics have to be consistently reproducible by different evaluators under similar circumstances and, therefore, a sufficient level of formality is expected from the defined measurement procedures. (Barabanov, Kowalski and Yngstrom, 2011, p.21)

The majority of these characteristics can be realized through "a high degree of standardization and, wherever possible, automation of the measurement related processes." ( )

III. Dimensions of Metrics

Various dimensions of metrics exist including the following stated dimensions:

(1) Governance, Management. And Technical;

(2) Management, Operational, and Technical;

(3) Organizational, Operational, and Technical

(4) Program Development, Support, Operational, and Effectiveness

(5) Organizational and Performance, Operational, Technological, Business Process, Business Value, and Compliance

(6) Implementation, Effectiveness and Efficiency, and Business Impact. (Barabanov, Kowalski and Yngstrom, 2011, p.16)

For the purpose of this study, the metrics focused on in this study are those of (1) governance and (2) technical metrics.

IV. Governance Metrics

Governance metrics are those "that address the responsibilities of the Board of Directors or Trustees and associated controls." (Barabanov, Kowalski and Yngstrom, 2011, p.5) Technical metrics are those that "deal with controls contained within and executed by and IT environment." (Barabanov, Kowalski and Yngstrom, 2011, p.5) Metrics are reported to be separated into three different subsets including: (1) All or complete set of metrics established in the report and which are used as a reference and likely to be impractical for implementation in its entirety; (2) baseline or the minimum required set of metrics for use as a starting point for a metrics program that is more comprehensive; and (3) SME or metrics that are suitable to be implemented in both small and medium organizations. (Barabanov, Kowalski and Yngstrom, 2011, p.6)

The work of Pironti (2008) reports that key to effective governance is "meaningful understanding of business effectiveness," the "ability to measure processes for constant improvement," and "early warning radar for threats and vulnerabilities." (p.1) Business aligned knowledge is stated to be a great benefit in reporting to management and business and that business and security intelligence includes: (1) trend analysis; (2) anomaly detection; and (3) threat intelligence. (Pironti, 2008, p.1)

Metrics are reported to include those that are 'subjective' and those that are 'objective'. Subjective metrics include those that are "powerful and harmful," those that are 'high risks," those that are "hard to substantiate" and the one cited as the best and worst indicator or that of human intuition. (Pironti, 2008, p.2) Objective measures are those, which are "low risk, supported by data, and able to be recreated." (Pironti, 2008, p.3) Key performance indicators include those which are business aligned quantitative and qualitative measures or the success or failure of "processes, personal, technology, and organizational effectiveness" as well as those which serve to "enable continuous improvement and facilitate effective governance." (Pironti, 2008, p.3)

It is necessary to define what it is that is being measured, what the business value of measurement is and the thresholds that should be established including "positive and negative boundaries, realistic goals and range of values." (Pironti, 2008,p.4) Data for metrics can be gathered through electronic methods and non-electronic methods. Electronic methods include such as system logs, automated system monitoring and sensor networks. Non-electronic methods include such as statistical tracking, human feedback, business process monitoring and business reporting. (Pironti, 2008, p.4)

Business goal alignment includes the defining of required measures and the mapping of business processes to define metrics as well as understanding the motivation for the metrics. (Pronto, 2008, paraphrased) The baseline framework of metrics is inclusive of "people, processes, procedures, technology and compliance" and include value provided vs. The cost including monetary impact, the cost of labor the addition of complexity and the impact on user experience. (Pronto, 2008, p.4 )

Governance metrics are inclusive of employee performance, budget accuracy, and communication capabilities. Stoddard et al. (2005) reports that a key aspect of the information security program is that of 'governance' and that the Corporate Governance Task Force report (CGTF 2004) "includes an information security governance (ISG) assessment questionnaire, intended to be useful to both private and public sectors. The ISG assessment tool focuses on the "people" and "process" components of an information security program and may be useful to some SCADA stakeholder organizations." (Pronto, 2008, p.19)

The Corporate Information Security Working Group (CISWG 2005), building on NIST SP 800-55 and the ISG assessment tool, is reported to have identified "best practices and supporting metrics for enterprise security programs. Most of the metrics take the form of percentages (systems, procedures, personnel) that conform to a given best practice. The CISWG best practices and supporting metrics are intended to be used by (or tailored to) enterprises of all sizes, both public and private sector. The CISWG report identifies an initial minimum baseline set of security metrics based on enterprise size." (Pronto, 2008, p.19) In addition, Stoddard et al. (2005) report that the United States Computer Emergency Readiness Team's (U.S.-CERT) Task Force on Best Practices and Standards: Corporate Governance plans to '…consider cyber security roles and responsibilities within the corporate management structure, referencing and combining best practices and metrics that bring accountability to three key elements of a cybersecurity system: people, process, and technology'." ( Pironti, 2008, p.19)

V. Technological Metrics

Technological metrics are stated to include: (1) the number of security events; (2) the number of patches or fixes deployed; (3) the number of technological vulnerabilities enumerated; (4) the number of media mentions and media types; (5) the cost of incident investigation and remediation; and (6) cost of controls; (7) elapsed time from identification of incident to remediation; (8) the number of attacks identified; (9) the number of policy exceptions requested; (10) the number of policy exceptions granted; and (11) the effectiveness of controls. (Pronto, 2008, p.19)

Stoddard et al. (2005) reports that technical metrics activities are inclusive of: (1) common criteria evaluation assurance level (EAL); (2) NIST SP 800-53; (3) DoDI 8500 2; (4) IDS Comparison Metrics; (5) SAMATE; (6) ISECOM Risk Assessment Values; and (7) OWASP DREAD. (p.16) Major elements of technological metrics include asset classification and control and systems development and maintenance. (p.6)

Tools for technical metrics include: (1) Common Criteria Evaluation Assurance Level (EAL) which is based on Security Standard Assurance and is used to assess level of assurance that security requirements have been met; (2) NIST SP 800-53 which is based on security standard functional assurance and defines the strength of security controls as low, moderate or high; (3) DoDI 8500.2 which is based on Security Standard Function plus Assurance and defines the robustness of security control as basic medium or high; (4) IDS Comparison metrics which are comparison metrics which serve to "enable comparison of IDS products based on performance and other measures"; (5) SAMATE technical assurance providing ongoing effort to define metrics for software security assurance tools; (6) ISECOM Risk Assessment Values technical assurance that defines level of risk associated with a system or application and prioritizes testing level of effort; (7) OWASP DREAD metric technical assurance for definitive level of risk associated with a Web application and for prioritizing the level of effort in assuring its security. (Stoddard, 2005, p. 6)