of 117

Net Assessment as an Information Security Tool

Published on January 2017 | Categories: Documents | Downloads: 18 | Comments: 0



Net Assessment as an Information Security Tool

Submitted in partial fulfillment of the requirements for the degree of Master of Professional Studies in Security and Safety Leadership at George Washington University, College of Professional Studies

By Lorenzo Falciani

Committee Members President of the committee: Frederic Lemieux, Ph.D.
Director of Security & Safety Leadership, College of Professional Studies (CPS) Associate Professor, Columbian College of Arts and Sciences (CCAS)

Research advisor: Silvia Vargas, M.S.C.S., M.S.E.
Professorial Lecturer, College of Professional Studies (CPS)

Committee member: David A. Vargas, M.S.
Professorial Lecturer, College of Professional Studies (CPS) Professorial Lecturer, Columbian College of Arts and Sciences (CCAS)

Washington, D.C. January 31, 2012

Copyright © 2011 Lorenzo Falciani All rights reserved.


I dedicate this work to my wonderful family. Particularly to my beloved wife, Eleanor, who has always been understanding and supportive, showing me every day what is important in life; and to my mother and father for their love and precious example.


I would like to thank everyone who has helped me along the way. My sincere gratitude goes to my research advisor, Prof. Silvia Vargas, for her expertise and support; to Prof. David Vargas, for serving as a committee member and his valuable suggestions; and to the Director of the Program Frederic Lemieux, for his advice and mentorship throughout the Masters program. John, I will always cherish my memories of your Saturday’s lessons. Melinda, thank you for your support; you always had time to help, no matter how busy you were. I would also like to thank all my friends, on both sides of the Ocean, who helped me all along this endeavor. I cannot name you all, but you know how much I treasure your friendship. Half of the precious fruits harvested in this Masters come from you. And yes, Ludovico, I miss our philosophical conversations after an intense day of study. I want also to thank my parents that allowed me to sail away from the old continent and supported me continuously. Above all, I thank my wife, Eleanor. Without her I would not even have begun this adventure. She stood beside me, believed in me, encouraged me constantly, and helped proofreading the research multiple times. Thanks for all your love and help.

Abstract of Project With cybercrime’s yearly revenues estimated at $1 trillion and national security networks probed daily and often breached, cyber security is a vital concern for public and private organizations globally. This research ponders the problem of improving information security by looking at organizations under the lens of Socio-technical systems. The main objective of this study is to ascertain if Net Assessment, one of the principal frameworks for analyzing the United States’ national security strategy, can be used to improve an organization’s capabilities in the field of information and cyber security through its multidisciplinary and holistic approach. To reach this goal, the study identifies and analyzes the aspects Net Assessment offers which are not included in current information security risk management and assessment methodologies. The research follows an inductive process, designed using a qualitative approach which makes use of content analysis, comparative analysis and purposive sampling. The first step drafts a list of current information security risk assessment and risk management methodologies, and then isolates a representative sample. The second step identifies and analyzes existing metaanalytical methods and models devoted to the appraisal, selection, and indexing of risk management and assessment methods. It then defines a categorization framework capable of accurately capturing the different risk management and assessment method’s functionalities. A third step maps the coverage of the two selected methodologies (NIST and ISO) against the identified functionality categories. After a critical analysis of Net Assessment, and a reasoned estimate of its main differences and capabilities in the context of information security, Net Assessment’s features are mapped in the category list. The comparison of the features of the three methodologies identifies Net Assessment as a very credible avenue for improving multiple aspects currently neglected by the International Organization for Standardization (ISO) and National Institute of Standards and Technology (NIST) frameworks. The study results maintain that, if implemented as a companion to these standards, Net Assessment may improve an organization’s security strategy, its allocation of funding for security, provide more accurate estimates of risks and threats, better alignment between potential risks and countermeasures, and improve the inputs for real time and automatic tools.


Table of Contents Page 1. Introduction ............................................................................................................................. 1 1.1 Background ...................................................................................................................... 4 Literature review approach ....................................................................................... 7

1.1.1 1.2

Literature review .............................................................................................................. 8 Information security and its environment ................................................................. 9


Characterization of the system to be protected .............................................................. 11 Information security approaches .................................................................................... 13 1.2.2 Risk management .................................................................................................... 15

Risk management concepts ............................................................................................. 18 1.2.3 Risk assessment ...................................................................................................... 20

Risk assessment concepts ................................................................................................. 21 1.2.4 1.2.5 1.2.6 1.2.7 Vulnerability assessment ........................................................................................ 23 Threat assessment ................................................................................................... 24 Risk mitigation ........................................................................................................ 28 Risk monitoring and evaluation .............................................................................. 29

Risk and situational awareness ....................................................................................... 29 Risk scoring and characterization .................................................................................. 31 1.2.8 Automatic tools and mathematical models ............................................................. 32

Probability based models ................................................................................................. 32 Game theory based models .............................................................................................. 33



Problem statement .......................................................................................................... 34 Research questions .................................................................................................. 35

1.3.1 2.

Scientific Approach .............................................................................................................. 36 2.1 Methodology .................................................................................................................. 37 Theoretical framework ............................................................................................ 39 Conceptual framework ............................................................................................ 40 Research methods ................................................................................................... 43 Analytic design ....................................................................................................... 47

2.1.1 2.1.2 2.1.3 2.1.4 2.2

Data ................................................................................................................................ 50 Data sources ............................................................................................................ 51 Data sampling ......................................................................................................... 54 Output data .............................................................................................................. 55

2.2.1 2.2.2 2.2.3 3.

Discussion ............................................................................................................................. 55 3.1 Part I – Selected information security methods ............................................................. 56 Identification of Characteristics .............................................................................. 58 Characteristics coverage ......................................................................................... 61

3.1.1 3.1.2 3.2

Part II – Net Assessment methodology .......................................................................... 66 Critical analysis of Net Assessment ........................................................................ 67


Definition .......................................................................................................................... 67 Purpose and scope ............................................................................................................ 68 Differences with other assessment methods ................................................................... 69 Main characteristics......................................................................................................... 71 NA process ........................................................................................................................ 72 Value and utility ............................................................................................................... 74 3.2.2 Characteristics coverage ......................................................................................... 75 vii

3.2.3 4.

Net Assessment as a tool for information security ................................................. 78

Conclusions ........................................................................................................................... 83 4.1 4.2 4.3 4.4 4.5 4.6 Observations ................................................................................................................... 84 Extent and limits of investigation................................................................................... 85 Directions for future research ......................................................................................... 86 Implications for practice................................................................................................. 87 Significance of results .................................................................................................... 87 Code of ethics ................................................................................................................. 88

References……………………………………………...…....................................................88 Acronyms………………………………………………………………………………..….105


List of Figures Page Figure 1: Research outline .............................................................................................................. 3 Figure 2: STS structure composition ............................................................................................ 12 Figure 3: Risk Management components ..................................................................................... 17 Figure 4: Security concepts diagram............................................................................................. 19 Figure 5: Areas of research involved ............................................................................................ 38 Figure 6: Information systems research framework ..................................................................... 42 Figure 7: Inductive, bottom up, approach to identify covered areas ............................................ 44 Figure 8: Document analysis process ........................................................................................... 45 Figure 9: Deductive, top down, approach to identify possible benefits........................................ 46


List of Tables Page Table 1: RM and RA methods general characteristics.................................................................. 58 Table 2: RM and RA methods technical characteristics ............................................................... 60 Table 3: NIST general characteristics coverage ........................................................................... 62 Table 4: NIST technical characteristics coverage ......................................................................... 63 Table 5: ISO general characteristics coverage .............................................................................. 64 Table 6: ISO technical characteristics coverage ........................................................................... 65 Table 7: Net Assessment general characteristics coverage........................................................... 76 Table 8: Net Assessment technical characteristics coverage ........................................................ 77


1. Introduction
Technology has developed and advanced in the last decades at a staggering pace, and the technological infrastructure on which both government and businesses rely today is pervasive. As an example, by 2010, the U.S. Department of Defense already had about 90,000 personnel working on its 15,000 networks (McMichael, 2010). However, despite our recognized reliance on this technology, information systems and their networks continue to be threatened by a number of security risks with very tangible implications. In 2011, the Chief Security Officer of AT&T testified before US Congress that cybercrime yearly revenues were estimated to be $1 trillion, almost to 2 percent of the global economy and larger than the entire pharmaceutical industry (Friedman, 2011). This research paper addresses the topic of information systems security, focusing especially on IT security practices. The aim of the study is to analyze whether Net Assessment can provide additional insight to private companies and public organizations, and if it can be used as an additional tool to improve the effectiveness of their security efforts. Net Assessment is a process that affords a multidisciplinary and holistic approach to security assessment and promotes awareness and understanding at the highest level of any organization, enabling informed strategic decisions. Assessing the efforts of both the public and private sectors to secure their technological infrastructure, it becomes clear that most of the budget and resources dedicated to information security have been focused on identifying and analyzing vulnerabilities; and the remainder has been spent on mitigation and recovery measures. Analyzing and addressing vulnerabilities is an important step of the information systems’ security lifecycle, however, it can no longer be considered the only or the preponderant protective measure to be adopted to further improve the security of information and cyber systems. Vulnerabilities in cyber systems will always exist and this characteristic gives an attacker substantial advantage over its target. Today, to improve the security of information systems, more effort and attention should be paid to threat assessment, especially concerning threat agents and threat vectors, in order to better choose and prioritize countermeasures to be implemented; 1

attain a clearer situational awareness; and collect invaluable information for recovery. Some organizations are already pioneering the practice of performing cyber threat characterization; however, open source literature suggests that such practice is neither widespread nor well structured. Further, as information is increasingly managed in digital formats, data security now transcends the cyber domain, and if not properly maintained it can negatively affect people’s lives, through identity theft for example, or a business’ success through the theft of intellectual property, or public safety should critical infrastructure systems not be adequately protected. This research argues that more and better threat assessments are required to improve information systems security and that these improvements can be achieved at the organizational level. Net Assessment (NA), already in use by the US Department of Defense for strategic purposes, could be well suited also for other non-defense agencies and the private sector as a tool for ultimately improving the security of their information systems. Throughout the research when the terms ‘security’ or ‘risk’ are used alone for readability, they should be intended respectively as ‘information security’ and ‘information security risk’, while the term ‘cyber information security’ should be intended as a narrower, more delimited concept which, together with IT, is included in the former definition. Existing ambiguities in relevant technical terms definitions are addressed in the literature review. The research follows the logic flow depicted in the figure below.


Figure 1: Research outline

The research is organized as follows: Chapter 1 introduces the topics addressed in the study, and presents a comprehensive background research in the fields of interest and on the advancement of practice and research. Moreover, the chapter identifies the specific problem and defines a number of research questions. The objective of the introduction is to set the stage for the research, identify the main relevant concepts, and address the broader scholarly literature. Chapter 2 details the research approach. The chapter aims at clearly defining the adopted research methodology, including the theoretical framework and its operationalization. Applied research methods and analytical design are described in detail. Furthermore, the sources and sampling for the data used in this research paper are presented and described in detail. The 3

objective of the chapter is to provide the reader a description of the research methodology, and to give evidence of the application of a scientific method. Chapter 3 contains the discussion of the identified problem. The inquiry involves a critical analysis of information security methods currently in use and of Net Assessment. Such analysis is followed by a comparative analysis of Net Assessment characteristics and those of security risk methods previously addressed. The objective of this chapter is to assess the collected data in order to answer the questions identified in chapter 1.3.1. Chapter 4 summarizes the findings of the research. The research significance, scope and limits are identified with a number of implications for the improvement of information and cyber security. Furthermore, a number of directions for future research are laid out.

1.1 Background
The technological revolution of the last half century has deeply impacted our lives, strengthening our dependency on automatic information and communication systems. As more and more of our government services, critical infrastructure, and businesses operate with common off the shelf (COTS) tools; our dependency on technology can be translated in a serious concern, with potential impact ranging from national security to mass job loss. Current information systems represent a vulnerability for private citizens, organizations, and the government. The threat of disruption to our critical infrastructure is particularly alarming as it is exacerbated by several factors such as being online “24/7/365” and its extensive interconnectivity, making it a possible target of attacks from everywhere without a secure way to identify the threat agents. Furthermore, threats to financial systems, private intellectual property, and fair business increment the plethora of nationwide problems and prospect more worrying future scenarios. This means that our constantly connected, time-sensitive society and our infrastructure, be it energy transmission, communication, transportation, administrative, financial or business, are highly dependent on one tool: Interconnected networks of computers that offer the ability to create, collect, process, transfer, and store digital data in a secure manner. Our own national security agencies, the entities that protect our safety and security, directly rely on these networks,


taking advantage of Socio-technical systems that include cyber tools to accomplish their missions. Hence the question, does technology represent a concrete risk? Once we admit that not only our services, but also our safety and security rely and depend on the proper functioning of technological apparati, we begin to appreciate the fact that there is a potential risk. In fact, we realize that we consider ourselves safe and secure on the assumption that such systems perform correctly. Experts are still profoundly divided in their assessment and forecast of the potential impact of future information systems’ security breaches. Some experts maintain that cyber-attacks with potential strategic national security effects are impossible; others advocate that they are inevitable. Experts considered optimists often cite the robustness, resiliency, and redundancy of the Internet (Billo & Chang, 2004). James Lewis of the Center for Strategic and International Studies (CSIS), for example, argues that a cyber-attack will never lead to mass casualties and is not comparable to weapons of mass destruction (WMDs). On the other hand, experts with access to classified sources like the former White House Cyber Security Advisor, Richard Clarke, argue that the proper functioning of our technological infrastructure is one of the reasons for the U.S. economic and military domination of the world, and it is necessary to maintain political power and safeguard people. A third set of observers maintains that the truth lies somewhere in between, and that it is best to be vigilant. In order to guarantee that entities critical to our security and safety, whether public or private, perform their mission according to acceptable quality and efficiency standards, the technological systems they rely upon must be secured. Whatever the future will hold, it is a fact that information systems, and especially those that are interconnected through the Internet, cannot yet be considered secure. It is a fact that despite the increasing efforts in terms of advanced research and implementation of adequate measures (technical, organizational, legal, financial, etc.), sufficient security has not been achieved. It is a fact that in 2011 U.S. Congress was considering more than 20 bills on cyber security, in addition to proposed legislation from the White House (Friedman, 2011), to help secure the cyber-space for both public and private benefit. It is a fact that hackers attempt to map the IT networks of Fortune 500 companies on a daily basis (Harris, 2008). In the defense field it is safe to say that information technology enables almost everything the U.S. Military does (Hernandez, 2010),


while on the civilian side, as pointed out by Andrew Cutts from the Department of Homeland Security (DHS): “The most consequential and highest risk threat is attack by one or more nation-states intent on projecting power, and who are willing to damage or destroy critical information infrastructure by cyber means in order to achieve this objective. Threat actors falling into this category have the necessary time, resources, sophistication, and access to do so” (Cutts, 2009).

The budget allocated for advancing the security of governmental agencies, critical infrastructures, foster public-private partnership, and implement the national security strategy is enormous. While the potential of a cyber-attack has been thoroughly briefed to U.S. policy makers and has found a place in every agency and sector of intelligence and national security and defense, the question of whether this budget has been properly allocated and is producing the expected results is still debated. Marcus Sachs from DHS says that the complexity of computer networks is growing faster than our ability to understand and protect them (Parker, Sachs, Miller, & Devost, 2011). It is also under debate how big and small businesses can enhance their security posture, through additional legislation, public-private partnership or by other means. Today risk management is disproportionately based on vulnerabilities versus threats. This approach leads to a series of problems with the result that, despite the huge budget invested in cyberspace, it is still not secure. While it is of the utmost importance to be aware of vulnerabilities and to protect them, it should also be recognized that “differentiation of adversaries is essential to security planning and incident response” (Parker et al., 2011). Parker and many other practitioners stress that the goal is an effective defense, not a perfect one. To get ahead of the most serious cyber security risks, countries’ and organizations’ cyber security leadership must seek an appropriate balance of resources, energy, and focus between those threats that are most frequent and those that are most consequential (Cutts, 2009) and (Volk, 2010). 6

This research paper focuses on businesses, organizations or government entities which derive their main value from information. It studies the non-technical aspects involved in implementing and strengthening the security of information systems. After analyzing current security risk management methods and their area of coverage, this study will assess the potential merit of Net Assessment as an additional tool for improving information security. Net Assessment is one of the principal frameworks for analyzing the national security strategy of the United States, and usually any problem involving competition, in a Red vs. Blue fashion (Bracken, 2006). This inquiry thus, will take a look at Net Assessment to see what insights it can bring in the cyber realm. It has already been used in some studies, such as the “Social Software and National Security - An Initial Net Assessment” (Drapeau & Wells, 2009) which tackles the potential security problem created by the use of social software by government employees. It is proof that NA can also be used in non-military contexts with results. 1.1.1 Literature review approach The background research has been performed using full access to two main resources: • • the George Washington University (GWU) library system, and the unclassified part of the Homeland Security Digital Library (HSDL) of the Naval Postgraduate School (NPS) – Center for Homeland Defense and Security. The GWU library alone allows access to an immense quantity of titles and provides sophisticated instruments of documentation management, research and retrieval. GWU’s access to the Surveyor catalog includes millions of titles. The Washington Research Library Consortium (WRLC) member institutions are: George Washington (Estelle and Melvin Gelman Library, Jacob Burns Law Library, Eckles Library at the Mount Vernon Campus, Himmelfarb Health Sciences Library, and the Virginia Science and Technology Campus Library), American, Catholic University, University of the District of Columbia, Gallaudet University, George Mason University, Georgetown University, and Marymount University. Furthermore, the Homeland Security Digital Library (HSDL) holds almost 100,000 unclassified documents, and it is the US’s premier collection related to homeland security policy,


strategy, and organizational management with sponsorships by the DHS, the Federal Emergency Management Agency (FEMA) and the NPS. In order to perform a comprehensive background research on the topics of interest, three main leads have been used: • • • consolidated authorities: Authors and milestone books on the fields touched by this research; applicable regulations, laws, best practices and well established methodologies; keywords related to the fields under scrutiny.

The resulting review of literature spans from the 1980s’ up to today. While information systems’ security is an important topic since well before, it has to be acknowledged that it is in the 1990s’ that the information security problem begins to be approached substantively in public literature at the management and organizational level. While the evaluated time frame begins only about 30 years ago, it can be considered as entirely representative of the subject. The background research has been performed through a comprehensive literature review, which led to the collection of data to be used as input for the discussion of the problem, and to the identification of the theoretical framework for this research. Digital copies of the identified work, when available, have been collected and managed through an automatic tool – Qiqqa, which was used also to conduct a more advanced analysis of the relevant literature, including cross reference checks, theories comparison and brainstorming to refine initial ideas. Input data sources, data collection and sampling methods are described in more detail in Chapter 2.2.

1.2 Literature review
Through the literature review it is possible to contextualize the topic under scrutiny with respect to the broader field of information security and to identify the current state of practice and research. The presented review is a critical synthesis and evaluation of relevant literature according to the focus of the present research. Randolph’s “A Guide to Writing the Dissertation Literature Review” (Randolph, 2009) based on the milestone research by Boote and Beile (Boote & Beile, 2004, 2005) has been taken into account while performing the literature review.


Among the millions of titles in the available databases, the identified research threads generated tens of thousands of results related to the research’s topic. A manual inspection of those results, taking into consideration three factors – author, title and type (book chapter, journal, book, etc.) – led to a refinement of approximately a thousand of related references. The refined results were processed through a more detailed review, which included also their respective keywords, abstracts, introductions, conclusions and references. Typically the reviewed titles had at least three of the above characteristics. This further review identified a core of about 300 titles. Each of those titles was analyzed, and some were included in the literature review because of their significance for the present inquiry. Works included in the literature review conformed to a number of characteristics: • • • Are currently applicable or had applicability in the history of the field; Are relevant to the broader field of research, to the specific topic and/or to the research method; Bear scientific or practical significance, and have solid foundations.

Works and documents that did not match all these three characteristics at any one time of the three process reviews previously described, were excluded. The literature review is organized in a number of chapters that address the field of inquiry starting from its origins towards modern and contemporary resources. It also encompasses security research and practice starting from a higher abstraction level and ends with the lower abstraction level components. This approach allowed for a comprehensive analysis of the current status of information and IT security and led to the identification of specific problems and research questions in a top – down, deductive approach. The research questions are presented in chapter 1.3. Subsequently, the opposite approach (inductive) has been used for the discussion outlined in chapter 3 to answer the identified research questions. The following chapters are dedicated to the literature review. 1.2.1 Information security and its environment In 1989, Professor Lance J. Hoffman, a pioneer and scholar that dedicated much of his work to computer security, published an important article in ‘Computer and Security’, a seminal IT security journal at that time as well as today. The article, Risk analysis and computer security: 9

towards a theory at last (Hoffman, 1989), was a perfect snapshot of the state of computer security: “Right now computer security risk assessors and most computer security risk assessment packages have their own dogma. There is a tendency to argue about which methodology is “best”, and a tendency to avoid looking at or measuring the effectiveness of any specific one”.

In 1993, in the very same journal, Eloff and colleagues published a research paper driven by the need for clarity in information security (Eloff, Labuschagne, & Badenhorst, 1993). The research selected three information security risk analysis methods (CRAMM, LAVA and MELISA), claimed that terminology was used inconsistently, and tackled the arduous challenge of standardizing the concepts and terminology used in those three methods. Their work is one of the earliest of its kind. While the field has evolved significantly since 1993, inconsistency of terminology is still an issue. The number of active authors in the field of information security risk management and assessment is staggering, with whole journals devoted entirely to this topic. In order to better appreciate the available security assessment methods and how they are selected and used, it is necessary to address first their main driver, the concept of security. Security has been given many different definitions depending on the viewpoint; such definitions can be divided in two main classes: (1) Security as a process and, (2) Security as a property. Security has been defined as a process in order to highlight the fact that security should be a concern during the whole lifetime of a system and not only in a specific phase (design, implementation, operation, etc.). For example, Kizza says that “Security is a process. Security assurance is a continuous security state of the security process” (Kizza, 2005). However, security has been defined with good reasons also as a system property or characteristic: “A condition that results from the establishment and maintenance of protective measures that enable an enterprise to perform its mission or critical functions despite risks posed by threats to its use of information systems. Protective measures may 10

involve a combination of deterrence, avoidance, prevention, detection, recovery, and correction that should form part of the enterprise’s risk management approach” (Kissel, 2011). Without entering the debate around the definition of security, it should stand clear that the present research, while subscribing to both viewpoints, will use the security terminology as defined by the National Institute of Standards and Technology (NIST). In other words, information security will be considered a system property that can be attained by maintaining the system in such a way that it cannot be controlled by others or, from a different perspective, maintaining the system and its data in such a way that they always satisfy the security policies. Measures to achieve and maintain the security property of a system are part of overarching risk management activities. However, before considering such methods, it will be beneficial to first address what needs to be secured. Characterization of the system to be protected The object of our interest in information security is the security of computers, networks and interconnected technological systems in general, and the data processed by them. However, technological systems are inserted in an environment in which also humans operate. While some of the most disastrous security problems are driven by natural conditions, indeed the most frequent security problems of technological systems come from malicious attackers or negligent users, either external or internal to such systems. It would be limiting to consider only the technological components of a system, neglecting the human components. Considering information systems as Socio-technical systems (STS) affords us a wider and all-inclusive analysis because it considers not only technical issues but also the social aspects of the environment, both internal and external, in which the system operates. An STS can be considered as a complex network of relationships between humans and technological systems including users, stakeholders, regulations and policies, hardware and software, and data (Ropohl, 1999). Systems theory dates as far back as the 1950s’ (Bertalanffy, 1950 and 1950b; and Boulding, 1956). Subsequent improvement of the theory, together with the evolution of systems, led to the formulation of the Socio-technical system paradigm by Trist in the 1960s’ and then further refined afterwards by him and others (Trist, 1981). 11

It is important to define the system to be protected as an STS as too often it is thought that information or cyber security is a parallel, virtual reality almost disconnected from the tangible world. Instead it should be recognized that (1) cyber issues can be exacerbated by physical threats, and that (2) cyber security cannot be measured without taking into account also the human element. As Reijo Savola pointed out in his research, security metrics should be as objective as possible, however “in reality, as security contains a lot of human behavioral aspects, many metrics tend to be highly subjective” (Savola, 2007). But the human interaction cannot be relegated only to the behavioral aspects. The research by David Botta and colleagues (Botta, Muldner, Hawkey, & Beznosov, 2011) is a clear example of how scholars recognize the need to also understand and evaluate the cognitive aspects that influence IT security management. People are an integral part of the cyber system, both on the internal and external sides, as both data owners and attackers. Therefore, this inquiry will adopt the STS viewpoint and study the system under the aegis of the Socio-technical theory. STS’s structure and main components are identified in figure 2, which also presents their association.

Figure 2: STS structure composition


In the defense field, the military already recognized the necessity to embrace and manage the human part of security systems. The STS approach is embedded and coded in multiple aspects. One such aspect that is related to this research is that of network centric warfare (NCW). While typically models and analyses of NCW focus on technological aspects of a system, eschewing the roles, contributions and decisions made by humans, there are also attempts to represent relationships between technology and humans in a system (Miller & Shattuck, 2006). While not as explicitly, security risk strategies employed by private entities also take advantage, up to a certain extent, of the conception of Socio-technical systems as a whole. The identification of security flaws such as conflict of interests and their respective security measures – such as in this case the segregation of duties, where a task shall be completed through multiple actions performed by different actors, are evidence that indeed the STS aspect is somehow recognized in the private sphere as well. Recent research sponsored by the U.S. Department of Homeland Security identified and assessed information security risk perceptions, utilizing previous psychometric models that include understanding and consequences as two main characteristics for evaluating the perceptions of information security risks (Farahmand, Dark, Liles, & Sorge, 2009). Furthermore, previous research by Knapp demonstrates how important the human factor is in the development and implementation of a security risk management effort (Knapp, 2005). In sum, we really cannot leave the human factor out of the equation if the topic is information or cyber security. Information security approaches The panacea of cyber security would be to simply build completely secure systems. Imbuing security in a technological system since its inception, in the design phase, would doubtless be a great benefit and is a prime objective for modern design methodologies (Soudain, Raggad, & Zouari, 2009). However, despite this objective, secure design of systems is neither perfected enough nor fully applicable to systems as complex as STSs. Therefore, security remains an important dimension also during the system’s operational phase. Today IT security and information security in general are pursued by organizations through two major distinct approaches:


• •

Security as policy compliance; Security as risk minimization.

The two approaches differ substantially with respect to their goals, their costs, usability of the system, and their results. The main objective of the ‘security as policy compliance’ is to protect an organization from the legal perspective, but not necessarily to minimize its security risk. Its secondary objective is to minimize the cost of security rather than the cost of potential loss due to security incidents, and it is mostly implemented in the commercial sector. While government’s fundamental role is to provide common defense, industry's role is to make profits. Both use cyber means to collect, process, manage, and use information; however the mission and objectives of cyber security are quite different (Clinton, 2011). The second approach is aimed at minimizing the security risk of a system, or the information hosted on the system, and it is traditionally implemented in systems related to national defense or national security. However, the number of commercial organizations choosing this approach is increasing, as well as the number of private users of technological systems. This study is dedicated to entities that pursue this second approach. Security through risk minimization can be implemented in information security systems (ISS) through a number of different methodologies, depending on the specific requirements and situation (Siponen, 2005): • • • • • ISS Checklists, ISS Formal Methods (FM), ISS Standards, ISS Maturity Criteria, ISS Risk Management (RM).

These methodologies are not mutually exclusive and can be used to augment each other. Furthermore, ISS standards may contain methods belonging also to other classes. A recent, well supported analysis of all the five classes has been performed by Siponen, who concludes that ISS checklists are the most limited methodology available: “ISS checklists assume that ISS solutions and procedures can be observed and turned into a list: a checklist” (Siponen, 2005). This assumption is very strong and usually not fully applicable in most organizational environments,


which is why checklists are nowadays used only as complementary tools of more comprehensive security methods, especially for providing a way to ensure completeness in security auditing. ISS formal methods (FM) assert that a system, in our case a Socio-technical system, should be completely based on formally validated components or developed using only FM. The two main assumptions of this methodology are that (1) logic is suited to ensure that a system is secure, and that (2) it is feasible to follow this approach at the overall system level. The systems we are discussing in this study are dynamic, complex Socio-technical systems which thus include also the human element, hardly to be considered as a deterministic component, thereby adding to the complexity of the system. Furthermore, they are preexisting (not in development stage), again adding more burden to the complexity of the model. Formally proving the full security of such a system in its wholeness with logic instruments is absolutely impractical (Williams & Abrams, n.d.). In reality, formal methods have been successfully used only in proving security properties of subsystems, and are best suited only to the study of relatively small, abstract issues. The ISS maturity criteria is a class of security tools formulated to provide metrics and procedures for classifying ISS maturity. While recognized as a tool that can aid in improving security in a system, it is a mechanism that provides only feedback and cannot be considered a proactive security method. While definitely an important asset in a security program, it is mainly supportive of a more comprehensive risk management effort. For these reasons, the present inquiry focuses on ISS risk management, as the most complete and comprehensive class of methods for implementing information security in Socio-technical systems, which in fact may include and implement, upon necessity, all the other classes. 1.2.2 Risk management Considering that most of cyber and information security terminology is still affected by several ambiguities, a brief discussion on the basic terms adopted in this inquiry is needed. The objective is not to resolve any ambiguities in definitions, but to point out the specific meaning used in this study. Risk management is used to indicate the process of establishing and maintaining information system security within an organization (Wright, 1999) with the goal of lowering the residual risk 15

to an acceptable level, also called ‘risk tolerance’ or ‘risk appetite’. Information security risk management methods are aligned with STS theory, as their purpose is to protect the organization and their ability to perform their mission, focusing not just on the IT assets, but on all involved entities (Stoneburner, Goguen, & Feringa, 2002). Different risk management methods include different components, so while they follow a common structure, it is impractical to identify detailed characteristics. However, it is important to define at least some of their major conceptual components, in order to demarcate the area of coverage. Risk management methodologies thus encompass different processes but, no matter how they are defined or organized, will include at least the following conceptual elements (for example see (IWG, 2010) and (GAO, 1999)). • • • • Risk assessment, which includes identification and evaluation of organizational assets, potential risks and risk impacts, and recommendation for risk-reducing measures; Risk mitigation, which refers to prioritizing, implementing, and maintaining the appropriate risk-reducing measures recommended by the risk assessment process; Continual system monitoring and evaluation, including the assessment of implemented security measures, Risk awareness promotion among managers at strategic level, among users, stakeholders and security operators at a tactical level. The four different and complementary element of Risk Management are presented in figure 3.


Figure 3: Risk Management components

One of the major activities, the risk assessment (RA), also directly influences all the others. It is also the activity which relies mostly on interpretation and is highly dependent on the views of the implementer. Risk assessment tackles a number of peculiar difficulties, ranging from the assignment of a value to every part of the system, including information, to the analysis of yet unknown threats. Controversy or at least misalignment exists around the use of terms ‘risk analysis’ and ‘risk assessment’. This research adopts the definition sponsored by NIST which defines risk assessment as: “The process of identifying risks to organizational operations (including mission, functions, image, or reputation), organizational assets, individuals, other organizations, and the Nation, arising through the operation of an information system. Part of risk management, incorporates threat and vulnerability analyses and considers mitigations provided by security controls 17

planned or in place. Synonymous with risk analysis.” (Kissel, 2011). The first requirement for all U.S. federal government agencies to apply risk analysis techniques to computer systems dates back to 1978, and was set by the Office of Management and Budget (OMB) (Guarro, 1987). The requirement has been reaffirmed and updated multiple times since then, and now there is an entire framework concerning computer security and its management (RMF) developed by the NIST. Just as risk management methods differ in approach and coverage, risk assessment methods also vary in nature and depth. Their selection and application for the evaluation of cyber security issues is decided by the security staff on the basis of their capability to provide meaningful insight for the design and implementation of security measures in specific information systems. We have identified a precise foundation for the technical concepts of security, security approach, security methodologies, risk management, and pointed out how the technical terminology will be used in this context. Therefore we can proceed with the analysis of the four components or risk management methods and their characteristics. Risk management concepts The following is a list of concepts recurrent in information security risk management and assessment that must be clarified. The concepts illustrated in this list may be referenced with different terms in different risk assessment methods or standards (e.g. NIST and ISO) or may not be present at all, hence the necessity to indicate the meaning that such terms will bear in this study. The elements presented in the list and their relationships are also depicted in figure 4. • • • Risk will be intended as the combination of the probability of a threat and its potential consequences on one or more assets. Residual risk will be used to intend the remaining potential risk after all the security measures are applied. There is a residual risk associated with each risk. Security measure, countermeasure, security control, and safeguard, will be all considered synonyms and used to identify any action, equipment, procedure, or technique that reduce


a vulnerability or degrade completely or partially a threat through one or more of the following ways: o Prevention o Detection o Correction o Recovery o Deterrence o Compensation • • • • Threat will be used to refer to any event with a potential adverse impact on organizational operations, assets, or external stakeholders. Threat agent will be used to identify the potential source of an adverse event. Asset will be used in the research to indicate any tangible or intangible resource of an organization, including software, information, equipment, and personnel. Vulnerability will be considered as a weakness in the information systems, procedures or controls, that could be exploited or triggered by a threat source and allows a threat to cause harm.

Figure 4: Security concepts diagram


1.2.3 Risk assessment As previously pointed out, while RM methodologies mostly share the same objectives and cover similar areas, they are composed of different internal processes. While all of them include a risk assessment process, they may include different activities and objectives as different RM methods achieve the same goals through different processes. A number of risk management methodologies are being used in different STSes, and each of those, in every implementation, makes use of a different risk assessment method in order to tailor the evaluation to each individual situation. Today the number of information security risk assessment methods is huge. While the vast majority of those methods are proprietary and developer companies do not disclose much information other than what is present in their advertisement brochures, this should not be seen a major impediment or concern to our research for a number of reasons: • • • There is no evidence that such methods work better than publicly available ones; The public results presented through research journals are diverse and span on many areas, covering plenty of direction for this type of research; The proprietary methods are used either in-house by the developer companies or externally for consultancy work, therefore the use of such methods is very limited with respect to the most used public ones. Review of scholarly and selected practical documentation led to the identification of many risk assessment methods and frameworks currently in use. Despite a very extensive research in academic, governmental, commercial, public, and open resources it has been impossible to find comprehensive usage statistics over such information security management and assessment methods. However, ENISA, the European Network and Information Security Agency, in 2005 established a working group dedicated to the identification and assessment of information security risk management and assessment methods. ENISA lists the requirements for methods to be included (ENISA, 2006) and they are compatible with the present research. ENISA claims that their inventory of methods is not exhaustive, and that a separate and independent background research identified several other methods in use not listed by ENISA and still matching the stated requirements. The full inventory of methods currently in use created during the background research includes a total of 22 methods and is listed in chapter 3.1.


Historically, methods of information risk assessment and management are neither derived by empirical testing nor by a formal scientific base. Most of the well-established and also of the newly proposed methods employ some of the concepts used in computer science, engineering, safety, economy, and other fields but they are usually driven by conceptual reasoning and corroborated by theoretical ad-hoc case studies or other methods with unclear validity, rather than proper scientific method, leaving such proposals quite unsubstantiated on their effectiveness (Verendel, 2009). However, it is also worth identifying risk assessment methods not yet in widespread use, as they could include enhancements or improvement over those already identified. A further review of relevant scholarly literature led to identification of a number of novel security assessment approaches which can be added to the list of already identified risk assessment in use: ARIMA (Leitner & Schaumuller-Bichl, 2009), AURUM (Ekelhart, Fenz, & Neubauer, 2009a), CORAS (Aagedal et al., 2002) and (Lund, Solhaug, & Stølen, 2011), ISRAM (Karabacak & Sogukpinar, 2005), LRAM (Guarro, 1987), Risk Analysis and Management for Critical Asset Protection (RAMCAP) (ASME-ITI, 2006), and Value at Risk (VAR) (Jaisingh & Rees, 2001). Risk assessment concepts Before we pass to analyze those methods, it is first necessary to acquire the instruments to evaluate them. Risk assessment methods are under continuous scrutiny by scholars, who try to build upon them or challenge their assumptions to improve the effective results. A further review of relevant scholarly literature led to the identification of numerous comparative analyses of such methods employing many different comparison frameworks for their assessments. Risk assessment methods can be divided in four major approaches or categories (Tregear, 2001), (Vidalis, 2004): • Quantitative approach. This mathematical approach quantifies the amount of potential loss (Conrad, Oman, & Taylor, 2006). It is time consuming and expensive. It also involves as a necessary step an exact quantification of hard to quantify variables, like the loss of information; and it is in fact critiqued by many (Verendel, 2009) which argue that reliable quantitative measures are not attainable in almost all the cases.


Qualitative approach. Implemented by most of the available standards, this approach is simpler than the quantitative; it involves less uncertainty and aims at producing estimates of potential losses (Rot, 2008). It is the most widely used approach today.

Knowledge-based approach. Created at the rise of the information security, this approach involves reusing best practices from similar systems. While some recent efforts are taking it into consideration (Takahashi, Kadobayashi, & Fujiwara, 2010) and (Ekelhart, Fenz, & Neubauer, 2009a), this approach is almost unused now.

Model-based approach. This approach uses system modeling theory to describe and analyze the risk. More recently, it makes use of object oriented models (Sommestad, Ekstedt, & Johnson, 2009), like CORAS (Aagedal et al., 2002). There are multiple types of model-based approaches, however, the most used (Al-Hamdani, 2009) are: o Means-end based models; o Risk based models; o Benchmark based models; o Diligence based models.

In order to further classify risk assessment methods and to better compare risk assessment methods and risk management methodologies, various authors developed comparison frameworks, models, list of characteristics, and factors. The most interesting works have been collected and listed in chapter 2.2.1, dedicated to data sources. The identified comparison frameworks referenced in chapter 2.2.1 include structured quantitative approaches, qualitative analytical models, unstructured functional analyses, and lists of characteristics derived by qualitative conceptual analysis. The focus of such comparison efforts are disparate, from the analysis of whether a risk assessment is conducted with internal resources or mainly external consultants, to the analysis of how data is collected, to the presence of specific functions like awareness raising or threat evaluation. The soundness of the comparison method also varies greatly from study to study. While those cited above are indeed the most interesting and reliable research found during the background research, still not all of them follow a completely rigorous approach. The identified comparative studies have been conducted with different foci than the present inquiry, and cannot be fully endorsed for this 22

research purpose; however, their framework methodologies have been evaluated and taken into consideration for the development of the categorization framework used for this research. 1.2.4 Vulnerability assessment Hackers, organized crime, terrorists, and nation states are all using the internet and public networks to their advantage. Information and information systems are at risk and both defense posture and civilian businesses can suffer detrimental effects if they are not properly secured. Vulnerabilities are not intrinsic properties of a system, but they can be considered as such, so long as they can be exploited by a potential threat, and exercised by an existing threat agent. The current national security strategy is focused on fixing vulnerabilities instead of on threats (Parker et al., 2011). A recent research paper published in 2011 (Nath, 2011) analyzes the most common vulnerability assessment methods, from those aiming at low level (binary code) vulnerabilities as well as those aiming at high level vulnerabilities, up to the virtual machine (VM) level. The conclusions establish that there is a lot of confusion on the usage of such methods, and that researchers in the past focused too much on low level methods, thereby neglecting high level but still very important vulnerabilities. Among many research efforts focused on risk reduction and vulnerability analysis, it is notable the proposal of a type of i* based model that defines a vulnerability centric, goaloriented, framework for the specification of requirements to be used in the design phase of information systems (Elahi, Yu, & Zannone, 2010). The research proposes the use of a qualitative goal means-end model for assessing the risks associated with vulnerabilities exploitation, and includes the analysis of the impact of selected countermeasures on such risks. While the approach seems clever, it is difficult to imagine its use in a different context than that of requirements engineering, as higher level applications will be too complex to be manageable. And even if successfully used during the design phase, it can only ensure a better coverage of known issues, and be of no use during the lifecycle of a complex system as requirements can rarely be changed in complex systems. The same critique is valid for many other studies in the same field such as that by Satoh and Kumamoto (Satoh & Kumamoto, 2009).


Remedying existing and future vulnerabilities is important of course, as all the potential points of failure have to be protected up to a certain level. However, there will always be new vulnerabilities in Socio-technical systems, and there is the need to identify ways of minimizing the risk without intervening directly on the vulnerabilities. Considering that potential vulnerabilities will never be all mitigated in a complete and timely manner, it is necessary to consider also the threats and threat agents in order to fully evaluate their bearing towards a risk measurement. Other factors are also very important, such as the development of attack capabilities, and a better situational awareness of threats in the cyber realm. 1.2.5 Threat assessment One of the most advanced studies on security evaluation of a system is “Adversary-Driven State-Based System Security Evaluation” (LeMay et al., 2010). This research paper defines a quantitative evaluation method to measure the robustness of a given security system. The input factors are security features of the system and characteristics of the adversaries attacking the system. The model makes use of a stochastic process to identify attack scenarios and resulting impacts on the system, however there are many limitations. First, the method doesn’t account for safeguards and their effect on the system. Second, while the model is apt to calculate the probability related to the threat-unknowns, it still expects precise, quantitative, inputs regarding threat characteristics. Finally, while the model can be applied at different times, thus covering the system lifecycle, it is not a ‘live’ process; rather, it is a snapshot that can be taken multiple times, not an instrument that lives with the systems. A team of Swedish researchers presented at West Point in 2007 a rationale for and assessment necessary to understand why, where, and when capabilities required by methods and tools for IT security are needed (Hallberg, Hallberg, & Hunstad, 2007). This peculiar view of the security aspects led the authors to identify that the knowledge, rationale and needed capabilities constitutes a foundation for future development of methods and tools regarding IT security assessment. Among those they recognize the need for better threat characterization. In “Information Security Threats: A Comparative Analysis of Impact, Probability, and Preparedness” Sumner studies and analyzes the topics of impact, probability and preparedness, in relation to the cyber threats. The research focuses on the “perception” of those characteristics 24

and is based on a questionnaire (Sumner, 2009). While the topic is indeed relevant for the present research, the fact that such aspects have been assessed through the mediation of perception, it cannot give further insight in more general analyses and thus cannot be used as a foundation for the present study. “In the cyber realm, offense always dominates and always will. It is structural and axiomatic” (Crosston, 2011). Professor Dorothy E. Denning of Georgetown University and NPS, one of the most prominent information security researcher and specialists, argues that (cyber) threats are a function of three factors: intent, capability, and opportunity (Denning, 1999). And such an approach is also recommended by the MITRE Corporation: “An organization determines its target level of preparedness against cyber threats, including the advanced persistent threat, based on its assessment of the level of the adversary it faces. That is, an organization calibrates its cyber security measures, as well as its cyber security governance, to its cyber threat” (Bodeau, FabiusGreene, & Graubart, 2010). Many studies have been performed which take into account both cyber threats and cyber threat agents, especially those considered most pernicious. An example of threat agent assessment publically available is the DHS’s “Leftwing Extremists Likely to Increase Use of Cyber Attacks over the Coming Decade” (DHS, 2009), which “…examines the potential threat to homeland security from cyber-attacks conducted by leftwing extremists”. Other examples can be found in the hearing of Pat Choate, Richard Fisher, the Honorable Edward Timperlake, and Adam Segal before the Subcommittee on Oversight and Investigations (CFA, 2011) which addresses the Chinese funding of cyber warfare budgets, including the sums allocated for cyber espionage and offensive capabilities building. However such reports do not include a detailed description of how they have been performed. There aren’t many open source assessments of threat agents and their capabilities, willingness and opportunity to perform cyber-attacks. Two of the most complete that are available as of today are from Sandia National Labs (SNL) and Northrop Grumman (NG). Sandia’s report focused on a threat analysis framework based on five key elements: (1) adversary 25

identification, (2) development of threat profiles, (3) identification of generic attack paths, (4) near-real time monitoring, and (5) risk mitigation methods selection (Duggan & Michalski, 2007). The second referenced report is a research by NG: “Capability of the People’s Republic of China to Conduct Cyber Warfare and Computer Network Exploitation” which encompasses five broad categories to show how the People’s Republic of China (PRC) is pursuing computer network operations (CNO) and the extent to which it is being implemented: “a) The PLA‘s strategy for computer network operations at the campaign and strategic level to understand how China is integrating this capability into overall planning efforts and operationalizing it among its field units; b) Who are the principal institutional and individual “actors” in Chinese CNO and what linkages may exist between the civilian and military operators; c) Possible targets of Chinese CNO against the US during a conflict to understand how the PLA might attempt to seize information control over the US or similar technologically advanced military during a conflict; d) The characteristics of ongoing network exploitation activities targeting the US Government and private sector that are frequently attributed to China; e) A timeline of alleged Chinese intrusions into US government and industry networks to provide broader context for these activities” (DeWeese, Krekel, Bakos, & Barnett, 2009). Those two assessments, while pursuing very similar tasks and approaching the problem from a similar perspective, differ on many details. This is quite natural and also welcomed, as different


analyses may uncover different aspects of the same problem and produce tailored results for a specific receiver. A number of threat analyses have been found at the country level concerning computer network operations (CNO) and computer network exploitation (CNE) capabilities. Such analyses can be used to identify main traits of specific threat agents. The reports are: (Denning, 2007), (DeWeese et al., 2009), (Patterson & Smith, 2005), (Duggan & Michalski, 2007), (Hjortdal, 2011), and (Greitzer & Hohimer, 2011). One of the greatest contributors of computer security theory, Prof. Dorothy Denning, summarized in a recent study the results of 3 years of research assessing the CNO threat of foreign countries. Her work is focused on defense systems but it is still relevant as it describes the methodology used and gives a brief qualitative description of the findings. The methodology seems based more on past experiences than widely tested mechanisms. Furthermore, RAMCAP (ASME-ITI, 2006) are a set of guidelines used in the mechanical engineering field. They were promoted by the DHS after the horrible events of 9/11. Even these guidelines talk about threat characterization, scenario assessment, and the use of metrics such as “attractiveness”. Matt Rosenquist uses a knowledge-based approach to prioritize information security risks through the use of a novel method based on threat agent assessment (Rosenquist, 2009). However, while the provided knowledge base can be of help as a checklist for comprehensiveness of the analysis, the “expert” approach to the assessment process seems approximate and at the end simplistic. At a lower level, Myagmar’s research (Myagmar, Lee, & Yurcik, 2005) shows how fundamental it is to do a proper threat assessment, in software development. Such approach is fundamental at every level, but there is no such threat assessment described at higher level, security strategy and governance, in any standard or applicable regulation. There are a number of other methods used in different disciplines to identify the characteristics of attackers, similar to those used in the strategy analysis field: Multiple Pure Strategy Nash Equilibria, Mixed Strategy Nash Equilibria, Minimax Theorem (Solving a TwoPerson, Zero-Sum Game), and Berejikian Approach to Domain of Gains/Losses. These methods make use of a number of characteristics (Taquechel, 2010): 27


Attacker’s will as a function of interest; Risk propensity; Capability (organizational, economic, and technical); Absolute and relative loss/gain analysis.

1.2.6 Risk mitigation Another aspect to be considered is that of risk mitigation. Research is becoming so specialized in information security that other active fronts towards the minimization of risk do not even take vulnerabilities in consideration. This is the case of research like that of Kumar and colleagues (Kumar, Park, & Subramaniam, 2008) that focuses on countermeasures portfolios. Their research concludes that in order to choose the correct countermeasures you should assess the business environment and the threat environment. This research does not even consider vulnerabilities despite its objective to reduce the risk factor. Otero and colleagues in their research argue that current countermeasure selection methods do not take into consideration organization specific constraints such as costs of implementation, scheduling, and availability of resources (Otero, Otero, & Qureshi, 2010). They propose to rank the countermeasures taking into account benefits and penalties (restrictions) associated with their implementation. The authors use available data to better select countermeasures. It is necessary also to exploit “external” intelligence and data to better assess the organization’s threats, however this practice is still not widespread. Specialization of research and of practitioners in very narrow aspects of information security has enhanced our understanding and capabilities of securing our systems and the information contained therein. However, it also has created additional burden, as the complexity of these systems and of their interactions, made it more difficult to have a big picture view of the system, and understanding the security status of the whole. This problem has also been exacerbated by the growing number of people working in the field, combined with the growing dimensions of the system. In such a picture, the executives who are responsible for the strategy planning and the governance of an organization, and definitely for security funding and key decision making, are almost always removed from the core of the problem.


1.2.7 Risk monitoring and evaluation Risk and situational awareness Not only security related organizations, like INSA with its technical report “Cyber Intelligence: Setting the Landscape for an Emerging Discipline” (INSA, 2011), and hackers (Parker et al., 2011) call for a better threat and threat agents characterization, but also the government. A report by the Office of the Under Secretary of Defense titled “Capability Surprise, Volume II: Supporting Papers”, identifies a number of preventive actions for improving cyber security. Among their proposals for improving security it is possible to read: “a key step in preventing surprise is to understand adversary capabilities and intensions. The potential “penetrator” must himself be penetrated, and not solely by cyber means” and again: “All disciplines of intelligence, especially human intelligence and signals intelligence must be brought to bear and then correlated to understand present and future threats in cyberspace” (Board, 2009). Such statement definitely gives the audience a clear view on the government perspective! Attacker’s characterization is part not only of risk assessment, but also of cyber situational awareness, (which should be) covered in risk management. This discipline and its current status are well covered in (Jajodia, Liu, Swarup, & Wang, 2010). While the book is specifically dedicated to cyber defense and not cyber security a lot of things can be transposed. For example the basic seven aspects of cyber awareness: 1) Be aware of the current situation; 2) Be aware of the impact of the attack; 3) Be aware of how the situations evolve; 4) Be aware of actor (adversary) behavior; 5) Be aware of why and how the current situation is caused; 6) Be aware of the quality (and trustworthiness) of the collected situational awareness information items; 7) Assess plausible futures of the current situation. This last one requires also the understanding of adversary: a. Intent


b. Opportunity c. Capability (Jajodia et al., 2010).

The authors recognize human decision makers as an indispensable “component of the system”, and that “cyber situational awareness can be gained at multiple abstraction levels” and recognize: “Existing approaches to gain cyber situation-awareness consist of vulnerability analysis (using attack graphs), intrusion detection and alert correlation, attack trend analysis, causality analysis and forensics (e.g., backtracking intrusions), taint and information flow analysis, damage assessment (using dependency graphs), and intrusion response. These approaches however only work at the lower (abstraction) levels. Higher level situation-awareness analyses are still done manually by a human analyst” (Jajodia et al., 2010).

The researchers identify as a problem the fact that existing approaches are not efficient enough and need to handle uncertainty better. In chapter 2 of their book the authors describe a situation awareness reference and process models, while in chapter 3 they define a framework to connect cyber situation awareness with human analysts through a holistic decision-making model. Four principles are identified, of which the first three refer directly to our topic: • • • Principle 1: Full situation-awareness for cyber defense requires a holistic methodology to synthesize perception, understanding and projection. Principle 2: Information systems with full situation-awareness must manage uncertainty (e.g., through hypotheses and reasoning). Principle 3: Cyber situation-awareness must be gained at multiple abstraction levels (Jajodia et al., 2010). 30

There is also number of technical reports from MITRE regarding their methodology “Cyber Prep”: (Bodeau, Boyle, Fabius-Greene, & Graubart, 2010, Bodeau, Fabius-Greene, et al., 2010, Bodeau, Graubart, & Fabius-Greene, 2009). While these reports are only part of a larger methodology whose goal is to assess cyber preparedness and hence are not aimed specifically at risk, it is indeed important to see how preparedness is becoming increasingly important in this field. Most of this work is based on threat assessment rather than vulnerabilities. Risk scoring and characterization Many researchers recognize the same problem (Crowther, Haimes, & Johnson, 2010): Risk scoring is poorly executed now, and most of the current research focuses on two things: (1) information security resources that reduce the likelihood and consequences of successful information exploits (i.e. the countermeasures); and (2) security processes and capabilities that improve those from the previous point. “Formal approach to security metrics. What does ‘more secure’ mean for you?” is a recent research study published by Leanid Krautsevich and colleagues (Krautsevich, Martinelli, & Yautsiukhin, 2010) which provides a number of formal security metrics to represent security in a mathematical formal model. These metrics can be used to format input for a Bayesian decision model. However, they still need a way to get input, which is devoted to awareness and threat characterization. Their research can be considered a step before research oriented towards the creation of security metrics, taxonomies and ontologies: (Vaughn, Henning, & Siraj, 2001), (Cebula & Young, 2010), (Lee, G, & Ahn, 2005), (Fenz & Ekelhart, 2009), (Meyers, Powers, & Faissol, 2009). “A strategic modeling technique for information security risk assessment” by Subhas Misra and colleagues, (Misra, Kumar, & Kumar, 2007) presents an information security risk assessment modeling technique which represents a substantial advancement among common qualitative methods and uses, and tailors, some other consolidated theories. Its main value lies in the strategic view of the risk and the category of relationships that aims to build among the different entities involved in a security risk analysis scenario. However, while advancing the representation and modeling of security risk scenarios, not much attention was put into the 31

identification of input data, nor in the quantification, even if only discrete, for the final qualitative analysis. 1.2.8 Automatic tools and mathematical models Current research efforts identify two main roads towards security improvement: (1) automating as much as possible recovery and other mitigating tasks, in order to not let the human perception mislead or slow down system responses, creating faster and bias free proactive tools. And (2) embracing the human element and empower it for the benefit of the Socio-technical system security. We are not concerned with the first approach, which can be applied only at low level for now, and is unlikely to be feasible at the strategic level, at least for years to come. One of the reasons for not leaning too much towards statistical tools approach is that we do not have reliable statistical data, because (1) attacks are not systematically identified, and (2) attacks are not consistently reported. Without reliable data it is impossible to have reliable statistics, and thus predict attackers’ behavior. The present research thus follows the path of those that embrace the human decision making part with a holistic approach, which is applicable to almost all the risk management methods already seen. Probability based models Among those efforts oriented towards more automation and low level analysis, there are a number that make use of Bayesian belief networks (BBNs), belief functions (Dempster-Shafer Theory), and other variants to create probabilistic models (Kondakci, 2010), (Sun, Srivastava, & Mock, 2006), (Aven, 2006). Such models usually identify a security impact as the combination of possible consequences and related uncertainties. While they can be very effective in using probability as a measure of uncertainty, and in describing the propagation of risk in the system, such models need a number of quantitative inputs. Such inputs often include complex human or organizational traits that cannot be known (attackers capability level, and risk propensity) or probability of events for which there is no historic data (even today the number of companies collecting security statics on their infrastructure is abundantly low), and are translated into quantitative measures without a stable conversion mechanism. The results of these models rely


heavily on the performer’s ability to attain the necessary information, evaluate it, interpret it, code it and introduce it in the model in the correct way. There should be ways to not only update, but also refine threat and threat agents input after continuous analysis, and then build into the STS a way to implement changes, such as different security measures, etc. through policy and change procedures for example. Such an approach has been taken by Kim and colleagues in their research paper on time-variant risk analyses and damage estimation (Kim, Chung, Lee, & Won, 2005). Their research addresses the variance with respect to time in risk assessments at a very low level, such as that their method can be implemented by automatic tools. However, automatic tools cannot yet enforce security alone, and are still dependable by human supervision and main control; furthermore, they are nowhere near to be powerful enough to be used comprehensively at STS level on a wide, distributed and heterogeneous way. Game theory based models Game theory had a special role in security risk management and most current research in this field continues to add new stimuli to its application. Cremonini and Nizovtsev used game theory to model and assess the interaction between strategic attackers and defenders (Cremonini & Nizovtsev, 2009). Their study points out that well-protected targets can use signals of their superior level of protection as a deterrence tool. A similar research (Bier & Abhichandani, 2002) uses game theory to identify and characterize optimal defensive strategies, debunking a long standing myth that security cannot be achieved through obscurity. Their research shows that at the strategic level “secrecy or even deception can be an important strategy for improving security (especially for series systems), and/or reducing defensive costs” (Bier & Abhichandani, 2002). A lot of effort is being spent to this type of research. This tactic is the result of the understanding that one of the major faults in current security methodologies is the inability to identify and predict threats with a sufficient degree of certainty and early enough. Gueye’s is among those convinced that it is not feasible to design a defense against every specific possible attack, i.e. design and implement a specific countermeasure for each threat. His research (Gueye, 2011) argues that game theory can be used to “design a defense against a sophisticated attacker who plans in anticipation of a complex defense. By including this 33

‘second-guessing’ element into the design process, Game Theory has the potential of crafting improved security mechanisms” (Gueye, 2011). In another field, Hausken and Zhuang make use of a game theory model to analyze governments’ and terrorists’ defense and attack strategies (Hausken & Zhuang, 2011). Additional non-information security assessments can be identified in other studies (Willis, LaTourrette, Kelly, Hickey, & Neill, 2007) and (Burns, 2006). Such assessments, if contextualized in a cyber-scenario, for a specific government/organization, and with a specific threat agent, are exactly the kind of analytical tool that can help in assessing and characterizing cyber threat agents. However, currently there is no standard instrument to do this in the information security field. Data on threats and on non-technical aspects is often inadequate to approach information and cyber security in a comprehensive, effective, and efficient way.

1.3 Problem statement
From the background research and the literature review presented in the previous chapter, it is possible to see how the methodologies for information and cyber security risk management and assessment are still heavily and continuously debated and can indeed be considered hot topics despite more than 30 years of research in the field. Current security methodologies rely heavily on how such methods are implemented and on the knowledge and ability of the people involved. It is also difficult to understand what the different methodologies really provide in terms of security and what they lack. It is not clear if there are common pitfalls or weaknesses, and what improvement can be implemented at the non-technical level. A first analysis of collected material presented during the literature review suggests that some aspects may be missing or insufficiently developed in the previously discussed methodologies. Without attempting to define new methods, we can look to other fields which face similar problems such as national security and defense, which have always invested significant resources to advance know-how and technology for security purposes. These fields employ a series of mechanisms, methodologies, and processes that are not matched in the information security field. One such assessment methodology is called Net Assessment and it has been successfully applied in the Department of Defense (DOD) for more than 70 years.


“A net assessment is an appraisal of one’s ability to achieve desired military objectives in the face of opposition, and how that ability changes over time. […] As practiced in the Office of the Secretary of Defense/Net Assessment since 1973, it refers to the comparative analysis of military, technological, political, economic, and other factors governing the relative military capability of nations” (Mears, 2010). This chapter attempts to deduce some generalized observations on existing methodologies derived from the literature review. Such ideas will then be consolidated into a problem statement. This research will then further analyze if Net Assessment can improve or add features now missing in current information security risk management methods. The choice of analyzing Net Assessment as a possible addition to the field of information security risk assessment stems from a number of other research papers. For example, it is proposed as a “next step” in cyber defense research by Kugler in “Deterrence of Cyber Attacks” (Kugler, 2010) which, in turn, builds upon the findings of many other studies in risk assessment methods comparison, security strategy, and national defense. Should this research suggest that Net Assessment would be useful in the field of IT and cyber security, this would be of potential interest to every organization dealing with critical infrastructures or high security requirements, bearing a direct, immediate practical significance. Should it be discarded as a potentially effective method, the results would still be important for further scholarly research, as they may give more indications on intermediate findings or on what alternative risk assessment methods used in other fields may be possible candidates. 1.3.1 Research questions As discussed further in chapter 2, the present research will follow a qualitative approach. Such an approach is based on research questions, not objectives or hypotheses, which are answered through sound analysis. In qualitative research, the research question may assume two forms: (1) The grand tour question, and (2) Sub questions. Considering the problem statement of this research, the grand 35

tour question is: How can cyber and information security risk management be improved? This question is somewhat already specific, as it picks a specific activity: That of risk management in a specific field: Information security. However this question should still be considered very broad and general as we have already seen how such process encompasses multiple areas (it is multidisciplinary) and must be tackled from many angles (it is holistic). While it is true that the results for the present research can suggest ways to answer this question, it is still necessary to define a more narrow and focused questions. The following sub-questions help to further narrow the focus of the study in a single, specific direction: Q1 – What aspects are covered by current information security methods? Q2 – What aspects are covered by Net Assessment? Q3 – What aspects does Net Assessment include that are not present in current information security risk assessment and management methodologies? The goal of this study is not to prove that Net Assessment can improve the security posture of a particular organization, but rather to discuss what current information security risk management methodologies characteristics are and how they differ from Net Assessment. The study will then address the possibility of adding Net Assessment as a tool to improve information security risk management and analyze whether this may be an effective addition to the field.

2. Scientific Approach
This chapter is aimed at presenting and discussing the research methods used in this study, with a twofold objective: To familiarize the reader with the methods used, and to allow the reader to assess the validity of the conclusions based on this methodology. The ‘Frascati Manual’, used worldwide as standard for scientific research for more than 40 years, outlines three forms of systematic scientific research – basic research, applied research and experimental development. Pure research (also known as basic or fundamental research) is carried out to increase understanding and acquire new knowledge of fundamental principles and underlying foundation of observable facts. Applied research is undertaken in order to acquire new knowledge primarily towards a specific practical aim or objective. Experimental 36

development is performed to develop, install or improve materials, products, devices, processes, or systems (Frascati Manual, 2002). The aim of the present inquiry is to improve the understanding and implementation of information security through the analysis and improvement of information security risk management and assessment practices. This scientific inquiry has been conducted in a systematic way under the profile of applied research. The next chapters address the research methodology and all the details needed to appraise the applied methods.

2.1 Methodology
This study follows the canons of scientific method to pursue a secondary research in the field of information systems security. Guba, (Lincoln & Guba, 1985), Lincoln and Denzin (Denzin & Lincoln, 1994) “define the term paradigm as a systematic set of beliefs, and their accompanying methods, that provide a view of the nature of reality. They argue that scientific inquiry is defined by the positivist paradigm, which has prevailed until recently” (Savenye & Robinson, 2004). The research will follow the paradigm of Pragmatism, which has been adapted and applied to this specific inquiry through the frameworks described in the next two chapters. Qualitative research methods have been developed to address research problems related to the following questions: Why? how? in what way? Quantitative research instead is more concerned with questions: How much? how many? how often? to what extent? With qualitative methods, research is conducted through a holistic perspective and through time consuming data analysis (Hancock, 1998). This study makes use of qualitative methods. Qualitative research is not driven by hypotheses, which in quantitative research usually dictate the methods to be used, and is thus usually not preemptive. In this study, the adopted analytic design is obtained from the questions, from the chosen methods, from the selected topic and goals, and also take into account analyzed data (Richards, 2006). The research design has been shaped with a strong focus on achieving validity. Two general, basic requirements for validity are: (1) insuring that research question, input data, and research methods all correspond; and (2) instill in the design a number of instruments to ensure that each step in the analysis phase is properly accounted for (Richards, 2006). In the following chapters,


the research framework is discussed explicitly as it bears implications for the effectiveness of the study and because “good research requires making assumptions, paradigms, and frameworks explicit in the writing of a study, and, at a minimum, to be aware that they influence the conduct of the inquiry” (Creswell, 2007).

Figure 5: Areas of research involved

One of the main challenges of this study has been the multidisciplinary nature of the problem. Considering the problem statement and the research questions, a number of research areas have been identified and need to be addressed. Figure 5 depicts the five main areas identified. The branch of Socio-technical systems is necessary to understand the systems under scrutiny and the environment in which they operate. The field of computer science and IT in general provide models for understanding the physical, technological layers of such systems, comprised of hardware, software, and the data contained therein. The field of information security is the main territory for integrating organizational and business aspects of information


with technical aspects of security. The practice of risk management needs to be included as it contains the instruments for assessing and managing risks and security concerns. Finally, the security strategy perspective adds the strategic instruments with a higher abstraction level for framing the STS in the external environment and the strategic approach to information security. Not all the identified fields have been addressed at the same level of depth, however each of them has been analyzed for its role in the present research. 2.1.1 Theoretical framework “Theories are constructed in order to explain, predict and master phenomena. A theory makes generalizations about observations and consists of an interrelated, coherent set of ideas and models” (Khan, 2007). The theoretical framework of a study is the overall framing structure that supports the theory behind a research. It explains why the identified problem exists and why the proposed approach is a feasible solution. Thus, the theoretical framework is the basis for conducting research (Khan, 2007). Considering that the identified problem has to be addressed through a multidisciplinary approach, identifying a suitable methodology for answering the research questions was not an easy task. Information and cyber security can be addressed at multiple levels of abstraction. While at the lower levels of abstraction the basis for security are supported by formal methods, higher levels of abstraction, such as governance, are not based on logics or mathematics but on a number of concepts and practices that rely more on empirical experimentation and other scientific instruments related to the social sciences. According to Gregor’s scale, this research focuses mainly on the analysis and description (theory type I) (Gregor, 2006). While there is a component dedicated to the design (as a categories structure has to be designed) and comparison, these tasks do not fall in the explaining (type II), predicting (type III), EP (type IV) or design and action (type V) categories (Gregor, 2006). Internal to the type I research, this study will identify suitable mechanisms for pursuing the multiple identified objectives. Information security risk management is a truly interdisciplinary field and, while the inquiry includes elements from other fields, the theoretical framework for this research is identified in that of information security risk management. This study makes use of strategies, methods, and 39

results achieved in previous research on security risk management and the other identified fields. In sum, this study is focused on the functionality of current methodologies and approaches, as opposed to assessing their effectiveness. 2.1.2 Conceptual framework The conceptual framework is the operationalization of the theoretical framework. It gives direction to the study and shows the relationships of the different variables to be investigated. It addresses all aspects of the inquiry, acting like a map and offering coherence to empirical inquiry (Khan, 2007). Inside the given theoretical framework, it is possible to identify at least one problem relating to information security risk management: it doesn’t really work for minimizing security incidents. Despite much effort, security breaches occur daily with greater frequency than may be anticipated, and victims include even security organizations. This begs the questions: how can we improve security; What can be done more or better; What can be reused from other fields, and Can NA help? This description of the conceptual framework explains how to tackle these questions and especially the three questions identified in chapter 1.3.1 inside the previously presented theoretical framework. We need to better understand what aspects are included in the common body of knowledge and if they are complete. The study therefore assesses the standards, regulations, best practices, and common body of knowledge to ascertain if there is any aspect that is not present, and is instead present in Net Assessment. This research challenges the current information and cyber security risk management methodologies. It does so by identifying the base theories, collecting a representative set of them, and analyzing their characteristics under the perspective of completeness or coverage, in order to identify what they do or don’t provide. To do this, an analysis of methods was conducted. However, meta-analysis is typically a quantitative research (Hopkins, 2011), and therefore it cannot be proficiently used in this context for our purposes because (1) the risk assessment methodologies to be assessed are not quantitative methods, and (2) meta-analysis, as conceived in the scientific context, involves statistical analysis (Hale & Strube, 2003), while the sample collected for this research doesn’t follow the prescriptions required for statistical analysis.


Cyber and, in general, information security risk assessment and management methodologies are not a precise science. They do not provide exact quantification of potential risks, and any such effort must be mediated by the experience of the performer (Conrad et al., 2006). Security risk management and security risk assessment are in fact usually called “practices” to highlight the fact that such activities are composed of multiple different aspects and that their final results are also affected by the practitioner. The best method to analyze and compare them on their merit, strengths and weaknesses, but especially their functionalities’ coverage, is thus to use a qualitative approach. Grounded theory research (Glaser, 1992; and Glaser & Strauss, 1967) has already been used by Kennet J. Knapp (Knapp, 2005) and Wanda J. Orlikowski (Orlikowski, 1993) in a similar context; however it is not well suited for this specific research, as the aim is not to define a theory or basic idea, but rather to answer three specific questions through a holistic perspective. Considering the problem at hand, the best conceptual framework for the present investigation is a comparison strategy. The operationalization of the proposed methodology is composed of two different phases. The first phase of the research is concerned with identifying current information security risk management methodologies and comparing and categorizing their distinctive features. The second phase of the research will analyze Net Assessment and determine if it provides different functionalities and if those can improve current security risk management methods.


Figure 6: Information systems research framework

Figure 6 is a pictorial representation of the conceptual framework made by Hevner and colleagues for evaluating information systems research (Hevner, March, Park, & Ram, 2004). This research follows Hevner’s framework and will analyze both organizational and technological (in particular strategies, processes and infrastructure) and also the implementation of information security risk management in this context. In the literature review, the existing knowledge base has been assessed together with other existing theories, frameworks, models and methods. Assessment methods have been compared and integrated to create instruments and models for the analysis and comparison of risk management and assessment methods. In the first phase of this study, a classification will be established through the use of the qualitative method of “collection and content analysis” (Sofaer, 1999) and the technique of “selective coding” (Strauss & Corbin, 1990 and 1998). The typologies which are used to build the classification will consist of a number of factors identified through the analysis of selected security methodologies’ assessment methods, mainly considering their specific functions and 42

coverage. The process chosen for guiding the Risk Assessment Methodologies’ (RAM) analysis is that of “continuous identification and comparison” (Paterniti, 2007). The results of this process will be the categorization of the sample of risk assessment methodologies, the coding of their relative characteristics, and their comparison. The first phase then will provide insight on question Q1 (What aspects are covered by current information security methods?). The same approach will be used also for the analysis conducted in the second phase which is focused on NA. The findings of the second phase will give insight regarding the questions Q2 (What aspects are covered by Net Assessment?) and Q3 (What aspects does Net Assessment include that are not present in current information security risk assessment and management methodologies?). 2.1.3 Research methods The background research and literature review conducted in chapter 1.1 clearly support the identification, and suggest the existence of, such problem as defined in chapter 1.3. It is also understood that, given the nature of the problem and the resulting research questions, it is not possible to quantitatively assess the identified characteristics, thus a qualitative research will be used. Considering the different fields involved, the diverse subjects, and characteristics to be studied and the ephemeral state of security in organizations, the only mathematical instruments that could be used for our purposes is that of statistics. However, a statistical inquiry could be construed only on perceptions of security managers on the degree of security, of risk, and of residual vulnerability in relation to the implementation of the identified methods. There is no statistical data available for the methods’ usage and even more difficult will be to quantify their results in terms of effectiveness. It is still widely debated how to evaluate the effective increase (or decrease) in cyber/information security, especially concerning non-technical measures. In order to tackle the problem identified in chapter 1.3 and to answer the resulting questions, current information security risk assessment methods were analyzed through an inductive process that identified their characteristics, and then through a comparison method that gave insight on whether Net Assessment can bring added value with respect to the current methodologies. Using an inductive reasoning (bottom up), starting from the analysis of the qualitative input data, i.e. the details of standards, regulations, best practices, and other input 43

documentation, the research identified critical answers to the given questions and gave reasoned support for such insight. The whole process is represented graphically in figure 7. The main research method used in this secondary research was “content analysis”.

Figure 7: Inductive, bottom up, approach to identify covered areas

Since there is no consolidated standard for meta-assessment of Information Security (IS) RAMs and since the well consolidated meta-analysis techniques, such as that thoroughly described by Hale and Strube (Hale & Strube, 2003), apply only to quantitative, statistical analysis, the part of this research devoted to the analysis and categorization of IS RA and RM formal methodologies employed a qualitative method to identify their relevant characteristics and attributes. Such characteristics were then used as a basis for the methodologies’ assessment, which were conducted through constant comparison (Orlikowski, 1993). This inquiry used a very well tested method, already previously used in this context, i.e. “presence check” with respect to


specific categories. Available taxonomies and models created to evaluate risk management and assessment methodologies were analyzed and a single list of characteristics was synthesized. Once the categorization was finalized, the identified risk management and assessment methods were compared with the identified list. This helped to create a coverage map with respect to identified characteristics. The results of this first analysis provided a classification of IS RAMs through the lenses of the identified characteristics as presented below in figure 8.

Figure 8: Document analysis process

Written data was categorized for purposes of classification through content analysis. The content of the documents was analyzed mainly at the basic (or manifest) level, focusing on the descriptive representation of the data. This approach seems reasonable and sufficient as the selected sources of documents have been prepared with objectivity in mind and to enable a homogeneous, standardized, implementation of their contents. However, we also recognize that while being coherent in their singularity, differences in the definitions of concepts may arise. Terminology has not yet been fully standardized, and one of the challenges of this research is to compare documents that may use the same terms for different meanings. For this reason, upon necessity, a higher level of content analysis was used, termed “interpretative.” With this second


level of analysis it was possible to consider not only what is written, but also what was meant with those words or symbols, what was inferred or implied (Hancock, 1998). Consequently, in phase two, Net Assessment was analyzed and compared with the identified features’ category list. The final results of the analysis identify possible additional functions provided by the Net Assessment methodology following the logic process illustrated below in figure 9.

Figure 9: Deductive, top down, approach to identify possible benefits

The qualitative approach used in the present research can be assimilated to the first perspective that White identifies in his “Problems of comparative qualitative research”: “in one perspective, the distinguishing feature of the qualitative method is mainly that it seeks to draw conclusions from a small number of cases (King et al. 1994). The attempt to reach generally valid truths about the social world by the logic of inference from a sample to a wider population is held to be the same as with quantitative 46

research, albeit the ‘small-n’ sample makes conclusions more tentative” (White, 2008); The comparative aspect of the research is genuinely interpretive, with the major aim of identifying and increasing the understanding of diversity and consistencies in the assessed methodologies. This reflects White’s view that “comparative research, though also subject to competing interpretations, generally involves the search for patterns of similarity and difference across a number of observed phenomena” (White, 2008). The next chapter will detail the methods used and put them in context with the focus of research and the collected data. 2.1.4 Analytic design While “quantitative research sampling seeks to demonstrate representativeness of findings through random selection, qualitative sampling techniques are concerned with seeking information from specific groups and subgroups in the population” (Hancock, 1998). The proposed sampling technique is described in chapter 2.2.2. More details on data sources can be found in chapter 2.2.1. This chapter is devoted to the description of the operationalization of the identified research methods. The first step created a category list to describe coverage by risk management and assessment methods. The second step analyzed and compared the identified methods with the taxonomy and determined their coverage or approach towards such characteristics. The third step analyzed Net Assessment and identified its own characteristics. This was based on information from a number of different sources as there is no publicly available standard reference manual. Starting from the understanding of its features, Net Assessment has then been compared to the features category list and its coverage identified. Finally, potential advantages or improvements offered by Net Assessment were identified and discussed. The process of creating the category list includes a number of steps. While the basic process of data analysis is the same for both quantitative and qualitative data, there are still major differences and additional burden for the analysis of qualitative data. In a qualitative context there is usually no system for pre-coding, therefore the researcher needs a method of identifying and coding items of data which appear in the text of the selected documents so that all the selected items from multiple documents can be compared to each other (Hancock, 1998). This 47

effort was alleviated by the fact that other multiple coding techniques were used as basis for this qualitative analysis. The procedure used is described below: 1) Read through the identified risk management assessment method. A number of salient characteristics have been selected from the pre-existing analysis framework or model. The selected characteristics usually aimed at identifying functionalities and approach of RM methods. 2) Once the document was fully analyzed, a spreadsheet was created to keep track of the selected characteristics. 3) Repeated the process from steps 1 and 2 for every risk management or assessment comparison study identified. The product was a number of spreadsheets containing lists of items excerpted from the analyzed texts. 4) All the spreadsheets have then been merged. The merging brought all the data in a single spreadsheet. The categories have been organized in descending order, starting from higher abstraction level characteristics down to lower abstraction level characteristics. 5) The final single list has been harmonized: Duplicate characteristics have been eliminated, while conflicting terminology and granularity of details have been harmonized. 6) Read through the list of category items. Each item has been described in a way that describes how it is used, and what its contributions to the security process are. 7) Consistency and completeness check. Once all the data has been sorted into categories, the whole categorization framework was reviewed, considering selected categories and their meaning. At this stage some items may have been moved and category names changed. 8) After all the categories had their fixed position in the categorization systems, these were split in two distinct lists of “general characteristics” and “technical characteristics”. Any necessary element needed to imprint a big-picture understanding to the categorization was given.


9) Further review of all the analyzed documents with the knowledge of the final categorization system. This was useful to identify additional items initially missed.

After defining the categorization lists, the study evolved to the second step of the first phase, culminating with the holistic comparison of identified risk management and assessment methods. The first task of such analysis was to critically read each document, identify the critical variables, observe their interactions, and finally identify the coverage on the basis of the findings. Miles and Huberman state that qualitative data analysis consists of “three concurrent flows of activity: Data reduction, data display, and conclusion drawing/verification” (Miles & Huberman, 1994). Qualitative analysis of selected collected documentation took into account both the content and the structure of such documents. Constant comparison, involves categorizing, or coding, data as they are collected and continually examining data for examples of similar cases and patterns (Savenye & Robinson, 2004). This analysis does not include the detailed semiotic analysis of content and words, as it is out of our scope. However, the concept of meaning of special words in the specific context, in order to make the content analysis more sound, are addressed. The categorization framework will be presented in table form, as well as the coded results of the content analysis. The followed procedure (content analysis) is described below: 1) Read through the selected risk management or assessment document. When a concept relevant to the identified categories was detected, it was noted in a spreadsheet, indicating its presence, meaning, and use. 2) The product was a list of items excerpted from the analyzed text. Read through the list of data items and categorized each item resembling the categories present in the category lists. 3) Repeated steps 1 and 2 for every selected text. 4) Consistency check among all the spreadsheets: If any methods were found short of some characteristic that would be double-checked. If the same feature was found in multiple documents of the same family, the wider concept was used. 49

5) Compiled the final category tables with data related to the assessed methods.

The second and third research question were defined through the development of the research process as it becomes clear that while some differences (in some cases rather substantial differences) were present among the assessed security risk management and assessment methods, other aspects were partially or even completely neglected in many of them. This was most common in strategic defense assessments. Among the strategic assessment methods available in other fields such as that of international security, national defense, etc., one was selected as a proof of concept for its qualities: Net Assessment. Consequently, the two remainder questions were formulated with the Net Assessment in mind. Those two latter questions assess the Net Assessment methodology through the lens of the categorization lists identified in the first phase, and by going through the same process as identified above. However, there was a major challenge: Net Assessment doesn’t come with an official manual or standard reference, at least not in open sources. Net Assessment characteristics thus have been deduced from the assessment of multiple documents, that either were a description of the Net Assessment methodology, or were reports written using the Net Assessment methodology. Despite this difference, Net Assessment has been analyzed with the same process as outlined above. The results of the content analysis of Net Assessment related documents are presented in chapter 3.2.1 and in table form in chapter 3.2.2 while its critical comparison with information security risk management methodologies and the proposal for its possible utilization for information security are presented in chapter 3.2.3.

2.2 Data
It doesn’t matter if research follows a quantitative or qualitative approach; however, in order to correctly apply the scientific method, input data must be sound. The following two chapters provide details on which data sources have been used and why; what data has been collected and why; how data has been selected and sampled and why. The data sources, selection and sampling were done in order to ensure that the data is sufficiently valid, i.e.: sufficiently rich, complex, and contextual to address the three identified questions and support the outlined analysis


(Richards, 2006). The following details can be used to verify the attained results and to reuse it in similar approaches. 2.2.1 Data sources The proposed research required the identification and sampling of a number of information security risk management methodologies, security assessment methods, and many documents related to Net Assessment. The identification of relevant methodologies has been performed through the collection and analysis of research production, practitioner’s resources, public reports and guidelines and lists compiled by international security bodies. The scope of the identification has been worldwide. Sources on RM and RA assessment methods to create the category list are presented in alphabetical order in the following list: • • • • • • • • • A Comparative Framework for Evaluating Information Security Risk Management Methods (Bornman & Labuschagne, 2004); A framework for comparing different information security risk analysis methodologies (Vorster & Labuschagne, 2005); An analysis of the traditional IS security approaches: implications for research and practice (Siponen, 2005); CERTS: a comparative evaluation method for risk management methodologies and tools (Garrabrants, Ellis, Hoffman, & Kamel, 1990); Comparison of Risk Analysis Methods: Mehari, Magerit, NIST800-30 and Microsoft's Security Management Guide (Syalim, Hori, & Sakurai, 2009); Design of a modelling language for information system security risk management (Mayer, Heymans, & Matulevicius, 2007); Factors in the selection of a risk assessment method (Lichtenstein, 1996); Homeland Security Risk Assessment Vol. II Methods, Techniques, and Tools (Cummings, McGarvey, & Vinch, 2006); Methodology for evaluating usage and comparison of risk assessment and risk management items (ENISA, 2007);


• •

Multi-criteria model for evaluation of information security risk assessment methods and tools (Sajko, Hadjina, & Pesut, 2010, Sajko et al., 2010); and Towards a Measurement Framework for Security Risk Management (Mayer, Dubois, & Matulevičius, 2008).

The following list contains the references of RM and RA methods related to the NIST framework. These documents were assessed to determine the characteristics coverage of NIST methods: • • • • • • • • • • NIST IR7298-rev1, Glossary of Key Information Security Terms; FIPS Publication 199, Standards for Security Categorization of Federal Information and Information Systems; NIST Special Publication 800-30, Risk Management Guide for Information Technology Systems; NIST Special Publication 800-37-rev1, Guide for Applying the Risk Management Framework to Federal Information Systems; NIST Special Publication 800-39, Managing Information Security Risk; NIST Special Publication 800-53-rev3, Recommended Security Controls for Federal Information Systems; NIST Special Publication 800-53A-rev1, Guide for Assessing the Security Controls in Federal Information Systems; NIST Special Publication 800-60 Volume I-rev1: Guide for Mapping Types of Information and Information Systems to Security Categories; NIST Special Publication 800-60 Volume II-rev1: Appendices to Guide for Mapping Types of Information and Information Systems to Security Categories; NIST Special Publication 800-137-Final, Information Security Continuous Monitoring (ISCM) for Federal Information Systems and Organizations.


The following list contains the references of RM and RA methods related to the ISO framework. These documents were assessed to determine the characteristics coverage of ISO methods: • • • • • • ISO/IEC 27000 Information technology — Security techniques — Information security management systems — Overview and vocabulary; ISO/IEC 27001 Information technology — Security techniques — Information security management systems — Requirements; ISO/IEC 27002 Information technology — Security techniques — Code of practice for information security management; ISO/IEC 27003 Information technology — Security techniques — Information security management system implementation guidance; ISO/IEC 27004 Information technology — Security techniques — Information security management — Measurement; and ISO/IEC 27005 Information technology – Security techniques – Information security risk management.

Sources on other fields: • Collected through extensive research in multiple libraries and directly cited along the text in the literature review or in the respective chapters. In general, these sources are international standards, national standards, best practices, or peer reviewed specialist publications.

Sources on Net Assessment: • Collected through extensive research in multiple libraries and directly cited along the text in the respective chapters. In general, these sources are master thesis, DOD notes, lectures notes, and subject matter expert’s accounts of the process.


2.2.2 Data sampling There is a plethora of information and cyber security risk management methodologies currently in use: Open source, proprietary, and classified. It is not necessary for this research to assess all of them. In order to keep the research open and accessible, only the open source methodologies have been assessed, discarding proprietary and classified methodologies. Proprietary methodologies that were not described in public documentation or research, have not been addressed, as their status confines them in the niche of the developing organization, which is out of the scope of the present research. Among the remaining open source methodologies, “purposive sampling” (Teddlie & Yu, 2007) has been applied. Purposive sampling techniques, also called nonprobability sampling or qualitative sampling, involve selecting certain units or cases “based on a specific purpose rather than randomly” (Tashakkori & Teddlie, 2003). Of the three main different types of purposive sampling widely used in research, the one employed in this study is the “sampling to achieve representativeness or comparability” (Teddlie & Yu, 2007). The sampling strove to select those risk management and assessment methodologies that are public and cover the most features. The focus on features coverage instead of more widespread sampling comes from the unfortunate realization that there are little to no statistics on the risk management and risk assessment methods usage. The only statistical study found is not comprehensive because: (1) Geographically focused only on France; (2) Takes into consideration only methods used in small or medium private businesses and hospitals; and (3) Risk assessment methods are catalogued in only 4 classes: “ISO”, “EBIOS”, “Mehari” and “Others” (CLUSIF, 2010). Despite these limitations, the statistical study has been reviewed as it was the only study found with such data. Interestingly, the results show how the vast majority of the surveyed entities that actually do perform risk assessment as part of their risk management process, use the ISO method or ‘Other’, with a minimal use of the other two French methods (EBIOS and Mehari) despite their local origins. The full inventory of RM and RA methods, created during the background research, already takes into account several criteria at different levels: Requirements for the original document selection, and requirements on its content. The proposed further rules aim at the creation of a 54

more manageable sample without renouncing to its representativeness. The sampling process is then used to further process the RM/RA list and analyze the identified methods under the lens of the following rules: • • • • • Methodology must be public (not necessarily free); Methodology must include at least one risk assessment method; There should be documented usage by multiple entities in multiple countries; Public availability of documentation in at least one of the three currently most spoken languages in the world: English, Spanish, and Chinese; If two or more methods are compliant or compatible, only the method with greater and broader feature coverage is retained in the list. 2.2.3 Output data This inquiry analyzes the characteristics of security methodologies for information systems and Net Assessment, with a focus on features coverage. The principal metric used is the presence of identified characteristics. While “one of the distinctive challenges of qualitative research is that the data produced tends not to take standardized form” (White, 2008), the final results of this research will be presented in table format. An analysis of the conceptual description of Net Assessment is necessary for a full understanding of the inquiry results, as this process is quite different from current risk management methods; however it can be helpful to use the table presentation as a reminder and quick visualization tool. For this reason, all the categorizations, coverage, and comparison analyses results have also been presented in tables.

3. Discussion
Considering current worldwide regulations, standards and best practices, the theory founding information and cyber security is quite consolidated nowadays. The basic concepts on which best practices are based are normally not challenged in the scientific literature, and constitute an accepted common knowledge for security practitioners.


Our discussion of the research problem is divided in two different parts. The first part aims at the challenges related to Q1 (What aspects are covered by current information security methods?). The second part addresses Q2 (What aspects are covered by Net Assessment?), and Q3 (What aspects does Net Assessment include that are not present in current information security risk assessment and management methodologies?). Part I begins from the concepts already presented in the literature review. Considering the already identified risk assessment methods in use, the first step is to select a sample to put under study. The second step is to analyze in a critical manner such methods, with respect to the characteristics identified in previous research and related to the questions of the present study. The comparative analysis will lead to the confirmation of the presence of specific characteristics and the specification of certain attributes. Part II initially identifies characteristics and aspects of the Net Assessment process in terms of its original purpose, scope, differences with other assessment methods, and utility through a analysis of available literature. This analysis is then followed by an attempt to frame such process inside the risk management characteristics previously identified and to tailor its use to the information field, and identifying potential aspects of improvement over the current methods.

3.1 Part I – Selected information security methods
In the background research performed in chapter 1.2, risk management and assessment methods in use were identified. The present step calls for a purposive sampling of methods, described in chapter 2.2.2, which maximize coverage of featured characteristics. The following list is a subset of the identified risk management and assessment methods in use which takes into account only those fulfilling the primary selection rules: • • • • • • ARIMA (Leitner & Schaumuller-Bichl, 2009); AS/NZS 4360 standard (AS/NZS, 2004); AURUM (Ekelhart, Fenz, & Neubauer, 2009b) and (Ekelhart, Fenz, & Neubauer, 2009a); Austrian IT Security Handbook (Bundeskanzleramt, 2004); CORAS (Aagedal et al., 2002); CCTA Risk Analysis and Management Methodology CRAMM (Yazar, 2002);


• • • • • • • • • • • •

EBIOS, Expression des besoins et identification des objectifs de sécurité (EBIOS, 2010); IRAM, Information Risk Analysis Methodology (IRAM, 2011); ISO/IEC, Information technology — Security techniques family, (see chapter 2.2.1); ISRAM, Information Security Risk Analysis Method (Karabacak & Sogukpinar, 2005); IT-Grundschutz (IT Baseline Protection Manual, 2008); LRAM, Livermore Risk Analysis Methodology (Guarro, 1987); Magerit (Magerit, 2006); Mehari (Mehari, 2010); NIST RMF family (see chapter 2.2.1); OCTAVE, Operationally Critical Threat, Asset, and Vulnerability Evaluation, (Woody, 2006), (Caralli, Stevens, Young, & Wilson, 2007) and (Alberts, Dorofee, 2001); RAMCAP, Risk Analysis and Management for Critical Asset Protection (ASME-ITI, 2006); and VAR, Value at Risk, (Jaisingh & Rees, 2001).

The following list is a subset of the previous which takes into account only those fulfilling the purposive sampling rules: • • ISO/IEC family (including all relevant standards referenced in 2.2.1); and NIST family (including all relevant standards referenced in 2.2.1, (Stoneburner et al., 2002), (Ross, 2011), and (Dempsey et al., 2010)).

The previous list includes two families of standards for information risk management and assessment. It is necessary to include risk management and risk assessment methods in the same list as both processes in fact include different activities depending on the specific standard or the specific method under scrutiny. However, the combination of a risk management method with a risk assessment method identifies a set of borders that are comparable among all the analyzed methods, hence the decision to prepare a list that would contain both risk management and risk


assessment methods. Among those methods, the ones with wider coverage have been selected following the declared sampling method and objective. 3.1.1 Identification of Characteristics To compare both RM and RA methods at the same time and identify missing elements, the characteristics we want to analyze must be broken down into broad categories and not specific implementations. All the identified assessment methods of RA and RM methods have then been analyzed following the steps enumerated in chapter 2.1.4 to create a single list of characteristic categories to be used as a checklist for coverage verification. The evaluation process was based on documents and led to the identification of numerous characteristics. The list of characteristics has been split in two different lists on the basis of their properties. The first list represents general or high level characteristics of the methods. The second list contains a number of technical characteristics for which also a qualitative appraisal should be made as to their effectiveness. Such categories apply to RM and to RA as a whole. The two lists are presented below as table 1 and table 2.

Table 1: RM and RA methods general characteristics

Category Scope Purpose Strategy, Governance & Responsibilities Completeness

Description The scope covered by the method and its products. The purposes of the method and of its products. The coverage of the method of all abstraction levels including strategic (governance, management, operational), and tactical for what concerns alignment and responsibilities assignment. Refers to the coverage of all aspects of a system's information security, summarized as: information technology completeness, information security completeness and risk approach completeness (Eloff et al., 1993) The sum of characteristics of a method that make it suitable for employment in a particular situation and/or organization. Such characteristics are identified as affected by organizational structure, size and philosophy, level of faced risk, available implementation time, etc.

Range of Applicability


Category Flexibility

Description Refers to the capability to match an organization’s security needs. This often means compliance with the security policy and the business objectives. The possibility to augment a specific method with other features and/or to integrate it with other methods (e.g. for quality control, auditing, governance, CASE tools, etc.). Possibility to tweak and implement a specific method in a personalized way, customized upon a specific organization and situation. Possibility to be applied both to pre-existing information systems and new ones, respecting their own configurations (e.g. centralized vs. distributed systems). The complexity of implementation and easiness of use of a method. Complexity can be determined by considering a number of factors, including the number of required people, the level of required expertise, the availability of software tools for automation, the tools complexity, the need for training. The property of a method to address issues at the needed level of granularity. Level of residual uncertainty or ambiguity that remains after available evidence is considered. Level of automation afforded by the method, at every abstraction level and every management level.

Complexity & Usability

Granularity Uncertainty Automation Execution Time

Risk management is a process which parallels the lifetime of an information system. This characteristic refers to the frequency of its main deliverables, and the time required for specific products (like a risk assessment) to be prepared. The ability of a method to produce relevant, reliable and accurate Consistency & Validity results, irrespective of the certainty, validity, subjectivity or completeness of the data available as input and regardless of the method of their estimation and the assessor. Presentation, Communication The form used to present results and products of specific actions of the method. Includes the process to communicate and present risks & Awareness at every layer of the organization hierarchy and to build awareness. The ability of a method to produce and report clear, precise and Clarity well-justified recommendations to exec level management. Ability to allow the managers to understand all of the issues, proposed measures, and their respective main drivers. The form and content of security-related documentation required Documentation by the method. The cost of implementing a specific method. The cost may include Cost the purchase of a proprietary methodology, purchase of software, hiring of consultants, training, labor costs, etc.


Table 2: RM and RA methods technical characteristics

Category Assessment Type Underlying Model

Description The type of assessment(s) available for implementation through the method. The standard security model employed by the method. Ability to be recognized by management as credible, sound, complete, correct, reliable and relevant. If and how the method is able to identify and characterize the organization’s assets to be protected. If and how the method is able to identify, characterize and factor in the whole process the external environment in which the organization operates (laws, threat agents, etc.) The ability of the method to identify security objectives, also in relation to the organization’s business objectives. If and how the method is able to identify security vulnerabilities present in the system. If and how the method is able to identify and characterize threat agents. If and how the method is able to identify and characterize threats. If and how the method provides instruments for calculating qualitatively or quantitatively the exposure of the organization to risk. If and how the method is able to identify and characterize risk. If and how the method is able to evaluate and assess risk, and identify alternate possibilities. If and how the method includes procedures to identify, select, prioritize, optimize and plan the implementation of countermeasures. If and how it includes the ability to assess the identified scenarios from multiple different views and evaluate the outcome of the implementation of different sets of countermeasures. Ability to estimate the residual risk in a given scenario with a specific set of countermeasure implemented. Ability to perform consistently risk monitoring for the STS and institute process for triggering on-event actions.

Assets Identification & Characterization External Environment

Security & Business Objectives Vulnerability Identification Threat Agent Identification & Characterization Threat Identification & Characterization Exposure Assessment

Risk Characterization Risk Evaluation & Analysis Countermeasures Identification & Prioritization

Residual Risk Identification Continuous Risk Monitoring

The two sets of categories represent the characteristics used for assessing security and are derived from studies of RM and RA methodologies. It is then possible to say that these lists represent boundaries and functionalities of RM and RA methods, which can be used for comparison with the Net Assessment process. Tables 1 and 2 can be considered a comprehensive answer to the Q1 (What aspects are covered by current information security methods?) However,


before continuing to analyze the Net Assessment process, we must consider one more aspect related to the RM and RA side: the coverage of such characteristics. 3.1.2 Characteristics coverage It is not enough to check what the RM and RA methods cover and if NA covers more or different categories, but also how those aspects are implemented. Such characteristics can be very different from one RM or RA to another, thus how such categories are implemented becomes very important for the analysis performed in answering Q3 (What aspects does Net Assessment include that are not present in current information security risk assessment and management methodologies?). Therefore it is in this chapter where the selection of RM and RA methods comes into play. The rationale for this approach relies in the focus on practical improvements instead of theoretical, and in practical constraints, since it is extremely hard to mix different RMs. In fact, there is no evidence that different RM techniques can be successfully combined. It is possible to augment or complete a RM method with additional characteristics, upon necessity, but to mix different ones is impractical and not pursued in real life scenarios. Thus, in this research we have assessed existing individual RMs that cover a wide spectrum of categories vice fictional compilations of RM characteristics and implementations. ISO and NIST RM methods are used in practice as the foundation for all other RM methods that have been assessed during this study. In fact, all other risk management methods claimed compliancy towards either of these two standards. We are not concerned with the specific implementation of such standards, such as Magerit, or other specific implementations. We simply need to analyze what these two family frameworks allow and what are their respective approaches. The table 3 and 4 presented below represent the summary of the assessment of NIST family documents towards the selected characteristics.


Table 3: NIST general characteristics coverage

Category Scope

NIST family The scope covered by the NIST method includes risk management and assessment of ‘information systems’ (intended as a discrete set of information resources organized for the collection, processing, maintenance, use, sharing, dissemination, or disposition of information). The purpose of NIST documentation is to provide guidance concerning risk management, assessment and monitoring. Security strategy is associated to an executive role but appear disconnected from business strategy. Security governance is detailed in a hierarchy or roles. Responsibilities are extensively detailed and associated to roles. Information technology aspect: complete, Information security aspect: complete, Risk approach: Comprehensive but not complete as it doesn’t include quantitative risk assessment guidance and other technical methodologies. Applicable to every type of organization from large to SMEs. Some characteristics of the method dissuade from its use (or at least without any augmentation) in some specific contexts. It does provide methods for alignment with other IT processes, but not with business quality or maturity processes. It cannot be easily integrated with other risk assessment methodologies. The whole method is aimed at minimizing the work and technical expertise of the implementers, at the expense of a pre-packaged solution. The method is considered as highly usable concerning the security specific expertise. The required analysis techniques and the support documentation are not too onerous in terms of technical know-how. The method implementation however is complex because of its responsibilities allocation and chain of command. The method addresses issues through the use of lists. Such lists include asset and control lists at every level of the organization. The granularity of the method render difficult to correlate: o Vulnerabilities with controls, and o To correlate threat agents with risks. The method doesn’t address this issue. The method is not supported by a software tool to expedite the analysis method. However, some documents are nothing less than long sample or check lists, which help in checking completeness and enforcing auditing. Variable depending on the depth of the method implementation. Generally speaking the single actions can be performed in any range of time, from quick to long. The organizational implementation though is quite extensive and if fully implemented will take long time to complete the processes. Communication procedures are included only among those roles involved in the method implementation. Presentation requirements are not set. Awareness is not included out of involved roles. The method minimizes the technical complexities of the assessment and thus may have quite clear reports. Executive level management may be reported simple countermeasures suggestions, as all the other steps are shielded and managed at lower levels. Mid management also is shielded from the concepts of threat, threat agent and the overall risk analysis process.

Purpose Strategy, Governance & Responsibilities Completeness

Range of Applicability Flexibility

Complexity & Usability


Uncertainty Automation

Execution Time

Presentation, Communication & Awareness Clarity


Category Consistency & Validity

NIST family Considering that the method takes as input mainly internal data, that it makes use of checklists and that the risk analysis is not very detailed, it is possible to say that it is quite consistent and its result be replicated by different assessors. The results can be considered valid in the limited scope of the method. Present. The method drives towards the production of two different sets of documentation. One internal for the system operation and one aimed for certification and accreditation purposes. Method purchasing: free. Method software: None specific. Labor costs and training costs: variable; may be high depending on the size of the business and the depth of implementation. External experts are extremely suggested as the whole method is complex and require deep understanding before implementation: Potentially expensive.



Table 4: NIST technical characteristics coverage

Category Assessment Type Underlying Model Assets Identification & Characterization External Environment Security & Business Objectives Vulnerability Identification Threat Agent Identification & Characterization Threat Identification & Characterization Exposure Assessment Risk Characterization Risk Evaluation & Analysis

NIST family Qualitative. The reference assessment model employs three levels or risk categorization. The NIST risk management model adopts a risk driven model. The model is not particularly elaborate; it adopts the basic attributes deriving from the definition. Present. Include recommendation for many types of information assets. The recommendations provide the asset inclusion in a specific Security Category, providing three values for the relative confidentiality, integrity and availability. Not present. Boundaries are defined for the system, and only what is the scope of those boundaries is characterized and analyzed. The method connects security requirements with mission requirements at system level, but there is no clear connection with business objectives at organizational level. Present. Refer to external (non-NIST) vulnerability sources. Present. Includes samples. Supported attributes: o Motivation/Situation; o Method. Present. Includes samples. Supported attributes: o Likelihood of occurrence; o Impact. Not present. Risk exposure at the overall level or specifically for critical assets is not included. Present, qualitative, in three levels calculated directly from the threat characterization. Present. Risk analysis is detailed. Risk evaluation includes guidance for a cost benefit analysis related to risks.


Category Countermeasures Identification & Prioritization

NIST family Present. A set of documents include a checklist of countermeasures for the identification. Relationships among countermeasures or scenarios are not envisaged. Countermeasures selection, approval and implementation planning steps are described in detail. Present. Residual risk is calculated after the implementation of countermeasures. Partially present. The method includes an information system continuous monitoring, structured as an IT process; however it defines the structure at large and does not specify its content requirements.

Residual Risk Identification Continuous Risk Monitoring

A summary of the ISO assessment is provided in the following table 5 and table 6.

Table 5: ISO general characteristics coverage

Category Scope Purpose Strategy, Governance & Responsibilities Completeness

ISO family The scope covered by the ISO method includes the implementation of an information security management system that embraces the implementing organization. To set standard and guidance in implementing an information security management system, embedding security in the life cycle of the organization. Security strategy is associated to executive roles but appear disconnected from business strategy. Security governance is detailed in a hierarchy or roles. Process responsibilities are included. Information technology aspect: Complete but not detailed, Information security aspect: complete, Risk approach: complete. Applicable to every type of organization from large to SMEs, while for the latter it may be impractical. Better suited for organizations that need a comprehensive approach and need to employ techniques for the minimization of the risk. It is easily connected to other ISO standards for quality management (from which it derives), maturity measurement, It and security auditing. It can be integrated with the use of multiple, diverse, risk assessment methods. The method is considered very usable. The method implementation can be adjusted on the organization’s needs. The technical analyses can also be selected among a wide and diverse range. While the method is best suited for large organizations facing high risks, it can be adapted for simpler scenarios. The method addresses the governance and management level of the risk management process. The strategic (highest) and tactical (lowest) levels are not detailed much and granularity is very coarse at these levels.

Range of Applicability Flexibility

Complexity & Usability



Category Uncertainty

ISO family It may be addressed at the level of risk assessments through the implementation of specific methodologies that include this aspect. However it is not addressed at risk management level. The method is not supported by a software tool to expedite the analysis method. However it allows for use of multiple external analysis techniques, which may be supported by specific tools. Variable depending on the depth of the method implementation. Even in a mild implementation it will require a distinct amount of time. It is not the fastest method around. All the three aspects are included in the method, but not detailed.


Execution Time

Presentation, Communication & Awareness Clarity

Consistency & Validity



The documentation produced following the ISO standard is particularly comprehensive and not explicitly intended summarized for the executive management. The use of complex risk assessment methods may further complicate the products. The documentation produced in the process will probably require a medium to high degree of expertise. The use of provided checklist can improve the repeatability of results; however, the ample range of supported and available risk assessment methods can lead to significant differences. Thus the validity is highly dependent on the considered scenario and may be dependent on the technical abilities of the performer should be a complex mathematical model be selected. Present. Specific for risk management purposes. Technically detailed. There are requirements for communicating with users, but not regarding communication with the executives. Method purchasing: Circa $120.00 per document. Method software: none specific. Labor costs and training costs: variable; may be high depending on the size of the business and the depth of implementation. External experts may be useful to implement complex risk analyses: potentially expensive.

Table 6: ISO technical characteristics coverage

Category Assessment Type Underlying Risk Model Assets Identification & Characterization External Environment Security & Business Objectives Vulnerability Identification

ISO family The type of assessment(s) depends on the assessor’s choice. The guidance has a long list of diverse assessment types. The model is process driven. It employs the standard threat-vulnerability-risk model in a process driven approach. Present. It is described in general lines; however it is mainly dependent on the risk assessment method used (see ISO 31010 standard for some applicable methods). Present. Third parties and external environment are part of the analysis. Yes. The method’s process begins with the assessment of business processes and objectives. Action present. Refers also to a checklist.


Category Threat Agent Identification & Characterization Threat Identification & Characterization Exposure Assessment Risk Characterization

ISO family Present. It is described in general lines; however it is mainly dependent on the risk assessment method used. Present. It is described in general lines; however it is mainly dependent on the risk assessment method used. Present. It is described in general lines; however it is mainly dependent on the risk assessment method used. Present. It is described in general lines; however it is mainly dependent on the risk assessment method used. A sample chart indicates 5 levels for likelihood and 5 levels for impact. These are then used to first measure the risk level and then rank the risks. Present. The standard identifies the following alternatives: o Reduction, o Retention, o Avoidance, o Transfer. A process for the countermeasures identification is present and described in general lines. The activity of countermeasures prioritization is cited only once though. This lack of guidance on countermeasure prioritization is due to the effort in risk prioritization. The method focuses in prioritizing risks, while it leaves the burden or prioritizing countermeasures to the assessment. Present. Residual risk is calculated after the implementation of countermeasures and may trigger anew process. Present. It is included in the overall Information security management system (ISMS) definition.

Risk Evaluation & Analysis

Countermeasures Identification & Prioritization

Residual Risk Identification Continuous Risk Monitoring

With both the NIST and ISO standards analyzed, we have conclusively answered Q1 (What aspects are covered by current information security methods?) in relation to RM and RA methods. The second part of this discussion focuses in answering Q2 (What aspects are covered by Net Assessment?) and Q3 (What aspects does Net Assessment include that are not present in current information security risk assessment and management methodologies?) through a critical analysis of Net Assessment.

3.2 Part II – Net Assessment methodology
While public documentation on Net Assessment is not scarce, it is indeed very general and apparently drawn from very few original sources. However, after a careful analysis of many documents, spanning on various topics from defense strategy, nuclear warfare, historical accounts, and university dissertations and lectures, it was possible to delineate its main


characteristics quite clearly. The following chapter is devoted to the qualitative analysis of selected documentation on the topic. 3.2.1 Critical analysis of Net Assessment Net Assessment is a holistic, multidisciplinary approach, which includes analysis of future trends, involves simulations, opposition analysis, critical reviews, and low-probability, highimpact contingency planning. If not the first creator of the methodology, Andrew Marshall is at least its spiritual father (Karber, 2008). Net Assessment today comprises a number of different aspects that are generally focused on military objectives as this methodology is usually employed for defense purposes. However, it is easy to see how NA could also fit the national security field, and especially in a cyber-context of the private sector. Definition For the purpose of this inquiry, it is essential that NA is defined, described, and analyzed at least in its functional characteristics in order to identify potential areas of benefit or contribution in the field of information security. While the whole NA process has been described by practitioners and scholars, there is not a publicly-available, unique, singular document or source of documentation to be used as a point of reference. Net Assessment has not been codified, or at least publicly released, as a standard. As indicated above, information on NA has been collected through documentation analysis to include speeches, articles, research, books, and essays on Net Assessment created in the last 40 years by principal practitioners and various scholars with direct experience on its application. Various reports using NA have also been analyzed. Considering that there is no single point of reference for NA, it is worth proposing multiple definitions, selected among those found in the source documents: Def. 1: “Net Assessment is a systematic method of analysis that fulfills the need for an indirect decision support system and provides a major input to the strategic planning/management system in the Department of Defense” (Konecny, 1988). Def. 2: “Net assessment is a method of broad analysis normally characterized by simultaneously focusing on two or more competitors or opponents through a comparative process” (Marshall, 1976).


Def. 3: “Net Assessment (NA) as applicable to national security can be defined as comparative analysis of military balances between competing states or even as-yet-friendly but potential competitors. It covers a long enough period, both the past and the future, to be able to identify trends in instruments of national power, endemic strengths and vulnerabilities of the competitors. But the most important fact is that NA process deals with issues well beyond current military capabilities” (Dahiya, n.d.). Def. 4: “The best way to define net assessment is to understand that it is a practice. It isn’t an art (like military judgment), nor is it a science (like chemistry). Rather, it’s a way of tackling problems from certain distinctive perspectives that involve skills that can be improved. Any “practice” is made up of several skills. A business manager, for example, needs to know how to control costs, satisfy customers, and plan for where his company will be in the future. He uses accounting, marketing, and forecasting. Each of these elements offers a particular perspective on the business, and each also involves certain skills” (Bracken, 2006). All the previous definitions help in identifying some key aspects of NA, through the identification of its position in the organizational structure (input to strategic planning), main field of use (comparative analysis), and the recognition of its status as a “practice”, composed by several different perspectives and analytic tools. The following chapters will identify in more detail what NA does, its purpose and scope, differences with other comparable assessment types, characteristics, process, and utility. Purpose and scope Net Assessment cannot be superimposed with strategic planning, while it is clear that it does consider strategies, it is not a tool for creating strategy or policy recommendations; rather, it is focused on identifying and defining features and aspects of a good strategy (Bracken, 2006). NA can be used by senior executives and their advisers, as tool for evaluating external security threats and identifying strategic opportunities. Furthermore, Net Assessment is not a panacea. It is applicable to a limited set of problems, which include any problem involving competition (e.g. Red vs. Blue for the military, or internal sources vs. contractor for a private company) (Bracken, 2006).


NA has been used to ease a specific problem: “The future is always full of uncertainties. A common error is to underestimate the scale and multiplicity of the uncertainties” (Marshall, 1991). In the real world and in many practical cases, it is true that the main problem cannot be traced back to wrong estimates, but to the complete lack of estimates. Net Assessment improves the chances to identify and study issues that are important but often overlooked (Bracken, 2006). Net assessment is a multidisciplinary approach, defining a framework for the evaluation of longterm strategic political-military interactions with other entities seen through the lenses of competition. NA is a diagnostic tool that helps identifying strategic asymmetries between entities and environmental opportunities, with the ultimate objective of supporting senior executives in the strategy definition process (Skypek, 2010). The scope, or emphasis, of the assessment include at least 6 different areas: Political, military, economic, social, information, and infrastructure (Hannan, 2005). It also considers a number of different factors depending on the specific analysis. In literature, the following factors have been found: Technological change, military systems evolution, operational innovation, organizational adaptation, defense capabilities including military strength, strategy, programs, doctrine, operational concepts and tactics, demography, social cohesion, economic strength, technology application, diplomacy, policies, strategic culture, defense preparedness and policy. Therefore, it is clear that Net Assessment analyzes several aspects of every issue and tackles them in a holistic way, not as black boxes (Skypek, 2010), (Dahiya, n.d.). Differences with other assessment methods Net Assessment can be compared with other multiple types of assessments used for similar purposes, and shows significant differences both at the structural and content level. The main differences of NA with other modes of defense analysis are three structural level peculiarities: • • • Comparative Diagnostic Evolving.


NA in fact analyzes both sides’ capabilities together “in order to identify strategic asymmetries and areas of comparative advantage” (Skypek, 2010). Second, it is diagnostic rather than prescriptive (Karber, 2008), (Skypek, 2010). As example, NA is different from threat assessment, as the latter is an appraisal focused on the adversary’s intentions and capabilities, and does not consider the assessor’s side in the equation (Skypek, 2010), (Konecny, 1988). It is fundamentally different from and complementary to Long Range Planning, and they have been described as “diagnostic comparative analysis and prognostic, diachronic trend projection” (Karber, 2008). Last, it can also be seen as evolving as it makes use of trend analysis, and while it is forward looking, the analysis takes into consideration history and historical data in a continuous update (Konecny, 1988). At the content level, the main differences with other assessment methods are (Dahiya, n.d.), (Bracken, 2006): • • • Deeper analysis Multidisciplinary approach Holistic approach.

While traditional analysis techniques focus on statistical inputs, NA allows for a deeper analysis as it tries to consider the entire range of aspects that make up the condition of competition (Konecny, 1988). Marshall argued that NA “differed from traditional military assessments because political, economic and technological variables would be accounted for, not simply technical estimates of an adversary’s force structure or weapons systems” (Skypek, 2010). NA can be considered a multidisciplinary approach not only because makes use of different tools in its inquiry, but also because it takes different points of view as a useful way to better understand the issues at hand (Dahiya, n.d.). NA focuses on strengths and weaknesses of competitors through a holistic examination and analysis of factors with the aim of highlighting a range of probable alternate futures that need to be considered by the strategy planners to maintain advantage against competitors under all probable contingencies (Dahiya, n.d.). A further difference with automatic tools and methodologies such as game theory (GT) or social network analysis (SNA) is that “in place of modeling complex and thinking simple, net 70

assessment tries to model simple and think complex. The spirit is one of using relatively simple models, numbers, and trends, and to think long and hard about what they mean” (Bracken, 2006). Main characteristics In business, as much as in defense, there is a need: “you need to study the opponent, whether it is a terrorist cell or a company doing business with your department. Improve your understanding of how they see the world, what metrics drive their behavior, and so forth” (Bracken, 2006). Net Assessment is a tool to answer this need. It is important to present its characteristics before its process for two main reasons: First, there is no standard procedure for conducting net assessments, despite it can be considered a linear deductive reasoning process (Krepinevich, 2008); second, Net Assessment uses a number of different analysis methods (Konecny, 1988) and the quality of its results depend heavily on the quality of the input data and the level of uncertainty related to each level of the assessment. In a typical NA model, are considered at least 3 entities: 2 players with their individual goals, resources, characteristics, and strategies; and the ‘context’, which cannot be controlled by any of the players, for example, technology, climate, etc. Datasets for the model include long-term trends of identified variables, and enduring asymmetries between the players. Thus, Net Assessment serves as a tool to identify the characteristics of a good strategy. One that will accommodate all these aspects, reflecting identified trends during time, able to exploit the identified asymmetries, and providing advantages in achieving strategic goals. A good strategy will address adversaries, the environment, and will tend to limit the risks posed by present uncertainties (Zarate & Sokolski, 2009). From the structural point of view, Net Assessments have been divided into several categories (Konecny, 1988) and (Skypek, 2010): • Balance Assessments (further divided into): o Functional, and o Geographic areas. • • Policy Assessments; Net Technical Assessments;


• • •

Comparative System Evaluations; Operational Net Assessments; and Weapons comparisons.

The scope of these assessments is extremely broad but they still incorporate detailed analyses. They are updated periodically, and usually include a treatment of the considered qualitative factors (Pease, 1983). Bracken identifies 6 main perspectives of Net assessment and associated skills: • • • • • • Strategic Interactions; Longer Time Spans; Getting Things Right with a Little Thought; The Importance of Socio-Bureaucratic Behavior; Strategic Asymmetries; and The Multifaceted Nature of Strategy,

And he further states that “it is their combined application that distinguishes Net Assessment from other analytical frameworks” (Bracken, 2006). Bracken defines Net Assessment as a strategic framework composed by these six perspectives and implemented through a multiplicity of analytical tools, like “scenarios, war games, trend analysis, and considered judgment” (Bracken, 2006). He also identifies another fundamental characteristic: that NA can be applied without a high degree of problem structure, as opposed to systems analysis, which requires considerable problem structure. NA process As previously said, there is not a unanimous consensus on a fixed assessment process. It has been described differently in multiple documents that have been reviewed, starting from a simple list of characteristics, to the structured description of Sun Tzu’s “Five Strategic Arts”, whose five steps have been identified as a possible rubric (Karber, 2008) for Marshall’s multi-tiered 72

analytical framework: (1) Measurements; (2) estimates; (3) analysis; (4) balancing; and (5) triumph (Krepinevich, 2008). The process description that we selected has been drawn by John M. Collins, who contends that there are four basic phases to any assessment: Compile, certify, combine, and compare (Collins, 1980). Therefore Net Assessment can be described as a framework in which: • • Phase One: Collection of information about all the involved analyzed actors; Phase Two: Certification and validation of the collected data, to insure that it is accurate, objective, and as free of bias as possible; • Phase Three: "Considers characteristics on each side, first singly, then in combination, to ascertain intrinsic strengths and weaknesses" (Collins, 1980). “An effective starting point is to establish Measures of Effectiveness (MOEs). An MOE is defined as a quantitative expression of the extent to which specific mission requirements are attained by the system under study” (Taylor, 1984). • Phase Four: Compare, i.e. when all the concepts of a comparative study come into play. This is the most important and complex step in the Net Assessment process (Konecny, 1988).

While Net Assessment counts on specific characteristics and a basic structure that differentiates it from other assessment methodologies, it is not a fixed framework. “The assessor has significant creative license in how the assessment is conducted in terms of the questions asked and the methodologies employed” (Skypek, 2010). The assessor decides the structure, metrics, variables, and capabilities to be considered. The assessment is conducted with four pillars that need to be satisfied (at least in the defense sector): Trend analysis, doctrine, asymmetries or areas of comparative advantage, and scenarios, for testing identified hypotheses (Skypek, 2010). A notional outline of Net Assessment follows: 1. Political-Military Context for Analyzing the Competition


1.1. Trends in the Balance 1.2. Doctrinal Asymmetries 1.3. Analysis of Perceptions 1.4. Scenarios 2. Assessment of the Balance 2.1. Strategic Asymmetries 2.2. Environmental Opportunities 2.3. Impact of Third Party States or Alliance Systems 2.4. Issues and Questions that Require Further Exploration

The basic analytic product of NA has some similarity with the well-known Strengths Weaknesses Opportunities Threats (SWOT) analysis methodology, widely used in the business environment (Skypek, 2010). Value and utility Net Assessment expresses its biggest value as a tool allowing organizations which tend to focus disproportionately on short-term challenges to implement strategic thinking. In every field, too often there is a tendency to focus on day-to-day operations. Therefore Net assessment is a tool for facilitating and implementing long-term, strategic thinking. Net Assessment identifies strategic asymmetries between competitors, and these can be translated in areas of comparative advantage. These areas, once identified, can be codified in a strategy and then exploited: “by recognizing evils in advance (a gift granted only to the prudent ruler), they can be cured quickly; but when they are not recognized and are left to grow to such an extent that everyone recognizes them, there is no longer any remedy” (Machiavelli). Another of NA’s added value is that it embodies and builds upon the Clausewitzian perspective that war is a continuation of policy by other means and that “failing to consider military affairs in a broader political context can have catastrophic consequences for the state” (Skypek, 2010).


While the assessment is not prescriptive, it still is forward looking, and as such its main interest is in the future. Unlike many other assessment tools used to inspect the possible futures, NA better tackles the challenges of this type of assessments. It embraces the human aspect, promotes intellectual effort, curiosity, creativity, and invites reflection on uncertainties. While forward looking assessments’ input is usually composed of huge amounts of data in different forms, quantitative, qualitative, in complex structures, from different sources and thus conflicting, the results can often be flawed analyses and then ineffectual strategies. However, NA is built around a concept that minimizes those risks as “it brings order to the study of war and statecraft by decomposing complicated political-military relationships into understandable zerosum competitions” (Skypek, 2010). One of the questions that may arise challenges its usefulness in other sectors other than that of defense. As it has already been noted, “the purpose of net assessment is to provide executive level management with an appraisal of the state of affairs that affect the character and success of the total enterprise. Although emphasis is often placed on military analysis, the application of net assessment is just as functional in political and commercial arenas” (Konecny, 1988).

In order to highlight its utility, and to provide elements for direct analysis of NA’s final products, it is possible to point out two freely available technical reports based on NA, one by Cordesman and another by Drapeau and Wells (Cordesman, 2002), and (Drapeau & Wells, 2009). 3.2.2 Characteristics coverage Using the model for analysis presented in the previous chapter, tables 7 and 8 outline the NA’s covered characteristics. The tables present a summary of the NA process assessment and attempt to answer Q2 (What aspects are covered by Net Assessment?).


Table 7: Net Assessment general characteristics coverage

Category Scope Purpose Strategy, Governance & Responsibilities Completeness Range of Applicability

Net Assessment Net Assessment is used to perform an analysis on political, military, economic, social, information, and infrastructure aspects. The purpose of Net Assessment is to reduce uncertainty and provide long term guidance at the strategic level. The method does not include a management framework. It is a complex process itself but includes mainly risk assessment actions. It is devoted to report to executive level. The method is the most complete assessment process reviewed so far concerning the breadth and depth of investigation. The method is particularly suitable for long term planning and while it can have beneficial effect in any organization, it is more likely to be valuable in organization with very high security concerns and long term planning. The method is compatible with other supporting tools as it provides guidance only. It is highly flexible as it is up to the analysts to decide how to implement it, considering the received requirements. The products of the process are highly usable as they are drafted on specs for the executives. The method itself is usable as it requires only one group/layer of analysts in its most basic version. The complexity is dependent on the analysis performed and the input data sought after. Whatever analysis is chosen, analysts shall be experts in their fields. The granularity of the analysis performed in the process is decided prior the Net Assessment and is variable. The method is mostly suited for analyzing detailed information on a wide range of topics to provide the desired outcome. Net Assessment products usually contain also an estimate of residual uncertainty or ambiguity that remains after available evidence is considered. Automation is usually not available for most of the analytical process. However, it is possible to use specific tools to automate complex partial analyses (e.g. game theory assessment of specific scenarios). This characteristic is dependent on the product requirements and the selected assessment methodologies. It is usually longer than standard IT or information security risk analyses. These two aspects are usually set as product requirements and are therefore respected through the selection of apt analytical methods.


Complexity & Usability




Execution Time

Consistency & Validity

Presentation, Communication All these aspects are present. The main receiver being the executives. & Awareness Products usually allow the managers to understand the issues, and Clarity provide additional information for the decision making process. The process execution is documented upon necessity. The main Documentation final product focuses on the results.


Category Cost

Net Assessment Costs may include: o Labor costs of the analysts, o Purchase of input data, o External experts or training, o Additional software tools.

Table 8: Net Assessment technical characteristics coverage

Category Assessment Type Underlying Model

Net Assessment The Net Assessment process may include many different assessments, both qualitative and quantitative. It is construed as a comparative process making use of several more detailed analyses. Its systematic approach ensures the soundness of the model. Present, including both those of the organization and those of an identified competitor. Yes, usually widely accounted for. Yes. Yes, also with respect to specific possible scenarios. Yes. This step is performed as broadly and deeply as the internal analysis. Yes, with the identification performed mainly through the use of scenarios and the characterization through deep analysis. Yes, the method provides instruments for calculating qualitatively and quantitatively the exposure of the organization to identified risks. Present. The risks are identified and minutely characterized. Yes, the method is able to assess and evaluate the faced risk, and identify alternate possibilities. Usually not present. While this aspect can be introduced in the process, it is not part of the standard scope and purpose of the Net Assessment. Not present. The process doesn’t include the implementation of countermeasures. However, if scenarios are simulated, it is possible to infer the residual risk in specific conditions. Not in the purpose of the process. Achievable upon repetition of the process.

Assets Identification & Characterization External Environment Security & Business Objectives Vulnerability Identification Threat Agent Identification & Characterization Threat Identification & Characterization Exposure Assessment

Risk Characterization Risk Evaluation & Analysis Countermeasures Identification & Prioritization Residual Risk Identification

Continuous Risk Monitoring


As mentioned earlier, these two tables answer Q2 (What aspects are covered by Net Assessment?). The following chapter will attempt to answer Q3 (What aspects does Net Assessment include that are not present in current information security risk assessment and management methodologies?). 3.2.3 Net Assessment as a tool for information security In a rather comprehensive report, Kearney and Brügger mention and justify many shortcomings of the most widespread information security analysis techniques, identifying among others: The need for the concept of holistic security, the concept of human intervention, and stress the fact that the standard formula risk = threat x impact is too simplistic, mathematically and conceptually inadequate (Kearney & Brügger, 2007). Net Assessment, as described in the previous chapters, remarkably embodies all these qualities. Taking the previous tables as guideline, this chapter summarizes the major points of distinction between NA and classical IS RMs and RAs. One of the major differences is the scope of Net Assessment. This method is truly multidisciplinary and comprehends the whole range of possible direct and indirect entities and the external environment. NA allows for the analysis of intangible factors, not only quantifiable aspects like manpower and equipment, or other measures of capabilities. This is extremely important as far as strategy is concerned because “the real world outcome of war has been determined by intangibles” (Cordesman, 2002). The results of such an assessment can have a wide scope, including all the three parallel worlds in which we and our organizations interrelate: The physical world, the digital world, and “the virtual world (online interactive spaces like social networks, blogs, podcasts, and micro-sharing platforms with avatars and potential anonymity)” (Drapeau & Wells, 2009). The purpose of the NA is also quite different from that of the analyzed RM and RA methodologies. Risk management and assessment methods for information security aim at establishing a risk management framework, to guarantee organizational security through the analysis of risks, the monitoring of threats and the implementation of countermeasures. On the other hand, Net Assessment’s purpose is to perform a wide, deep, and accurate analysis towards the identification of future scenarios, possible risks, major contributors to those risks and their 78

consequences, and frame the identified results as long term guidance at the strategic level of an organization. Net Assessment is not an automated DSS (Decision Support System) (Druzdzel & Flynn, 2002) and should not be intended as such. However, it can ultimately work within a broader risk management methodology retaining much of its original purpose and scope. Such guidance can be tailored in a different manner in the context of information security applied in a corporate environment, for example to be aimed at the medium term. Further tailoring may involve the inclusion of recommendation as part of the purpose. This change will affect the now exclusively diagnostic nature of this instrument, but it may prove a useful addition. Another major difference is in the underling model. Net Assessment is firmly founded on a holistic model which encompasses not only many different fields, but also many different perspectives. While it may seem simple on one side as it can be reduced essentially to a comparative analysis, on the other side it should be recognized that the collection of supported assessments is omni-comprehensive, broad and highly detailed. It is able to employ methods from many other fields. For example, if we think about security risk, excluding natural hazards and non-malicious attacks in general, the rest, which is usually preponderant in any environment, is criminal activity. It stands reason that concepts and strategic theories from the criminal analysis field can be applied for threat analysis. Attackers are criminals. When they launch a cyber attack, they are performing a criminal act, and thus threat agents should also be analyzed from this point of view. None of the RM or RA methodologies tackle this aspect, but Mark Lowenthal (Lowenthal, 2009), Wayne Hall and Gary Citrenbaum (Hall & Citrenbaum, 2010), and Jerry Ratcliffe’s books (Ratcliffe, 2009) on intelligence and strategic thinking in criminal intelligence have many points that can be applicable. Net Assessment definitely represents a way to integrate these points of view. In Lowenthal, there is an entire chapter addressing intelligence and information analysis methods which are now common tools of the trade. Ratcliffe includes at least the following notable chapters: ch.1: the structure of strategic thinking, ch.6: the theory and practice of intelligence collection, ch.7: intelligence research, ch.8: exploratory intelligence tools, ch.9: threat and risk assessments that can bear direct significance towards an information security context. 79

Meanwhile, the manual by Wayne offers valuable concepts that will help tackling security from the point of view of the STS. The main advantage of NA’s underlying model is that it can host every type of assessment in the overall comparative framework and especially instill a new approach to systems’ analysis: “People who were systems analysts found it difficult to address the sorts of questions that we felt needed to be considered in strategic planning. To some extent, the systems analysts had by that time developed routine approaches to analysis and perhaps had ceased paying sufficient attention to the complex consequences of acquiring the systems they dealt with. Systems analysis proceeds by trivializing the measurement of effectiveness while perfecting the analysis and estimate of costs. The, often complex, consequences are not usually considered in the standard kinds of analysis” (Zarate & Sokolski, 2009). As Bracken puts it, Net assessment usually avoids complex computer models for the assessment of problems as, from its perspective, many of them are misleading, deliberately ignoring uncertainties on variables and their relationships and making assumptions to fit the model rather than to model real relationships (Bracken, 2006). Net Assessment differs substantially from both mathematical and formal methods. It “emphasizes that strategic interactions are shaped by the complex sprawling organizations that break big problems into manageable smaller ones” (Bracken, 2006). The security strategy of organizations today is mostly relegated to policy redaction and internal countermeasures implementation, and doesn’t attempt to map or asses the external environment for information security purposes. Net Assessment shows all its capabilities in this sector, as it allows the instruments for attempting a comprehensive assessment of external, environmental factors. The fact that they are not under control should not discourage their assessment, as the knowledge of the battlefield is fundamental for security planning and enemy evaluation.


Furthermore, nowadays IS RMs include continuous monitoring aspects but do not offer continuous updates on measures of risk, with risk being recalculated at each iteration at fixed times, that can be every 2 months, or 3 years. Threats, and thus risk, change dramatically on a continuous basis, as the world around us changes and evolves. Diplomatic relations can strain (see the Georgian conflict, or more recently the Iran problem), certain anniversaries can be source of malcontent (see the Estonian experience), new tools can be made publicly available (e.g. the release of DDoS tools) and many situations can affect the cyber and information security risk level. As much as the recurrence of Osama Bin Laden’s death will certainly temporarily increase the physical risk level in the U.S. That very event or other new ones may do the same concerning the cyber security level. Generally speaking, current information security efforts are mainly event-paced as opposed to time-paced. A pivotal article by Kathleen M. Eisenhardt and Shona L. Brown (Eisenhardt & Brown, 1998) explains very well the difference between the two approaches and shows through real life examples how big the influence of such a security approach is in the field of competition in dynamic markets. If we think about security incidents in terms of competition (organization vs. attacker) and set this competition in the dynamic world of IT, networks, services, and social relations (which ultimately build the STS which we are studying), such a parallel is appropriate. What does this mean in terms of minimizing risk? How can this lesson be used to improve the security of STSs? There is not a clear and univocal answer to this question. However, it surely indicates that further study should be conducted, and that models and methodologies that implement time-paced security efforts and recognize the importance of social interactions should be seriously considered at the strategic and governance level of STS. A critical difference related to the final product of the assessment is the presence of the exposure assessment, typically not performed by RM and RA methodologies, and not mandated or detailed in any of the two assessed standards’ families: NIST and ISO. Instead, Net Assessment provides the means to assess the exposure to risks, also making use of alternate scenarios. When employing alternative scenarios and risks exposure, the process usually involves the identification and assessment of consequences. While business continuity may ask for identification of operational consequences to information security risks, and the Chief 81

Information Security Officer (CISO) may be asked to identify the annual loss expectancy (ALE), these consequences are just a small window into the future, and do not give a clear picture of the overall organization posture as Net Assessment does. The whole comparative approach and its use of scenarios during the risk evaluation analysis with multiple assessment methods depending on the specific field considered is peculiar of Net Assessment. “NA employs tools like scenario planning exercises, strategic war gaming and cross-impact matrix analysis. These tools invariably involve testing of ideas, probable events and force application” (Dahiya, n.d.). NA’s risk evaluation seems perfect for the qualitative analysis that information security requires. Despite all the models and quantitative analysis proposed for information security risk assessment, none of them turned out to be accurate enough to be substantially implemented on a large scale. The most used quantitative methods are specific to financial loss calculations and usually result in wild estimates as intangibles such as information can rarely be quantified correctly. NA provides instead a natural environment in which to perform qualitative assessment with the available data. Furthermore, such assessments prove also to be more informative than those implemented through classical RAs as they also address uncertainty and provide measures and metrics to take into account ambiguity, both in the input data and in the results. They can apply risk evaluation methods like those proposed by Klinke and Renn, which includes assessment of advanced concepts like complexity, uncertainty, ambiguity, and ripple effects (Klinke & Renn, 2002). In their extensive research, Klinke & Renn modeled a risk evaluation process that can take into account not only the extent of damage and the probability of occurrence, but also the incertitude, ubiquity, persistency, reversibility, delay effect, violation of equity, and potential of mobilization (Klinke & Renn, 2006) and (Stirling, Renn, Klinke, Rip, & Salo, 2001). The same method would not be easily applicable inside one of the two assessed security management frameworks. Among the major differences identified, the validation step present in NA’s risk analysis merits also a special mention and can be considered a positive addition or improvement in current risk assessment and management methods. This NA’s step is devoted to the validation of the input data collected to perform the various analyses. While logic would dictate that it is a 82

necessary step and would be always performed in any condition, this is not the case concerning classic RM and RA. Partly this is due to the fact that they do not take into account uncertainty and thus there is less value in determining the accuracy of input data. However, this step should not be overlooked as an assessment’s result is only as good as the data in input.

4. Conclusions
Security of information and cyber systems is mainly implemented by organizations through the use of risk management. This practice is a substantial improvement with respect to previous methods like checklists and formal methods; however it is still not producing the expected results. How can it be improved? The literature review demonstrated how research in this area is particularly active and follows many different directions. After the analysis of two assessment standards’ families, the present study ascertained that some areas of fundamental importance for security are not fully addressed in the NIST and ISO standards. These areas, however, can be addressed through the tailoring and application of a method called Net Assessment, to be added to rather than substituted for NIST, ISO or other RAMs. Net Assessment has been fully described to give a reader an understanding of its inner workings and products. While the study could not show if such method will practically enhance the security posture of an organization, it certainly identified a number of potential improvements. In “Deterrence of Cyber Attacks”, Kugler, an authority in security affairs, policy and defense, calls for better Net Assessment of capabilities and vulnerabilities, arguing that “comparable analyses of capabilities and vulnerabilities in the cyber domain will be needed for a mature and effective U.S. cyber deterrence strategy” (Kugler, 2010). This study set forth the idea that such approach can be useful not only for cyber but also for information security at large, and may be of interest not only in the framework of national security, but also to private organizations. Its advantages over current methods would help not only in the deterrence, but also in mitigation through more sound decision making. Applied at lower level it can directly improve the risk assessment and, in turn, the technical security of an organization. At higher level it can indirectly improve the security posture of an organization by providing more awareness and better information for decision and strategy making.


4.1 Observations
The answers to question 1 and 2 were intermediate passages, functional to tackling the most interesting aspect of the study, which is represented by Q3 (What aspects does Net Assessment include that are not present in current information security risk assessment and management methodologies?). The answer to question 3 is extensively laid out in chapter 3.2.3. It would be redundant to summarize it here. However, a number of implications from that analysis can be pointed out: First of all, the study does show that Net Assessment involves concepts and results that complement those of current information security risk management and assessment methods, possibly improving their efficacy. It can improve current methods both at a low and a high abstraction level, providing an invaluable support in technical or qualitative estimation and a proven way of briefing high level decision makers. Practical possible benefits are: • • • • • • • More informed, more sound, security strategies; Better allocation of funding; More accurate estimates of risks and threats; Better understanding of the environment of operation, with possible benefits also in other non-security related areas; Better alignment between business objectives and security requirements; Better alignment between actual potential security risks and implemented countermeasures; Better inputs for real time and automatic tools eventually available.

These identified potential benefits may prove critical in improving security, practically minimizing security breaches and providing better tools for the identification of threat agents. While the aspects of how to implement such process or how much it will cost have not been addressed by this study, two other important issues can be tackled here referencing the critical analysis: (1) If NA implementation can create problems, and (2) What is needed to implement it. Concerning the first issue, considering both the Net Assessment process’ critical analysis in 84

chapter 3.2.1 and the answer to question three presented in chapter 3.2.3, there is no evidence of potential conflicts. Specific cases should be addressed singularly, and may create inconsistencies if not thoughtfully planned and implemented, however, at the methodological level, there is no identified problem of coexistence among these methods. Not unlike the two selected information security risk management and assessment methods, NA’s implementation should be analyzed in its specific deployment in an organization and in a specific situation. However, the main implementation’s requirements are skilled analysts. Differently from analysts currently employed in cyber and information security field, those required to perform an effective Net Assessment should include subject matter expertise also in other disciplines in order to be able to identify, collect, validate, assess, and hypothesize on a number of different subjects, including but not limited to extremist religion, fringe politics, criminal science, financial data, and demographics.

4.2 Extent and limits of investigation
The present study assesses a number of risk management practices with the aim of identifying areas for improvement and potential advantages in the use of Net Assessment. The present study was based on an extensive background research, involving several related fields of research and practice, and on a deep assessment and critique of current information security risk management methodologies and Net Assessment. The sampling approach should provide a high degree of validity and assurance that the research is significant for the most real life scenarios. The main limitation of the present research stems from an inherent problem of predictive research and the contingencies of the specific field. Considering: • • • The field of application: Socio-technical systems; The modality of possible implementation: Identification and selection of new procedures, processes, and personnel with a different skillset; That security cannot be measured ex-ante and cannot be proved through mathematical or logical constructs (yet) in such a complex environment. The present research thus cannot observe if the proposed method will in fact enhance security of the analyzed systems. Furthermore, the use of sampling and of the comparative methodology 85

implies that intermediate results can be considered valid only in the context of the methodologies compliant with those included in the present study. The intermediate results, i.e. Net Assessment differences with respect to other IS RAMs, are thus not generalizable. However, considering the large span of RM and RA methods compliant with the NIST and ISO standards, and the widespread application of those standards, the identified limitations are not particularly severe and the study has an intrinsic operational relevance.

4.3 Directions for future research
This study tackles the problem of addressing security concerns for Socio-technical systems. The findings of the present research shows that current security risk management standards and guidelines, while pursuing multiple strategies to increase effectiveness, still neglect to include a practice that has already been proven successful in other contexts, Net Assessment. While the methods used in this study observed some substantial differences with current RAM methods and possibly some avenues for their improvement, they do not assess whether Net Assessment will necessarily have a positive impact in the improvement of information security. Interactions among classical security strategy, defense assessments, and information and cyber security should be further investigated. Intelligence and security informatics (ISI) is a rising research field. It is a “an interdisciplinary research area concerned with the study of the development and use of advanced information technologies and systems for national, international, and societal security-related applications” (PAISI, 2011). Eventually, the other face of the same coin will become an increasingly important area of research, techniques for the study of and methods to improve the security of Socio-technical systems. Furthermore, it can be predicted that implementing a Net Assessment process in a nondefense environment will raise a major issue: Is it going to be cost effective? While the question of whether this process can effectively improve information security in a STS can hardly be tested a priori, this latter question on cost-effectiveness can absolutely be investigated and should indeed find an answer before a serious attempt would be made.


4.4 Implications for practice
While further studies will be needed to thoroughly assess feasibility, effectiveness and other properties of the identified possible improvements, organizations relying on Socio-technical systems and with special security concerns can already benefit from the results of this study. The identified process can be tried and tested, after a careful feasibility study, by a number of organizations which could potentially benefit by them. Such organizations can be identified among multiple diverse entities: • All public and governmental organizations whose reliability is based on the correct functioning of their technical support/main systems, such as critical infrastructure, law enforcement agencies, security agencies, and every agency whose operations have to be kept secure for national concerns. • All organizations that have a security approach consistent with the principle of “minimization of the risk”: Banks, private organizations and public entities whose value is substantial and primarily embodied by information and/or which operations have to be kept secure by mandate. • Cyber-insurance companies. As the ability to factor threat assessment in statistical models is complex, such a tool can improve their ability to build better insurance products.

4.5 Significance of results
The improvement of cyber security is one of the most important topics in the IT field nowadays, as testified by the number papers, books, and international conferences on this subject. However, no consensus has been reached on any specific type of risk management method. The issue has a high profile and a wide audience as it is of interest for governments, critical infrastructure, and private companies. This qualitative study is focused on information security, how it could be improved, and the use of Net Assessment for such a purpose. Through the performed critical analyses, this research has possibly broadened the understanding of how methods from other fields can be used to improve information security. This research raises a number of new questions for further qualitative and quantitative studies. 87

The results from this research add a novel approach to current information security research that is however well tested and with effectiveness proven through more than 70 years of employment by the U.S. Department of Defense (DOD) in a related field. It is the author’s hope that this research would spark debate and possibly be a stimulus for further research on the topic.

4.6 Code of ethics
The researcher is a member of ISACA and (ISC)² and currently holds active security certifications from both organizations, thus enforcing the code of ethics required by both entities. All information systems security professionals who are certified by (ISC)² are required to commit to fully support the (ISC)² Code of Ethics (COE, 2011). As a member and ISACA’s certifications holder, the Code of Professional Ethics is also enforced (CPE, 2011).



Aagedal, J. O. , Braber, F. den, Dimitrakos, T. , Gran, B. A. , Raptis, D. , & Stølen, K. . (2002). Model-Based Risk Assessment to Improve Enterprise Security. Proceedings of the 6th International Enterprise Distributed Object Computing Conference, 51-. Washington, DC, USA: IEEE Computer Society.

Al-Hamdani, W. A. . (2009). Non risk assessment information security assurance model. 2009 Information Security Curriculum Development Conference, 84-90. New York, NY, USA: ACM.

Alberts, C. & Dorofee, A. (2001). OCTAVE Criteria, Version 2.0 (CMU/SEI-2001-TR-016, ADA3399229). Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 2001. URL = <http://www.sei.cmu.edu/publications/documents/01.reports/01tr016.html>

AS/NZS 4360. (2004). Australian/New Zealand Standard. Risk management AS/NZS 4360:2004.

ASME-ITI. (2006). RAMCAP Risk Analysis and Management for Critical Asset Protection The Framework.

Aven, T. . (2006). A unified framework for risk and vulnerability analysis covering both safety and security. Reliability Engineering & System Safety, 92, 745 - 754.

Bertalanffy, Ludwig von. (1950). An outline of general system theory. The British Journal for the Philosophy of Science, 1(2):134–165, August 1950.

Bertalanffy, Ludwig von. (1950b). The theory of open systems in physics and biology. Science, 111(2872):23–29, 1950.

Bier, V. M. , & Abhichandani, V. . (2002). Optimal Allocation of Resources for Defense of Simple Series and Parallel Systems from Determined Adversaries, 129, 5-5. ASCE.


Billo, C. , & Chang, W. . (2004). Cyber Warfare An Analysis of the means and motivations of selected nation states. Office.

Board, D. S. . (2009). Report of the Defense Science Board 2008 Summer Study on Capability Surprise: Volume II: Supporting Papers. Presentations, II.

Bodeau, D. , Boyle, S. , Fabius-Greene, J. , & Graubart, R. . (2010). Cyber Security Governance.

Bodeau, D. , Fabius-Greene, J. , & Graubart, R. . (2010). How Do You Assess Your Organization’s Cyber Threat Level?

Bodeau, D. , Graubart, R. , & Fabius-Greene, J. . (2009). Improving Cyber Security and Mission Assurance via Cyber Preparedness (Cyber Prep) Levels.

Boote, D. N., & Beile, P. (2004, April). The quality of dissertation literature reviews: A missing link in research preparation. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.

Boote, D. N., & Beile, P. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher, 34(6), 3-15.

Bornman, W. G. , & Labuschagne, L. . (2004). A Comparative Framework for Evaluating Information Security Risk Management Methods. Peer-reviewed proceedings of the ISSA 2004 Enabling Tomorrow Conference. Republic of South Africa: South African Institute for Computer Scientists and Information Technologists.

Botta, D. , Muldner, K. , Hawkey, K. , & Beznosov, K. . (2011). Toward understanding distributed cognition in IT security management: the role of cues and norms. Cognition, Technology & Work, 13(2), 121-134. Springer London.

Boulding, K. E. (1956). General systems theory: The skeleton of science. Management Science, 2(3): 197–208, 1956

Bracken, P. . (2006). Net Assessment: A Practical Guide. Political Science, 36, 90.


Bundeskanzleramt. (2004). Österreichisches IT-Sicherheitshandbuch. Teil 2: ITSicherheitsmaßnahmen. Version 2.2, November 2004.

Burns, A. . (2006). Fearful Asymmetries: Herman Kahn’s Nuclear Threat Models and the DPRK’s Nuclear Weapons Program.

Caralli, R. A. , Stevens, J. F. , Young, L. R. , & Wilson, W. R. . (2007). Introducing OCTAVE Allegro : Improving the Information Security Risk Assessment Process. CMUSEI Tech Rep CMUSEI2007TR012.

Cebula, J. J. , & Young, L. R. . (2010). A Taxonomy of Operational Cyber Security Risks.

CFA aa.vv. (2011). Communist Chinese Cyber-Attacks, Cyber Espionage and Theft of American Technology. Hearing before the subcommittee on Oversight and Investigations of the Committee on Foreign Affairs. House of Representatives 112th Congress, First Session, April 15, 2011, Serial No. 112-14.

Clinton, L. . (2011). A Relationship on the Rocks: Industry-Government Partnership for Cyber Defense. Journal of Strategic Security, IV, 97-112.

CLUSIF. (2010). Menaces informatiques et pratiques de sécurité en France. http://www.clusif.asso.fr/fr/production/sinistralite/docs/CLUSIF-rapport-2010.pdf.

Code of Ethics (COE), International Information Systems Security Certification Consortium, Inc., (ISC)²®. Web Resource. Last accessed: October 2011. URL = <http://www.isc2.org/ethics/default.aspx>

Code of Professional Ethics (CPE), ISACA. Web Resource. Last accessed: October 2011. URL = < http://www.isaca.org/Certification/Code-of-Professional-Ethics/Pages/default.aspx>

Collins, M. John. (1980). US-Soviet Military Balance; Concepts and Capabilities 1960-1980. MacGraw-Hill, 1980.


Conrad, J. , Oman, P. , & Taylor, C. . (2006). Managing Uncertainty in Security Risk Model Forecasts with RAPSA/MC. Security Management, Integrity, and Internal Control in Information Systems, 193, 141-156. Springer Boston.

Cordesman, A. H. . (2002). Iraqi War Fighting Capabilities: A Dynamic Net Assessment.

Cremonini, M. , & Nizovtsev, D. . (2009). Risks and Benefits of Signaling Information System Characteristics to Strategic Attackers. J. Manage. Inf. Syst., 26(3), 241-274. Armonk, NY, USA: M. E. Sharpe, Inc.

Creswell, W. John. (2007) Qualitative inquiry & research design. Choosing among five approaches. Second edition. Sage Publications.

Crosston, M. D. . (2011). World Gone Cyber MAD. How “Mutually Assured Debilitation” Is the Best Hope for Cyber Deterrence. Strategic Studies Quarterly, 5, 17. Air University Press, Maxwell AFB, AL.

Crowther, K. G. , Haimes, Y. Y. , & Johnson, M. E. . (2010). Principles for Better Information Security through More Accurate, Transparent Risk Scoring. Journal of Homeland Security and Emergency Management, 7.

Cummings, M. C. , McGarvey, D. C. , & Vinch, P. M. . (2006). Homeland Security Risk Assessment Vol. II Methods, Techniques, and Tools.

Cutts, A. . (2009). Warfare and the Continuum of Cyber Risks: A Policy Perspective, 11. http://www.ccdcoe.org/publications/virtualbattlefield/04_CUTTS_national%20cyber%20risk %20v2.pdf.

Dahiya, B. R. . (n.d.). Net Assessment and Jointness. Integrated Defence Staff, India.

Dempsey, K. , Johnson, A. , Jones, A. C. , Orebaugh, A. , Scholl, M. , & Stine, K. . (2010). Information Security Continuous Monitoring for Federal Information Systems and Organizations.

Denning, D. E. . (1999). Information Warfare and Security (Vol. xvii, 522 p. : ill. ; 24 cm. ). New York : ACM Press ; Reading, Ma. : Addison-Wesley: Addison-Wesley. 92

Denning, D. E. . (2007). Assessing the Computer Network Operations Threat of Foreign Countries. Information Strategy and Warfare, 187-210. Routledge.

Denzin, N. K., & Lincoln, Y. S. (Eds.). (1994). Handbook of qualitative research. Thousand Oaks, CA, Sage.

DeWeese, S. , Krekel, B. , Bakos, G. , & Barnett, C. . (2009). Capability of the People’s Republic of China to Conduct Cyber Warfare and Computer Network Exploitation.

DHS aa.vv. (2009). Leftwing Extremists Likely to Increase Use of Cyber Attacks over the Coming Decade. Office of Intelligence and Analysis, DHS. Assessment. 26 January, 2009.

Drapeau, M. , & Wells, L. I. . (2009). Social Software and National Security: An Initial Net Assessment.

Druzdzel, M. J. , & Flynn, R. R. . (2002). Decision Support Systems. Encyclopedia of Library and Information Science, Second Edition. New York: Marcel Dekker, Inc.,.

Duggan, D. P. , & Michalski, J. T. . (2007). Threat Analysis Framework. Sandia National Laboratories. Albuquerque, New Mexico 87185 and Livermore, California 94550.

EBIOS. Last accessed: October 2011. URL = <http://www.ssi.gouv.fr/fr/bonnes-pratiques/outilsmethodologiques/ebios-2010-expression-des-besoins-et-identification-des-objectifs-desecurite.html>

Eisenhardt, K. M. , & Brown, S. L. . (1998). Time Pacing: Competing in Markets That Won’t Stand Still. Harvard Business Review.

Ekelhart, A. , Fenz, S. , & Neubauer, T. . (2009). Ontology-Based Decision Support for Information Security Risk Management. Proceedings of the 2009 Fourth International Conference on Systems, 80-85. Washington, DC, USA: IEEE Computer Society.


Ekelhart, A. , Fenz, S. , & Neubauer, T. . (2009). AURUM: A framework for information security risk management. 42nd Hawaii International Conference on System Sciences, HICSS '09. New York, 0, 1-10. IEEE.

Elahi, G. , Yu, E. , & Zannone, N. . (2010). A vulnerability-centric requirements engineering framework: analyzing security attacks, countermeasures, and requirements based on vulnerabilities. Requir. Eng., 15(1), 41-62. Secaucus, NJ, USA: Springer-Verlag New York, Inc.

Eloff, J. H. P. , Labuschagne, L. , & Badenhorst, K. P. . (1993). A comparative framework for risk analysis methods. Comput. Secur., 12(6), 597-603. Oxford, UK, UK: Elsevier Advanced Technology Publications.

ENISA, European Network and Information Security Agency. (2006). Inventory of risk assessment and risk management methods. Deliverable 1, Version 1, Date: 30/03/2006. Last accessed October 2011; URL = <http://www.enisa.europa.eu/act/rm/files/deliverables/inventory-of-risk-assessment-and-riskmanagement-methods>.

ENISA, European Network and Information Security Agency. (2007). Methodology for evaluating usage and comparison of risk assessment and risk management items. Deliverable 2, Version 1, Date: 26/04/2007. Last accessed October 2011; URL = <http://www.enisa.europa.eu/act/rm/files/deliverables/methodology-for-evaluating-usage-andcomparison-of-risk-assessment-and-risk-management-items>.

Farahmand, F. , Dark, M. , Liles, S. , & Sorge, B. . (2009). Risk Perceptions of Information Security: A Measurement Study. Computational Science and Engineering, 2009. CSE ’09. International Conference on, 3, 462 -469.

Fenz, S. , & Ekelhart, A. . (2009). Formalizing information security knowledge. Proceedings of the 4th International Symposium on Information, Computer, and Communications Security, 183-194. New York, NY, USA: ACM.

Frascati Manual. (2002) The Measurement of Scientific and Technological Activities. Proposed Standard Practice for Surveys on Research and Experimental Development. Published by OECD, 11 Dec 2002.

Friedman, A. . (2011). Economic and Policy Frameworks for Cybersecurity Risks. 94

GAO, General Accounting Office. (1999). Information Security Risk Assessment. Practices of Leading Organizations. A Supplement to GAO’s May 1998 Executive Guide on Information Security Management. Accounting and Information Management Division, GAO, 1999.

Garrabrants, W. M. , Ellis, A. W. I. , Hoffman, L. J. , & Kamel, M. . (1990). CERTS: a comparative evaluation method for risk management methodologies and tools. Computer Security Applications Conference, 1990., Proceedings of the Sixth Annual, 251 -257.

Glaser, B. G. (1992). Emergence vs. Forcing: Basics of Grounded Theory Analysis. Mill Valley, Ca.: Sociology Press. Glaser, B. G. & Strauss, A. L. (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Publishing Company.

Gregor, Shirley. (2006). The Nature of Theory in Information Systems. MIS Quarterly, 30(3):611–642, September 2006.

Greitzer, F. L. , & Hohimer, R. E. . (2011). Modeling Human Behavior to Anticipate Insider Attacks. Journal of Strategic Security, IV, 25-48.

Guarro, S. . (1987). Principles and procedures of the LRAM approach to information systems risk analysis and management. Computers Security, 6, 493-504.

Gueye, A. . (2011). A Game Theoretical Approach to Communication Security.

Hale, S. & Strube, M. (2011). An Introduction to Meta-Analysis. Department of Psychology, Washington University in St. Louis. Last accessed: 08/05/2011. URL =<artsci.wustl.edu/~psych/pdf/method.strube.pdf>.

Hall, M. Wayne & Citrenbaum, Gary. (2010). Intelligence analysis. How to think in Complex Environments. Praeger Security International, 2010.

Hallberg, N. , Hallberg, J. , & Hunstad, A. . (2007). Rationale for and Capabilities of IT Security Assessment. Information Assurance and Security Workshop, 2007. IAW ’07. IEEE SMC, 159 -166. 95

Hancock, B. . (1998). Trent focus for research development in primary health care: An introduction to qualitative research. Trent Focus Group, 1-26 ST - Trent focus for research and developmen.

Hannan, M. J. . (2005). Operational Net Assessment: A Framework for Social Network Analysis. Iosphere - Joint Information Operations Center, Fall 2005, 27-32.

Harris, S. . (2008). China’s cyber-militia: Chinese hackers pose a clear an present danger to U.S. government and private-sector computer networks and may be responsible for two major U.S. power blackouts. The National Journal.

Hausken, K. , & Zhuang, J. . (2011). Governmentsʼ and Terrorists’ Defense and Attack in a TPeriod Game. Decision Analysis, 8, 46-70.

Hernandez, Rhett Maj. Gen. (2010) Statement of Major General Rhett Hernandez, USA, Incoming Commanding General, U.S. Army Forces Cyber Command Before the House Armed Services Committee, Subcommittee on Terrorism, Unconventional Threats and Capabilities. 2nd Session, 111th Congress. September 23, 2010. Accessed on June 30, 2011. URL = <http://democrats.armedservices.house.gov/index.cfm/files/serve?File_id=067ffc96e5c1-4cef-baa2-010d16e3be57 >.

Hevner, A. R. , March, S. T. , Park, J. , & Ram, S. . (2004). Design Science in Information Systems Research. MIS Quarterly, Vol. 28 , pp. 75-105.

Hjortdal, M. . (2011). China’s Use of Cyber Warfare: Espionage Meets Strategic Deterrence. Journal of Strategic Security, IV, 1-24.

Hoffman, L. J. . (1989). Risk Analysis and Computer Security: Towards a Theory at Last. Computers Security, 8, 23-24.

Hopkins, W. G. (2011). An Introduction to Meta-analysis. Faculty of Health Science, Auckland University of Technology, NZ. Last accessed: 08/05/2011. URL =< www.sportsci.org/jour/04/Introduction_to_meta-analysis.ppt>.

INSA. (2011). Cyber Intelligence: Setting the Landscape for an Emerging Discipline. 96

IRAM, Information Risk Analysis Methodology. (2011). Last Accessed: October 2011. URL = <https://www.securityforum.org/whatwedo/publictools/#anchor3iram>

IT Grundschutz. (2008). Last Accessed: October 2011. URL = <https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/BSIStandards/standa rd_100-3_e_pdf.pdf?__blob=publicationFile>

IWG, Interagency Working Group. (2010). Guide for Applying the Risk Management Framework to Federal Information Systems. Nist Special Publication, 800-37 Revision 1, 80037 rev 1.

Jaisingh, J. , & Rees, J. . (2001). Value at Risk : A methodology for Information Security Risk Assessment. Information Security, 1-15.

Jajodia, S. , Liu, P. , Swarup, V. , & Wang, C. . (2010). Cyber Situational Awareness: Issues and Research (1st ed., Vol. 46). Springer Publishing Company, Incorporated.

Karabacak, B. , & Sogukpinar, I. . (2005). ISRAM: information security risk analysis method. Computers Security, 24, 147-159. Elsevier.

Karber, P. A. . (2008). Net Assessment & Strategy Development for the Secretary of Defense: Future Implications from Early Formulations.

Kearney, P. , & Brügger, L. . (2007). A risk-driven security analysis method and modelling language. BT Technology Journal, 25(1), 141-153. Springer Netherlands.

Khan, R.E. (2007). Developing the theoretical and conceptual framework. J199 lecture. Unpublished. Last accessed October 2011; URL = <http://journclasses.pbworks.com/f/theoretical+framework.ppt>.

Kim, I. J. , Chung, Y. J. , Lee, Y. G. , & Won, D. . (2005). A Time-Variant Risk Analysis and Damage Estimation for Large-Scale Network Systems. Computational Science and Its Applications – ICCSA 2005, 3481, 48-49. Springer Berlin / Heidelberg.


Kissel, Richard. (2011). Glossary of Key Information Security Terms. Nist Interagency report, 7298 Revision 1.

Kizza, J. . (2005). Security Assessment, Analysis, and Assurance. Computer Network Security, 177-207. Springer US.

Klinke, A. , & Renn, O. . (2002). A new approach to risk evaluation and management: riskbased, precaution-based, and discourse-based strategies. Risk analysis an official publication of the Society for Risk Analysis, 22, 1071-1094.

Klinke, A. , & Renn, O. . (2006). Systemic Risks as Challenge for Policy Making in Risk Governance. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 7, 13.

Knapp, K. J. . (2005). A Model of Managerial Effectiveness in Information Security: From Grounded Theory to Empirical Test. Alabama.

Kondakci, S. . (2010). A causal model for information security risk assessment. Information Assurance and Security (IAS), 2010 Sixth International Conference on, 143 -148.

Konecny, A. D. . (1988). Net Assessment: An Examination of the Process. Monterey, California.

Krautsevich, L. , Martinelli, F. , & Yautsiukhin, A. . (2010). Formal approach to security metrics.: what does “more secure” mean for you? Proceedings of the Fourth European Conference on Software Architecture: Companion Volume, 162-169. New York, NY, USA: ACM.

Krepinevich, A. F. . (2008). Net Assessment and Planning for National Security PUBP 710-002.

Kugler, Richard L. (2010). ‘Deterrence of Cyber Attacks’, in Franklin D. Kramer, Stuart H. Starr, and Larry K. Wentz (ed.), Cyberpower and National Security. Center for technology and National Security Policy. National Defense University. NDU Press. Potomac Books, pp. 309-340.

Kumar, R. L. , Park, S. , & Subramaniam, C. . (2008). Understanding the Value of Countermeasure Portfolios in Information Systems Security. Journal of Management Information Systems, 25, 241-279. M.E. Sharpe Inc. 98

Lee, S.-won, G, R. A. , & Ahn, G.-J. . (2005). Security Requirements Driven Risk Assessment for Critical Infrastructure Information Systems. In Proceedings of the Symposium on Requirements Engineering for Information Security (SREIS 05), RE ’05.

Leitner, A. & Schaumuller-Bichl, I. (2009). ARIMA - A new approach to implement ISO/IEC 27005. Logistics and Industrial Informatics. LINDI’09. 2nd International, 10-12 September, 2009.

LeMay, E. , Unkenholz, W. , Parks, D. , Muehrcke, C. , Keefe, K. , & Sanders, W. H. . (2010). Adversary-driven state-based system security evaluation. Proceedings of the 6th International Workshop on Security Measurements and Metrics, 5:1-5:9. New York, NY, USA: ACM.

Lichtenstein, S. . (1996). Factors in the selection of a risk assessment method. Computers Security, 4, 20-25.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA, Sage.

Lowenthal, M. Mark. (2009). Intelligence. From Secrets to Policy 4th Edition. CQ Press. 2009

Lund, M. , Solhaug, B. , & Stølen, K. . (2011). Risk Analysis of Changing and Evolving Systems Using CORAS. Foundations of Security Analysis and Design VI, 6858, 231-274. Springer Berlin / Heidelberg.

Magerit. (2006). Last accessed: October 2011. URL = <http://www.mpt.gob.es/dms/es/publicaciones/centro_de_publicaciones_de_la_sgt/Monografi as0/parrafo/01111111111111111118/text_es_files/Magerit-v2-book-I.pdf>

Marshall, W. Andrew. (1976). Net Assessment in the Department of Defense. Memo from the Office of Net Assessment, Washington D.C., September 1976.

Marshall, W. Andrew. (1991), “Strategy as a Profession for Future Generations,” in Marshall, J. J. Martin and Henry S. Rowen, eds., On Not Confusing Ourselves: Essays on National Security Strategy in Honor of Albert and Roberta Wohlstetter, Boulder, CO: Westview Press, 1991, pp. 302-311.


Mayer, N. , Dubois, E. , & Matulevičius, R. . (2008). Towards a Measurement Framework for Security Risk Management. Proceedings of Modeling Security Workshop.

Mayer, N. , Heymans, P. , & Matulevicius, R. . (2007). Design of a modelling language for information system security risk management. Information Science RCIS 2007, 1-11.

McMichael, William H. (2010) “DoD Cyber Command is officially online”, AirForceTimes, May 21, 2010. Last accessed October 2011; URL = <http://www.airforcetimes.com/news/2010/05/military_cyber_command_052110>. Mears, Z. . (2010). International Affairs 264 Defense Policy/Program Analysis.

Mehari. (2010). Last Accessed: October 2011. URL = <http://www.clusif.asso.fr/fr/production/ouvrages/type.asp?id=METHODES#doc184>

Meyers, C. , Powers, S. , & Faissol, D. . (2009). Taxonomies of Cyber Adversaries and Attacks: A Survey of Incidents and Approaches.

Miles, M.B. & Huberman, A.M. (1994). Qualitative data analysis: an expanded sourcebook, 2d ed. Thousand Oaks, CA: Sage.

Miller, N. L. , & Shattuck, L. G. . (2006). A Dynamic Process Model for the Design and Assessment of Network Centric Systems A Dynamic Process Model for the Design and Assessment of Network Centric Systems. Symposium A Quarterly Journal In Modern Foreign Literatures.

Misra, S. C. , Kumar, V. , & Kumar, U. . (2007). A strategic modeling technique for information security risk assessment. Information Management Computer Security, 15, 64-77.

Myagmar, S. , Lee, A. , & Yurcik, W. . (2005). Threat Modeling as a Basis for Security Requirements. StorageSS ’05: Proceedings of the 2005 ACM workshop on Storage security and survivability, 94-102. New York, NY, USA: ACM Press.

Nath, H. V. . (2011). Vulnerability Assessment Methods – A Review. Advances in Network Security and Applications, 196, 1-10. Springer Berlin Heidelberg.


Orlikowski, W. J. . (1993). CASE Tools as Organizational Change: Investigating Incremental and Radical Changes in Systems Development. MIS Quarterly, 17, 309-340. JSTOR.

Otero, A. R. , Otero, C. E. , & Qureshi, A. . (2010). A Multi-Criteria Evaluation of Information Security Controls Using Boolean Features. International Journal of Network Security & Its Applications, 2(4), 1-11.

PAISI (2011). PAISI’11: Proceedings of the 6th Pacific Asia conference on Intelligence and security informatics. Berlin, Heidelberg: Springer-Verlag.

Parker, T. , Sachs, M. , Miller, T. , & Devost, M. . (2011). Cyber Adversary Characterization Know thy enemy! Black Hat USA 2011.

Paterniti, D. A. (2007). Qualitative Research Methods. Center for Health Services Research in Primary Care. Departments of Internal Medicine & Sociology. Last accessed: August 2011. URL =<http://phs.ucdavis.edu/downloads/EPI298_Paterniti_071007.pdf>.

Patterson, J. P., & Smith, M. N. (2005). Developing a Reliable Methodology for Assessing the Computer Network Operations Threat of Iran. Monterey , California.

Pease, C. Charles. (1983). Strategic Assessment: the state of the art. Unpublished notes. Office of the Net Assessment, Washington D.C., 1983.

Randolph, J. . (2009). A Guide to Writing the Dissertation Literature Review. Practical Assessment, Research & Evaluation, 14, 13.

Ratcliffe, H. Jerry. (2009). Strategic Thinking in Criminal intelligence. 2nd Edition. (ed.) The Federation Press. 2009.

Richards, L. (2006). Qualitative Research Design. Thinking research.

Ropohl, G. (1999). Philosophy of socio-technical systems. Society for Philosophy and Technology, 4(3): 59–71, 1999.


Rosenquist, M. . (2009). Prioritizing Information Security Risks with Threat Agent Risk Assessment.

Ross, R. S. (2011). Managing Information Security Risk. NIST SP 800-39.

Rot, A. . (2008). IT Risk Assessment: Quantitative and Qualitative Approach. Proceedings of the World Congress on Engineering and Computer Science WCECS, 22-24.

Sajko, M. , Hadjina, N. , & Pesut, D. . (2010). Multi-criteria model for evaluation of information security risk assessment methods and tools. MIPRO, 2010 Proceedings of the 33rd International Convention, 1215 -1220.

Satoh, N. , & Kumamoto, H. . (2009). Analysis of Information Security Problem by Probabilistic Risk Assessment. International Journal of COmputers, 3.

Savenye, W. C. , & Robinson, R. S. . (2004). Qualitative research and methods: An introduction for educational technologists. Handbook of Research on Educational Communications and Technology, 1045-1071. Lawrence Erlbaum Associates.

Savola, R. . (2007). Towards a Security Metrics Taxonomy for the Information and Communication Technology Industry. International Conference on Software Engineering Advances ICSEA 2007, 60-60. Ieee.

Sofaer, S. (1999). Qualitative Methods: What are they and why use them? Health Services Research 34:5 Part II, 1101 – 1118.

Siponen, M. T. . (2005). An analysis of the traditional IS security approaches: implications for research and practice. European Journal of Information Systems, 14, 303-315.

Skypek, T. M. . (2010). Evaluating Military Balances Through the Lens of Net Assessment: History and Application. Journal of Military and Strategic studies, 12, 1-25.

Sommestad, T. , Ekstedt, M. , & Johnson, P. . (2009). Cyber Security Risks Assessment with Bayesian Defense Graphs and Architectural Models. Hawaii International Conference on System Sciences, 0, 1-10. Los Alamitos, CA, USA: IEEE Computer Society.


Soudain, N. , Raggad, B. G. , & Zouari, B. . (2009). A formal design of secure information systems by using a Formal Secure Data Flow Diagram (FSDFD). Risks and Security of Internet and Systems (CRiSIS), 2009 Fourth International Conference on, 131 -134.

Stirling, A. , Renn, O. , Klinke, A. , Rip, A. , & Salo, A. . (2001). On Science and Precaution in the Management of Technological Risk.

Stoneburner, G. , Goguen, A. , & Feringa, A. . (2002). Risk Management Guide for Information Technology Systems. Nist Special Publication, 800-30, 800–30.

Strauss, A. & Corbin, J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Sage. 1990.

Strauss, A. & Corbin, J. (1998). Basics of Qualitative Research. Techniques and Procedures for Developing Grounded Theory (2nd ed.). Sage.

Sumner, M. . (2009). Information Security Threats: A Comparative Analysis of Impact, Probability, and Preparedness. Information Systems Management, 26, 2-12.

Sun, L. , Srivastava, R. P. , & Mock, T. J. . (2006). An Information Systems Security Risk Assessment Model Under the Dempster-Shafer Theory of Belief Functions. J. Manage. Inf. Syst., 22(4), 109-142. Armonk, NY, USA: M. E. Sharpe, Inc.

Syalim, A. , Hori, Y. , & Sakurai, K. . (2009). Comparison of Risk Analysis Methods: Mehari, Magerit, NIST800-30 and Microsoft’s Security Management Guide. Availability, Reliability and Security, 2009. ARES ’09. International Conference on, 726 -731.

Tashakkori, A. & Teddlie, C. (2003). Handbook of mixed methods in social & behavioral research. Sage.

Takahashi, T. , Kadobayashi, Y. , & Fujiwara, H. . (2010). Ontological approach toward cybersecurity in cloud computing. Proceedings of the 3rd international conference on Security of information and networks, 100-109. New York, NY, USA: ACM.


Taquechel, E. F. (2010). Validation of Rational Deterrence Theory: Analysis of U.S. Government and Adversary Risk Propensity and Relative Emphasis on Gain or Loss. Monterey, California.

Taylor, G. James. (1984). Operations Research for Naval Intelligence. Unpublished notes, Naval Postgraduate School, Monterey, CA, 1984.

Teddlie, C. & Yu F. (2007). Mixed Methods Sampling. A Typology With Examples. Journal of Mixed Methods Research, Volume 1 Number 1 January 2007. Sage Publications.

Tregear, J. (2001). "Risk Assessment." Information Security Technical Report 6(3): 19-27.

Trist, E. (1981). The evolution of socio-technical systems. Occasional Paper, 2, 1981.

Vaughn, R. B. J. , Henning, R. , & Siraj, A. . (2001). Information Assurance Measures and Metrics - State of Practice and Proposed Taxonomy. Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS’03).

Verendel, V. . (2009). Quantified security is a weak hypothesis: a critical survey of results and assumptions. Proceedings of the 2009 workshop on New security paradigms workshop, 37-50. New York, NY, USA: ACM.

Vidalis, S. . (2004). A Critical Discussion of Risk and Threat Analysis Methods and Methodologies.

Volk, N. . (2010). Threat based risk management in the federal sector. 2010 Information Security Curriculum Development Conference, 97-106. New York, NY, USA: ACM.

Vorster, A. , & Labuschagne, L. . (2005). A framework for comparing different information security risk analysis methodologies. Proceedings of the 2005 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries, 95-103. , Republic of South Africa: South African Institute for Computer Scientists and Information Technologists.

White, J. . (2008). Problems of Comparative Qualitative Research. Conference on ‘European Citizenship Revisited. 104

Williams, J. G. , & Abrams, M. D. . (n.d.). Formal Methods and Models. Information Security.

Willis, H. H. , LaTourrette, T. , Kelly, T. K. , Hickey, S. , & Neill, S. . (2007). Terrorism Risk Modeling for Intelligence Analysis and Infrastructure Protection.

Woody, C. . (2006). Applying OCTAVE: Practitioners Report.

Wright, M. (1999). "Third Generation Risk Management Practices." Computer Fraud & Security 1999(2): 9-12.

Yazar, Zeki. (2002). A Qualitative Risk Analysis and Management Tool –CRAMM. Unpublished, Sans Institute.

Zarate, R. , & Sokolski, H. . (2009). Nuclear Heuristics: Selected Writings of Albert and Roberta Wohlstetter.




Annual loss expectancy American Telephone and Telegraph Bayesian belief network Central Computing and Telecommunications Agency Chief Information Security Officer Computer network exploitation Computer network operations Commercial off-the-shelf CCTA Risk Analysis and Management Method Department of Homeland Security Department of Defense Federal Emergency Management Agency Formal methods George Washington University Homeland Security Digital Library Information security Information security management system International Organization for Standardization Information system security Information and technology Net Assessment Northrop Grumman National Institute of Standards and Technology Naval Postgraduate School Office of Management and Budget People’s Liberation Army 106


People’s Republic of China Risk assessment Risk assessment methodology Risk Analysis and Management for Critical Asset Protection Risk management Risk Management Framework Sandia National Labs Socio-technical system Strengths Weaknesses Opportunities Threats Washington Research Library Consortium


Sponsor Documents


No recommend documents

Or use your account on DocShare.tips


Forgot your password?

Or register your new account on DocShare.tips


Lost your password? Please enter your email address. You will receive a link to create a new password.

Back to log-in