Case Study Research in Software Engineering: Guidelines and Examples

Foreword

This book is very timely given the increasing interest in case study research within the software engineering community and the realization by many that research that uses a case study approach provides us with a good understanding of what actually happens in the real world. What use is our research if we do not actually understand what is really happening and cannot provide useful insights into organizations targeting their practical needs?

Doing case study research in software engineering and ensuring that the research is thorough is not easy. Although there is a long history of case study research in the social sciences, it has been difficult to translate their research guidelines into the software engineering domain. This book will help both experienced and novice case study researchers improve their research methodology. The authors provide comprehensive examples of case study research they, and others, have conducted. They also critique the examples. This is very useful for researchers wanting to undertake case study research and will help them to avoid some of the problems already experienced by the authors and other researchers.

In case study research, we choose to study some phenomenon within its real-life setting. Our unit of analysis may be some aspect of a software engineering project, a software engineering methodology and its use within an organization, a software engineering section of an organization, or the whole or a particular part of a new or ongoing development or maintenance project. The unit of analysis will vary depending on the research question; the authors provide us with a number of case studies where the chosen unit of analysis varies according to the research question.

Yes, much of the data can be subjective, but I do not agree with the view that case studies in software engineering are somewhat suspect, lacking in rigor, and somehow not as good as other research methods. Properly done case studies can provide much useful information, both for the researchers and the organization involved. The authors, who are all well known for their case study research in software engineering, make a very telling comment regarding one of their research papers that used a case study methodology. They tell us that the paper was rejected by journal reviewers due to the subjective nature of their data. Such a comment from a reviewer or an editor illustrates the timeliness of this book and a very real need within the software xiii engineering community.

Case studies provide us with research results from real-world projects; these results would be difficult to achieve with any other research method. While surveys and experiments can supply useful information, I do not believe that there is any substitute for a case study when we want to find out what is happening in real projects or when methodologies and so on are implemented within a specific environment. Not only is this type of research interesting for researchers, but it is also imperative that organizations understand what is happening so that they can make informed decisions regarding what works well and what does not work, within their own particular environment.

Case studies can be very time consuming for both the researchers involved and the organizations concerned, and we cannot generalize from a single case as we do not have enough data for statistical analysis. To generalize, we need replications that use exactly the same protocol as was used for the original case. Hence, it is important to carefully develop and use a case study protocol, to accurately describe the context of the case, and to make the protocol available to other researchers. Context is very important when we are trying to answer a particular software engineering research question as we cannot begin to understand what is happening in a project or is an organization without carefully considering the context of the case we are investigating. The research question(s), proposition(s), and any hypotheses must be explicitly stated. Replications are important so that we can understand how much context influences our results. If we replicate some case study research and get the same results as the research we replicate, that is an important result; these results deserve to be published so that generalizations regarding the particular research question(s) can be made. Owing to my innate cynicism, I can see an exact replication, which yields the same results as the prior research, being rejected by journal and conference reviewers. They will say that the research does not provide anything new, even though the result is important and does add to our body of knowledge, thus making generalizations from case study research even more difficult.

In the first part of this book, readers will find useful advice covering all aspects of case study research in chapters that include discussion on case study design, data collection, data analysis and interpretation, the reporting of case studies, scaling up case study research, and using case studies; the second part of the book comprises useful, informative, and comprehensive examples of actual case study research. All in all, this book provides the means to help us all do better case study research.

Dr. June M. Verner

Conjoint Professor of Software Engineering, CSE, University of New South Wales, Sydney, Australia Marie Curie Fellow, Keele University, Staffordshire, UK

Preface

The authors' first contact with case study research and qualitative analysis was around the turn of the millennium. For Rainer, the journey started when entering his PhD program in 1995, and guidance was given by an earlier edition of Yin's book on case studies [216] from social sciences and Benbasat et al's paper [19] from information systems. For Runeson, Höst, and Regnell, the journey began by studying the first edition of Robson's book [161] and by inviting a sociologist, the late Dr. Peter Arvidson, to give a seminar on sociologic research methodology, which was a first step of our journey toward using these fuzzy tools for research.

Our experience of adapting and applying case study methodology from other disciplines to software engineering has motivated us to write this book. We intend to provide comprehensive guidance for researchers and students conducting case studies, for reviewers of case study manuscripts, and for readers of case study papers; and we do so to help these groups of people in their efforts to better understand and improve software engineering practice. The nature of case study research means that it is hard to provide precise guidelines, so we complement our guidelines with a range of examples; examples of not only good practice but also of mistakes that we have made and from which we hope others can learn. Hence, we provide examples that the reader may learn from and adapt to their situations.

The book is constituted of two main parts: methodology and examples. Part I begins with Chapter 1 dealing with motivation and a historical background to case studies in software engineering. Chapter 2 defines terms in the field of empirical research, which we use throughout the book, and sets case study research into the context of other research methods. Chapter 3 elaborates the design of a case study and planning for data collection. Chapter 4 describes the process of data collection and validation. In Chapter 5, issues on data analysis are treated, as well as the validity issues for the analysis and the whole study. Reporting case studies to different audiences is discussed in Chapter 6. Chapter 7 describes issues on scaling up to large case studies and Chapter 8 discusses different uses of case studies. In Part II of the book, Chapter 9 gives an introduction to the example case studies in Chapters 10–14. These five examples of case studies are intended as xv illustrations of the presented guidelines in a more concrete way and are taken from research areas on eXtreme programming (XP), project management (PM), quality assurance (QA), requirements management tools (RMT), and requirements engineering and verification and validation (REVV). Finally, the appendices contain checklists for reading and reviewing case study papers, together with examples of documents for the case study process.

We hope that those who design, conduct, and report case studies and those who read the results of case studies may build upon our guidelines and examples, for better understanding of and improving the software engineering practice.

P. Runeson, M. Höst, A. Rainer and B. Regnell

Lund, Sweden and Hatfield, UK

December, 2011

Acknowledgments

We are very grateful to Professor June Verner for the support she has given us in her Foreword. We are also very grateful to Professor Verner and Professor Barbara Kitchenham for reviewing an earlier version of the book. Both Professors have contributed enormously to the development of the field of software engineering research and we greatly appreciate their constructive feedback.

This book began as an article in the journal of Empirical Software Engineering and the first two authors (Runeson and Höst) thank the editor of the journal, Professor Lionel Briand, for his encouragement to prepare and submit the article. The first two authors also thank the International Software Engineering Research Network (ISERN: http://isern.iese.de/Portal/) for contributing to the development and evaluation of the case study checklists that appear in the original article, and that are reproduced here in Appendix A.

We also thank students at Lund University and Blekinge Institute of Technology for reviewing earlier drafts of this book as a part of a course on case study research: Nauman bin Ali, Elizabeth Bjarnason, Markus Borg, Alexander Cedergren, Ronald Jabangwe, Samireh Jalali, Nils Johansson, Christin Lindholm, Jesper Pedersen Notander, and Michael Unterkalmsteiner.

Several people and organizations were involved in making the example case studies possible. We acknowledge their specific contributions below.

The XP study in Chapter 10 was conducted with main contributions from Dr. Daniel Karlström, and this would not have been possible without the people that were available for interviews at ABB, Ericsson, and Vodafone.

For the material in Chapter 11, Austen Rainer thanks IBM Hursley Park, Paul Gibson, John Allan, and all the project members of Project B and Project C for their support during the two case studies; and also the other stakeholders at IBM Hursley Park who agreed for their projects to be studied. Thanks are also due to Professor Martin Shepperd for supervising Austen's PhD.

The iterative quality assurance study in Chapter 12 was conducted with main contributions from Dr. Carina Andersson, and this would not have been possible without the people that supported the data collection and were xvii available for discussions and validation of the analyses.

Chapter 13 represents Austen Rainer's interpretation of the work carried out by Cei Sanderson, while supervising Cei's Masters of Research degree at the University of Hertfordshire. Austen also thanks all the employees at 1Spatial for their cooperation and assistance during the Knowledge Transfer Project (KTP). KTP was funded by a grant (grant number: KTP000933) from the UK's Technology Strategy Board.

For the material in Chapter 14, Björn Regnell and Per Runeson thank Dr. Annabella Loconsole, Dr. Giedre Sabaliauskaite, Michael Unterkalmsteiner, Markus Borg, Elizabeth Bjarnason, Emelie Engström, Dr. Tony Gorschek, and Dr. Robert Feldt for collaboration in the study. The study project is led by Björn Regnell, with Tony Gorschek as vice leader and Per Runeson as manager of the research program EASE, to which the presented study belongs. The researchers of this project are very grateful to the anonymous interviewees for their dedicated participation in this study.

Per Runeson's work with case studies and this book were partially funded by the Swedish Research Council under grants 622-2004-552 and 622-2007-8028 for a senior researcher position in software engineering. Austen Rainer's work on this book was partially funded by a grant from the UK's Royal Academy of Engineering International Travel Grant scheme (grant number: ITG10-279) and from Lund University, Department of Computer Science. Martin Höst's and Björn Regnell's work was partly funded by the Industrial Excellence Center EASE—Embedded Applications Software Engineering (http://ease.cs.lth.se).

We are most thankful to our families for their support in the preparation of this book and helping us find the time to write it: Kristina, Jesper, Malin, Lovisa, and Hampus; Anna, Tilde, and Gustav; Clare, Samuel, and Maisie; Susanne, Rasmus, and Felix. They are closer to our hearts than case study research.

P. Runeson, M. Hö, A. Rainer and B. Regnell

Part I

Case Study Methodology

Chapter 1

Introduction

1.1 What is a case study?

The term case study appears every now and then in the title of software engineering research papers. These papers have in common that they study a specific case, in contrast to a sample from a specified population. However, the presented studies range from very ambitious and well-organized studies in the field of operations (in vivo) to small toy examples in a university lab (in vitro) that claim to be case studies. This variation creates confusion, which should be addressed by increased knowledge about case study methodology.

Case study is a commonly used research strategy in areas such as psychology, sociology, political science, social work, business, and community planning (e.g., [162, 196, 217]). In these areas, case studies are conducted with the objectives of not only increasing knowledge (e.g., knowledge about individuals, groups, and organizations and about social, political, and related phenomena) but also bringing about change in the phenomenon being studied (e.g. improving education or social care). Software engineering research has similar high-level objectives, that is, to better understand how and why software engineering should be undertaken and, with this knowledge, to seek to improve the software engineering process and the resultant software products.

There are different taxonomies used to classify research in software engineering. The term case study is used in parallel with terms like field study and observational study, each focusing on a particular aspect of the research methodology. For example, Lethbridge et al. use the term field studies as the most general term [118], while Easterbrook et al. call case studies one of the five classes of research methods [47]. Zelkowitz and Wallace propose a terminology that is somewhat different from what is used in other fields, and categorize project monitoring, case study, and field study as observational methods [218]. Studies involving change are sometimes denoted action research [119, 162], pp. 215–220]. This plethora 5 of terms causes confusion and problems when trying to aggregate multiple empirical studies and to reuse research methodology guidelines from other fields of research.

Yin defines case study as

an empirical enquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident. [217, p. 13]

This fits particularly well in software engineering. Experimentation in software engineering has clearly shown that there are many factors impacting on the outcome of a software engineering activity, for example, when trying to replicate studies [182]. One of Kitchenham et al.'s [105] preliminary guidelines for empirical research in software engineering states

Be sure to specify as much of the industrial context as possible. In particular, clearly define the entities, attributes, and measures that are capturing the contextual information.

On the subject of observational studies, which would include case studies, Kitchenham et al. write

There is an immense variety to be found in development procedures, organizational culture, and products. This breadth implies that empirical studies based on observing or measuring some aspect of software development in a particular company must report a great deal of contextual information if any results and their implications are to be properly understood. Researchers need to identify the particular factors that might affect the generality and utility of the conclusions. [105, p. 723]

Case studies offer an approach that does not require a strict boundary between the object of study and its environment. Case studies do not generate the same results on, for example, causal relationships, as controlled experiments do, but they provide a deeper understanding of the phenomena under study. As they are different from analytical and controlled empirical studies, case studies have been criticized for being of less value, being impossible to generalize from, being biased by researchers, and so on. This critique can be met by applying proper research methodology practices and by reconsidering that knowledge is more than statistical significance [56, 115, 128]. However, the research community has to learn more about the case study methodology in order to conduct, report, review, and judge it properly.

1.2 A brief history of case studies in software engineering

The term case study first appeared in software engineering journal papers in the late 1970s. At that time, a case study was typically a demonstration case, that is, a case that demonstrated the implementation of some software technology or programming concept.

In the mid-to late-1980s, papers started to report case studies of a broader range of software development phenomena, for example, Alexander and Potter's [3] study of formal specifications and rapid prototying. For these types of papers, the term case study refers to a self-experienced and self-reported investigation. Throughout the 1990s the scale of these self investigations increased and there were, for example, a series of papers reporting case studies of software process improvement in large and multinational organizations such as Boeing, Hughes, Motorola, NASA, and Siemens.

Case studies based on the external and independent observation of a software engineering activity first appeared in the late 1980s, for example, Boehm and Ross's [23, p. 902] extensive case study of the introduction of new information systems into a large industrial corporation in an emerging nation. These case studies, however, did not direct attention at case study methodology that is, at the design, conduct, and reporting of the case study.

The first case study papers that explicitly report the study methodology were published in 1988: Curtis et al.'s [37] field study of software design activities and Swanson and Beath's [199] multiple case study of software maintenance. Given the status of case study research in software engineering at the time, it is not surprising that Swanson and Beath were actually researchers in a school of management in the United States, and were not software engineering researchers. Swanson and Beath use their multiple case studies to illustrate a number of challenges that arise when conducting case studies research, and they also present methodological lessons. Their paper therefore appears to be the first of its kind in the software engineering research community that explicitly discusses the challenge of designing, conducting, and reporting case study research.

During the 1990s, both demonstration studies and genuine case studies (as we define them here) were published, although only in small numbers. Glass et al. analyzed software engineering publications in six major software engineering journals for the period 1995–1999 and found that only 2.2% of these publications reported case studies [61]. Much more recently, a sample of papers from Sj berg et al.'s large systematic review of experimental studies in software engineering [195] were analyzed by Holt [72]. She classified 12% of the sample as case studies. This compares to 1.9% of papers classified as formal experiments in the Sj berg study. But differences in the design of these reviews make it hard to properly compare the reviews and draw firm conclusions.

The first recommendations, by software engineering researchers, regarding case study methodology were published in the mid-1990s [109]. However, these recommendations focus primarily on the use of quantitative data. In the late 1990s, Seaman published guidelines on qualitative research [176]. Then, in the early twenty-first century, a broader set of guidelines on empirical research were published by Kitchenham et al. [105]. Sim et al. arranged a workshop on the topic, which was summarized in Empirical Software Engineering [189], Wohlin et al. provided a brief introduction to case studies among other empirical methods [214], and Dittrich et al. edited a special issue of Information and Software Technology on qualitative software engineering research [43]. A wide range of aspects of empirical research issues for software engineering are addressed in a book edited by Shull et al. [186]. But the first comprehensive guides to case study research in software engineering were not published until 2009, by Runeson and Höst [170] and Verner et al. [208]. Runeson and Höst's paper was published in the peer-reviewed journal Empirical Software Engineering and provides the foundation for this book.

1.3 Why a book on case studies of software engineering?

Case study methodology handbooks are superfluously available in, for example, social sciences [162, 196, 217], which have also been used in software engineering. In the field of information systems (IS) research, the case study methodology is also much more mature than in software engineering. However, IS case studies mostly focus on the information system in its usage context and less on the development and evolution of information systems. Example sources on case study methodology in IS include Benbasat et al. who provide a brief overview of case study research in information systems [19]. Lee analyzes IS case studies from a positivistic perspective [115] and Klein and Myers do the same from an interpretive perspective [111].

It is relevant to raise the question: what is specific for software engineering that motivates specialized research methodology? In addition to the specifics of the examples, the characteristics of software engineering objects of study are different from social sciences and also to some extent from information systems. The study objects in software engineering have the following properties:

They are private corporations or units of public agencies developing software rather than public agencies or private corporations using software systems.

They are project-oriented rather than line-or function-oriented organizations.

The studied work is an advanced engineering work conducted by highly educated people, rather than a routine work [60].

There is an aim to improve the engineering practices, which implies that there is a component of design research [71] (i.e. prescriptive work).

Sj berg et al. [194] write that in the typical software engineering situation actors apply technologies in the performance of activities on an existing or planned software-related product or interim products. So, for example, requirements analysts (the actors) use requirements engineering tools (the technologies) during requirements elicitation (an activity) to produce a requirements specification (an interim software-related product). Like Pfleeger [139], we use a broad definition of technology: any method, technique, tool, procedure, or paradigm used in software development or maintenance. Sj berg et al.'s use of the term actor is not restricted to mean individual people, but can refer to levels of human behavior. For example, Curtis et al. [37] identified five layers of behavior: the individual, the team, the project, the organization, and the business mileu.

There is a very wide range of activities in software engineering, such as development, operation, and maintenance of software and related artifacts as well as the management of these activities. A frequent aim of software engineering research is to investigate how this development, operation, and maintenance is conducted, and also managed, by software engineers and other stakeholders under different conditions. With such a wide range of activities, and a wide range of software products being developed, there is a very diverse range of skills and experience needed by the actors undertaking these activities.

Software engineering is also distinctive in the combination of diverse topics that make up the discipline. Glass et al. [60] describe software engineering as an intellectually intensive, nonroutine activity, and Walz et al. [212] describe software engineering as a multiagent cognitive activity. Table 1.1 provides an indication of the topics in the computing field, and therefore the expertise needed by practitioners and researchers.

Table 1.1 Topics in Computing (from Glass et al. [59]).

Many of the interim products are produced either intentionally by the actors (e.g., the minutes of meetings) or automatically by technology (e.g., updates to a version of control system). Therefore, one of the distinctive aspects of software engineering is the raw data that are naturally, and often automatically, generated by the activities and technologies.

There are clear overlaps with other disciplines, such as psychology, management, business, and engineering, but software engineering brings these other disciplines together in a unique way, a way that needs to be studied with research methods tailored to the specifics of the discipline.

Case studies investigate phenomena in their real-world settings, for example, new technologies, communication in global software development, project risk and failure factors, and so on. Hence, the researcher needs to consider not only the practical requirements and constraints from the researcher's perspective, but also the objectives and resource commitments of the stakeholders who are likely to be participating in, or supporting, the case study. Also, practitioners may want to intervene in future projects –that is, change the way things are done in future projects –on the basis of the outcomes from the case studies, and often software engineering managers are interested in technology interventions, such as adopting a new technology. This includes both software process improvement (SPI) work [201] and design of solutions [71]. There are, therefore, distinctive practical constraints on case study research in software engineering.

In addition, the software engineering research community has a pragmatic and result-oriented view on research methodology, rather than a philosophical stand, as