See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/41308389

Context in industrial software engineering research

Article · October 2009

DOI: 10.1145/1671248.1671293 · Source: OAI

CITATIONS READS

204 507

2 authors, including:

Kai Petersen
Hochschule Flensburg
101 PUBLICATIONS 10,482 CITATIONS

SEE PROFILE

All content following this page was uploaded by Kai Petersen on 19 May 2015.

The user has requested enhancement of the downloaded file.

 Copyright © 2009 IEEE.
Citation for the published paper:

Context in industrial software engineering research

Kai Petersen, Claes Wohlin

3rd International Symposium on Empirical Software Engineering and Measurement (ESEM)

2009 Orlando

This material is posted here with permission of the IEEE. Such permission of the IEEE does
not in any way imply IEEE endorsement of any of BTH's products or services Internal or
personal use of this material is permitted. However, permission to reprint/republish this
material for advertising or promotional purposes or for creating new collective works for
resale or redistribution must be obtained from the IEEE by sending a blank email message to
[email protected].

By choosing to view this document, you agree to all provisions of the copyright laws
protecting it.
 Third International Symposiumm on Empirical Software Engineering and Measurement

Context in Industrial Software Engineering Research

Kai Petersen, Claes Wohlin

Blekinge Institute of Technology
Box 520 SE-372 25 Ronneby, Sweden
{kai.petersen | claes.wohlin}@bth.se

Abstract reviews, it is possible to classify the outcome of the stud-

ies based on the context. Thereby, companies can compare
In order to draw valid conclusions when aggregating ev- their context to the contexts described in the review, allow-
idence it is important to describe the context in which indus- ing them to make a good choice when selecting a solution.
trial studies were conducted. This paper structures the con- Furthermore, it will be easier to integrate evidence based on
text for empirical industrial studies and provides a check- solutions for software engineering problems (e.g. specific
list. The aim is to aid researchers in making informed deci- processes, practices, and techniques) with good context de-
sions concerning which parts of the context to include in the scriptions. This helps to give answers to a question such
descriptions. Furthermore, descriptions of industrial stud- as: How should a large-scale company developing embed-
ies were surveyed. ded systems (further information about the context can be
added here) handle their quality assurance?
To aid researchers in describing the context in a way
1 Introduction to allow better aggregation of evidence, this paper makes
the following novel contributions: (1) Provide a checklist
Evidence-based software engineering aims at integrating to describe the context in industrial studies; (2) Conduct
evidence from different studies. The purpose is to give rec- a small literature survey of industrial studies to determine
ommendations which solutions are the most useful for the to what degree industrial case studies covered the context
software industry [3]. When integrating evidence from in- facets identified in our checklist.
dustrial studies we argue that context has a large impact
on the conclusions. Often different studies have the same
object of investigation (let’s say an agile process), but the 2 Related Work
process is studied in different domains, different sizes of
companies, different people with different roles and skills,
different cultures, and so forth. This leads to two important Runeson and Höst [4] emphasize that case studies are
implications: (1) The object of investigation is implemented embedded in a real-life context/environment. They pro-
in different ways to fit into the context. From our experience vide a checklist to guide researchers in designing good case
of doing research with industry we know that companies do studies, and in judging the quality of case studies. In their
not adopt general methods and models one to one, but tailor checklists context is implicitly considered. Thus, the check-
them to fit their specific needs; (2) The conclusion drawn list presented here can serve as a complement to the check-
in a specific study is only true in the context in which it lists for case study research. Kitchenham et al. [2] provide
was conducted, although some generalization may be pos- three elements considered context in case studies, namely
sible as well. Though, if a solution works in many different objectives of the study, baseline against which is evaluated,
contexts, this is an indication that the solution is generally and external project constraints. The external constraints
applicable. can be interpreted as the environment impacting the out-
This means that in order to judge whether a specific solu- come of the study regarding a specific case. Though, no
tion can be successful it is necessary to describe the context guidelines on how to consider context are provided. A later
as complete and accurate as possible for the considered ob- paper by the same authors states that it is important to de-
ject of study. When having a comprehensive description of scribe as much of the context as possible [1]. Overall, the
the context, evidence-based software engineering can lever- related work underlines the importance of a checklist for
age on this richer description. For example, in systematic context descriptions.

978-1-4244-4841-8/09/$25.00 ©2009 IEEE

401
 Third International Symposiumm on Empirical Software Engineering and Measurement

3 A Checklist for Context Documentation • Programming language: This element describes the
programming language in which the system was de-
The checklist is structured in six different context facets, veloped.
namely product, processes, practices and techniques, peo-
Processes: The process describes the work-flow of the
ple, organization, and market (see Figure 1). In the center
development. Context elements:
of the context facets the object of the study is shown. The
object of the study interacts with the context. For example, • Activities: The activities are different steps in the de-
when having an agile process as the object of study, the pro- velopment process (e.g. specifying requirements).
cess is used to develop a product, is executed by people, in-
teracts with other processes, and is supported by practices, • Work-flow: The work-flow describes the order in
tools and techniques. Furthermore, the object of study is which activities are executed (including branching,
embedded in an organization (as is the case for all other merging, iterations, etc.).
context facets). The organization is operating within a mar- - Market • Artifacts: Artifacts are the results of the activities (e.g.
- Organization
ket. - Processes
the requirements specification).
- Practices and Techniques
- Product
Market
- People Practices, Tools, Techniques: Practices, tools, and tech-
niques describe systematic approaches that are used in the
Organization organization, and are interacting with the object of study.
Context elements:
Product Processes
• CASE tools: This element describes tools that are
Object
of used to support or automate software development (for
Study
Practices, example, integrated development environments, auto-
Tools, People
Techniques mated test tools, etc.).
• Practices and Techniques: This can be systematic ap-
proaches interacting with the object of study. For ex-
ample, when studying an agile process practices could
be time-boxing, frequent integration, or pair program-
Figure 1. Context Facets ming.
Organization
Each context facet comprises a set of context elements People: The human factor is very important when study-
describing the facet. In the following we provide a descrip- ing software development, as it has a major impact on the
tion of the facets and examples of related elements. success of software development. Thus, the factor has to be
Product: The product is the software system developed covered in the context. Context elements:
with the help of the object of study. Context elements:
• Roles: This element describes what type of roles are
• Maturity: The maturity of the product needs to be de- involved in using the object of study. This includes a
scribed. For example, by saying how long it was on description of the main responsibilities and authorities
the market, how many releases there were, etc. associated with the roles. For example, which roles
make use of a specific technique studied in a project.
• Quality: The product development is driven by differ-
ent quality aspects (e.g. a development effort is under- • Experience: Experience is concerned with the areas
taken to increase maintainability of the product). that people affected by the object of study have worked
in, and for how long. Furthermore, education and
• Size: Size of the product measured in, for example, trainings are of interest.
lines of code, or number of function points. The size is
an important indicator for product complexity. Organization: The organization describes the company
structure in which the other context facts and the solution
• System type: The system can be of different types, such are embedded in. Examples for elements are:
as an information system, embedded system, web-
application, or distributed system. • Model of overall organization: The organization
model describes how the company is organized, such
• Customization: This means that the product is either as matrix-organization or hierarchical organization.
general, or customizable and can be tailored to differ- Furthermore, it could be discussed whether the orga-
ent market segments. nization is flexible or strict.

978-1-4244-4841-8/09/$25.00 ©2009 IEEE

402
 Third International Symposiumm on Empirical Software Engineering and Measurement

• Organizational unit: The organizational unit is a part further work, the aim should be for a more complete de-
of the organization closely interacting with the object scription of context elements, which would result in a com-
of study. For example, this can be a project (temporary prehensive framework for context description.
existing unit) or department (permanent unit). The or- Not all elements can be described in industrial stud-
ganizational unit can be complemented with manage- ies. Rather a sub-set of elements should be described that
ment related information such as responsibilities of the are most relevant for the conclusions drawn in relation to
unit, and size measured as number of persons involved. the object of study. That is, the checklist should be gone
through and informed decisions should be made of what to
• Certification: This tells whether the organization is include and what not to include. To motivate researchers
certified (e.g. ISO/CMMI). considering context, it is important to emphasize context
more in guidelines for systematic reviews and case study
• Distribution: The organizational units are either collo- research.
cated or distributed (nationally or internationally).

Market: The market represents the customers and com- 4 Context in Industrial Studies
petitors. Elements describing the market are:
The research question (RQ) for the survey is: To what de-
• Number of customers: This element refers to bespoke gree do industrial studies cover the context facets presented
versus market-driven development. In bespoke devel- in this study in their descriptions?
opment the customer is known and can be addressed, In order to provide at least a partial answer to the ques-
and a contract is established already between devel- tion, we limit our search to the journal of empirical soft-
oper and customer. On the other hand, market-driven ware engineering (EMSE). The reasons for doing so are: (1)
development targets a large and open market of poten- the journal has an explicit focus on publishing high quality
tial customers that might buy the product after release. empirical work; and (2) there are no strict page limitations
which does not give authors a reason to neglect descriptions
• Market segments: The market segments describe of the context.
groups of customers that share a common need. Within the journal we focus on empirical studies con-
ducted in industry. In order to identify these we searched
• Strategy: The strategy describes how to address the
the journal with the following search string:
market in the long-term. For example, the company
can run a niche-strategy where they compete with a • Industrial OR case study IN title/abstract
special product (e.g. in terms of extremely high qual-
ity) or a strategy to compete on a low price. Only industrial studies are included. Experience reports,
short papers, surveys, experiments and case studies of open
• Constraints: The market can put constraints on soft- source software were excluded. To identify the context
ware development, such as a very short time-to- facets described in the studies we read the method sec-
market, or certifications (e.g. CMMI). tions of the identified articles and noted down which context
facets and elements were covered.
We would like to emphasize that context has a large im- One threat to validity is the bias of the main author when
pact on the solution, and also that context elements influ- classifying the articles and extracting the context elements.
ence each other. For example, when having several market To reduce the risk the author used the keywords as an aid to
segments that share a common need, it is normal to develop classify the papers. Furthermore, a template for data extrac-
a product that is customizable to address as many market tion was developed. Another threat is that we only focused
segments as possible with little effort, which would call on the EMSE journal, which reduces the coverage of topics.
for product-line approaches as a solution. How the solu- However, we believe that similar observations will be made
tion would look like is further influenced by the complex- when considering other sources.
ity of the product. Another example is the strategy on the Results: The search returned 56 studies, of which 27
market-level. If we would have a strategy to target a market were industrial studies relevant for the analysis. Only three
with low costs, then testing solutions would look different studies cover five and more context facets. Eleven studies
in comparison to a market where a niche is addressed, call- cover three to four facets. The remaining part (13 studies)
ing for very high quality. only cover less than three facets.
Regarding the context elements it is important to men- As which context facets to cover depends on the object
tion that we do not claim completeness. The checklist is of study we grouped the studies. The identified groups re-
rather a first step to raise awareness that there are multiple garding the object of investigation are: software reliability
facets that ought to be covered in context descriptions. In (SR) covering topics such as predicting faults in fault-prone

978-1-4244-4841-8/09/$25.00 ©2009 IEEE

403
 Third International Symposiumm on Empirical Software Engineering and Measurement

Table 1. Coverage of Research Facets Considering Objects of Investigation

Facets SR(8) SP(4) QA(7) SC(2) UML(2) RM(1) RES(1) AE(1) CS(1) Total(27)

Product 8 4 6 2 2 1 1 1 1 26
Process 2 2 3 1 1 0 0 0 0 9
Pract./Tool 3 2 5 0 1 1 0 0 0 12
People 0 1 1 0 2 0 0 0 1 5
Organization 0 4 4 1 2 1 0 0 1 13
Market 0 2 1 0 1 0 0 0 0 4

models, reliability growth models, and quality predictions; nities (focusing on a specific object of investigation)
software process (SP) covering topics such as evaluating to discuss and arrive at a common ground for context
processes, and software process improvements; quality as- descriptions.
surance (QA) evaluating quality assurance approaches such
as testing techniques, reuse for QA, or inspections; size and 5 Conclusion
cost estimation (SC); object orientation and UML (UML);
requirements management (RM); software restoration (RS);
This paper proposes a checklist for the description of
architecture evaluation (AE); maintainability through code
context, consisting of context facets (product, process, prac-
smells (CS).
tice/tools, people, organization, market) and related context
Table 1 shows the coverage of the facets grouped by the
elements. A literature review of industrial studies showed
objects of investigation mentioned before. The top line of
that studies investigating a similar object do not agree on
the table shows the abbreviation of the objects of investiga-
which context facets are important to mention. Our check-
tion and the total number of papers in brackets. The table
list aims to help researchers to take informed decisions on
allows us to make the following observations:
what to include and not to include. The checklist also serves
• Almost all studies consider it important to provide in- as a basis for discussion to reach a consensus in different
formation of the product being studied. This mostly communities which context facets/elements are most impor-
included information of the type of product, and its tant for their focus area. In future work the checklist has to
complexity. be further extended. We also plan to do in-depth analysis of
a selection of articles focusing on context facets as well as
• Studies investigating SP, QA, and the UML studies elements.
have the highest coverage of facets. This indicates that
some objects of investigation seem to aim at a higher References
coverage of the context facets. This supports our pro-
posal regarding the usage of the checklist, i.e. identi-
[1] B. Kitchenham, S. L. Pfleeger, L. Pickard, P. Jones,
fying those context facets and elements that are likely
D. C. Hoaglin, K. E. Emam, and J. Rosenberg. Pre-
to influence the outcome of the study. This will differ
liminary guidelines for empirical research in software
depending on the object of investigation.
engineering. IEEE Trans. Software Eng., 28(8):721–
• Even though there is a trend visible which objects of 734, 2002.
investigation require higher coverage of context facets, [2] B. Kitchenham, L. Pickard, and S. L. Pfleeger. Case
the studies did not agree on which context facets are studies for method and tool evaluation. IEEE Software,
important to mention. For example, only a sub-set of 12(4):52–62, 1995.
studies investigating SR consider it important to de-
scribe processes and practices/tools (see Table 1). Sim- [3] B. A. Kitchenham, T. Dybå, and M. Jørgensen.
ilar patterns can be observed for the other objects of Evidence-based software engineering. In Proceedings
investigation as well (e.g. processes, practices/tools, of the 26th International Conference on Software Engi-
and market for SP). Thus, this supports the need for neering (ICSE 2004), pages 273–281, 2004.
a checklist of context facets and elements to increase
awareness of potentially relevant context facets, and [4] P. Runeson and M. Höst. Guidelines for conducting and
by that aiding the researcher in making informed deci- reporting case study research in software engineering.
sions of what to include and not to include. Further- Empirical Software Engineering, 14(2):131–164, 2009.
more, the checklist can be used in different commu-

978-1-4244-4841-8/09/$25.00 ©2009 IEEE

404
View publication stats