Advancing Qualitative Methods

Qualitative Health Research

Think-Aloud Technique and Protocol 20(4) 565­–575

© The Author(s) 2010
Reprints and permission: http://www.
Analysis in Clinical Decision-Making sagepub.com/journalsPermissions.nav
DOI: 10.1177/1049732309354278

Research http://qhr.sagepub.com

Heljä Lundgrén-Laine1 and Sanna Salanterä1

Abstract
In this article, we aim to describe the practical aspects of research actions that should be taken into account when two
methods—the think-aloud technique for data collection and protocol analysis as an analysis technique—are combined
with research of decision making in acute clinical settings. These methods are rarely used together or written about
in health care research. In addition, careful consideration of the pros and cons of study sampling, data collection,
data management, and analysis techniques or approaches is still lacking when these methods are used in health care
research. We discuss the background of the think-aloud technique and protocol analysis and review previous studies
that have used these methods.The examples in the article are based on our study, in which we investigated the decision
making of critical care experts. We find that the combination of the think-aloud technique and protocol analysis is
applicable when investigating complex and overlapping decision-making processes and rapid, ad hoc decisions made by
critical care experts.

Keywords
critical care; decision making; mixed methods; protocol analysis; qualitative methods, general; research, design

Scholars often describe the nursing process as a very logi- and in realistic simulations. In aiming to describe deci-
cal process based on deductive decision making. However, sion making in health care, scholars have used studies
in clinical situations—especially in critical care settings— based on different methodological approaches and differ-
the decision making of professionals is much more complex. ent methods (e.g., Daly, 2001; Kuipers & Kassirer, 1984;
It involves information specific to both the medical and Offredy, 2002; Simmons, Lanuza, Fonteyn, Hicks, &
the nursing disciplines. A large amount of data, informa- Holm, 2003). Nonetheless, despite a wide body of litera-
tion, and knowledge is available, which nurses utilize for ture focusing on decision making in general, there is still
decision-making purposes every day: Nurses might base a lack of current health care research about how experts
their decision making on various cues and inductive rea- collect and represent their knowledge in clinical settings.
soning, or they might make decisions intuitively, using Another deficiency is that many publications lack infor-
tacit knowledge and previous experience (Buckingham & mation about the research methods or procedures used in
Adams, 2000; Lauri & Salanterä, 2002; Taylor, 2000). sampling, data collection, data management, and data
This kind of decision making includes many overlap- analysis (Carter & Little, 2007; Mercado-Martinez, Tejada-
ping and parallel processes. In addition, it can be difficult Tayabas, & Springett, 2008). One of the study methods
to explain and describe to a nonexpert in the field what that researchers have not frequently used or written about
really happens during the decision-making processes (van in clinical settings is the think-aloud technique, which
Someren, Barnard, & Sandberg, 1994). This presents chal- can be used in combination with protocol analysis (also
lenges for health care researchers trying to find and develop
methods that reveal these decision-making processes in clin-
ical settings and make them more accessible to nonexperts. 1
University of Turku, Turku, Finland
Scholars have studied decision making in nursing since
Corresponding Author:
the 1960s (Hammond, 1996). They have used theoretical Heljä Lundgrén-Laine, Department of Nursing Science, University of
and mathematical models to assess the thinking of profes- Turku, 20014, Turku, Finland
sionals during decision making, both in clinical contexts Email: [email protected]

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015

566 Qualitative Health Research 20(4)

called the verbal protocol technique or the think-aloud Step 3: Third, I will go and check on the patient.
method; see Jones, 1989; van Someren et al., 1994). (What is the situation?)
In this article, we aim to describe how combining two Step 4: Fourth, I will make my final decision about
research methods, the think-aloud technique and protocol staff allocation.
analysis, sheds new light on decision-making processes Step 5: Finally, I will notify the staff that Nurse A will
in clinical settings. We discuss the practical standpoints treat Patient B and Nurse C will treat Patient D.
of research actions that should be taken into account
when health care experts combine and use these methods The main objective in using the think-aloud technique
in acute clinical settings. We also describe the pros and and protocol analysis together is not to judge the outcomes
cons of combining these methods, based on the results of of a participant’s cognitive process as either successful or
a decision-making study we conducted in a critical care unsuccessful decisions, but rather to explore the process
unit. In our study, we observed the immediate, ad hoc of the performance (Ericsson & Simon, 1993; van Someren
decisions made by critical care charge nurses (n = 12) and et al., 1994).
physicians (n = 8). We relate these decisions to the man- Newell and Simon (1972) described human informa-
agement of daily activities in the unit. We consider this tion management as an information process where an
aim to be highly relevant to qualitative health care re- information stimulus (e.g., patient admission) enters the
se­arch dealing with the organizational issues of providing working part of the human system as input and connects
patients with quality health care. short-term memory processes to those of long-term memory.
Scholars have long assumed that the basic differences
between these different types of memory reside in cap-
Background acity and accessing features; that is, limited vs. large
The psychological research and information processing capacity and intermediate duration vs. permanent storage.
model developed by Newell and Simon (1972) forms the Moreover, Ericsson and Simon (1993) assumed that infor-
basis for the think-aloud technique and protocol analysis. mation in short-term memory is immediately available and
Subsequent scholars have argued that verbalization high- can be verbalized, whereas information in long-term
lights the cognitive behavior of an individual and reveals memory must be transferred to short-term memory before
information stored in the working memory of that partic- verbalization can take place.
ular individual at the present moment (Ericsson & Simon, The latest scientific investigations usually divide these
1993; Jones, 1989; van Someren et al., 1994). memory functions into three parts: working memory,
Scholars have primarily combined these two methods short-term memory, and long-term memory. Although
in studies of psychology, education, and computer sci- cognitive architecture, capacity limitations, and function-
ence when investigating the performances of research alities still compose the main differences between these
participants or usability testing (van den Haak, De Jong, three facets of memory, they overlap and are closely related
& Schellens, 2003; van Someren et al., 1994). They have to each other. Researchers assume that working memory,
frequently used the think-aloud technique together with including the capacity to process visual and verbal repre-
an analysis method other than protocol analysis (e.g., sentations, operates when information has to be retained,
Fonda et al., 2008; Qiu & Yu, 2007). In addition, scholars manipulated, and quickly updated for an immediate res­
have used protocol analysis without the think-aloud ponse (Linden, 2007). Thus, working memory operates as
technique. They have applied it, for example in studies a processing unit between short-term and long-term memory.
investigating mouse clicking and keyboard use and in
studies observing the eye movements of participants (Fu,
2001; Salvucci & Anderson, 2001). Think-Aloud Technique and Protocol
Typically, health care researchers use the term proto- Analysis in Previous Studies
col to describe certain linear decision-making processes Using a literature search, we aimed to find articles that
in medical and nursing care, such as skin care or wound used these two methods together and discussed the meth-
care protocols. However, in protocol analysis, protocol odological issues. Without any further limitations, we
refers to a verbal process, expressed aloud, which reveals used the following keywords in our search: decision making,
the “step-by-step” progression of a person’s problem-solving think aloud, protocol analysis, and methodology. We
ability (Jones, 1989). found only seven articles (in the Cumulative Index to
Nursing and Allied Health Literature and PubMed, May
Step 1: First, I will check the names of the nurses 2008, English language; see the word combinations in the
who will be starting the next shift. note to Table 1). Five of the articles use both methods to-
Step 2: Then, I will check the patients they have treated. g­ether (Fisher & Fonteyn, 1995; Fonteyn & Grobe, 1994;

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015

Lundgrén-Laine and Salanterä 567

Table 1. Articles Found in the Literature Search

Study Research Questions Method and Settings

Fisher and Fonteyn (1995) What strategies do experienced critical care Think-aloud technique and protocol
nurses use to assist in reasoning and making analysis; acute clinical setting;
decisions about patient care? retrospective
Fonteyn and Grobe (1994) How do experienced, critical care nurses reason Think-aloud technique and protocol
when planning care and making decisions about analysis; simulation
a critically ill unstable patient? Is this information
useful for expert system development?
Fonteyn, Kuipers, and Grobe (1993) A description of the think-aloud method and
protocol analysis; no research questions
Fowler (1997) What thinking processes do home health nurses Think-aloud technique; acute clinical
use to plan care for chronically ill clients? setting
Funkesson, Anbäcken, and Ek (2007) What are the process and content of nurses’ Think-aloud technique and protocol
reasoning during care planning? Pressure ulcer analysis; simulation
prevention as an example.
Göransson, Ehrenberg, Ehnfors, and How was qualitative data analysis software used Think-aloud technique and protocol
Fonteyn (2006) to manage and support protocol analysis and analysis; a qualitative data analysis
think-aloud data in a study examining emergency software evaluation
nurses’ reasoning during triage?
Grobe, Drew, and Fonteyn (1991) What data do nurses verbalize during a planning Think-aloud technique and protocol
task? What description of nurses’ clinical analysis; simulation
reasoning can be derived from protocol analysis?
Simmons, Lanuza, Fonteyn, Hicks, and What information do experienced nurses Think-aloud technique and protocol
Holm (2003) concentrate on while reasoning? What bits of analysis; acute clinical setting;
information do experienced nurses link together retrospective
to form relationships? What thinking strategies
do experienced nurses use?

Note: The keywords for the literature search were decision-making, methodology, protocol analysis, and think aloud. The word combinations were
“think aloud” AND “protocol analysis,” “think aloud” AND “protocol analysis” AND “decision-making,” “think aloud” AND “protocol analysis” AND “methodol-
ogy” , and “think aloud” AND “protocol analysis” AND “decision-making” AND “methodology” .

Funkesson, Anbäcken, & Ek, 2007; Grobe, Drew, & According to the literature, the benefit of the think-
­Fonteyn, 1991; Simmons et al., 2003). One of the articles aloud technique compared to other observation methods
discusses how researchers used qualitative data analysis is that it links the thinking processes of the participant
software in a think-aloud and protocol analysis study with concurrent perceptions, thus revealing information
(Göransson, Ehrenberg, Ehnfors, & Fonteyn, 2006). In available in the working memory. Scholars have shown
three studies, researchers conducted patient simulations that even with a rather small number of participants, this
(Fonteyn & Grobe, 1994; Funkesson et al., 2007; Grobe technique provides rich and extensive data for analysis
et al., 1991), whereas in three other studies researchers (Nielsen, 1994). However, there are also many disadvan-
worked in a clinical context (Fisher & Fonteyn, 1995; tages related to this technique. Some scholars have criticized
Fowler, 1997; Simmons et al., 2003). In all of the studies, it for the fact that thinking aloud and the limited capacity
researchers used retrospective data collection and asked of memory hinder the cognitive processes of the par-
the participants to talk aloud after their performance. ticipant, thus affecting performance if the tasks involve
Most of the articles do not explicitly discuss the chal- a high cognitive load. In addition, if researchers give
lenges faced by using the various research methods but, insufficient instructions to the participants, the researcher’s
instead, merely present individual studies that have used verbal or nonverbal cues during the observation might
the various methods. Among the references, we found result in an inappropriate level of verbalization (Branch,
only one article dealing with the research methods in a 2000; Hoppmann, in press; Stratman & Hamp-Lyons, 1994).
decision-making study (Fonteyn, Kuipers, & Grobe, The benefits of protocol analysis reside in its capacity
1993). Typically, the studies lacked a description of to reveal, in detail, what information the participants are
design and data analysis, which Nielsen, Clemmensen, concentrating on while performing their tasks. With pro-
and Yssing (2002) have already pointed out. tocol analysis it is possible to describe in an orderly way

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015

568 Qualitative Health Research 20(4)

the strategies that participants employ during their per- video camera was that we aimed to reveal the most im-
formance (Ericsson & Simon, 1993). Researchers have m­ediate decisions related to the management of daily
also noticed numerous problems when protocol analysis activities in the unit, not the interpretations of the study par-
is applied to ill-structured and semantically rich tasks in ticipants. Another reason for not using the video camera
complex environments. Researchers have criticized pro- involved eth­ical considerations related to the acute settings.
tocol analysis in its purest form for being too reductive Studies using think-aloud as a collection method have
and mechanical in complex environments (Yang, 2003). focused on rich and in-depth data, and, as in most quali-
tative research, sample sizes have been quite small.
Some scholars have suggested using a sample size as
Performing a Study With small as five for stable results. Other scholars, however,
the Think-Aloud Technique have criticized such small sample sizes (e.g., Lewis,
In practice, the idea of the think-aloud technique consists 1994). In addition, some scholars have suggested that a
of asking people to verbalize their thinking while per- heterogeneous sample of participants affects sample
forming tasks or solving problems. Researchers using size, stability, and equivalence. More important than the
this technique typically both observe and audiotape or sample size for a study using the think-aloud technique
videotape the participant (Ericsson & Simon, 1993; Jones, are the characteristics of the study participants—their
1989). Some scholars have criticized the technique for skills in verbaliza­tion and the applicability of those skills to
trying to combine both observation and introspection, thinking aloud (Caulton, 2001; Fonteyn et al., 1993; Hall,
which might result in researchers missing significant De Jong, & Steehouder, 2004; van den Haak et al., 2003;
bits of information (van Someren et al., 1994). When van Someren et al., 1994). Researchers expect the partici-
researchers use the think-aloud technique, they must be pants in the study to contribute toward an understanding
experienced enough in observing people that they can of the focus of the study, whereas the aim of the
notice and interpret the cognitive processes of a particu- researcher is to obtain a sample that reveals all the
lar individual in an accurate and reliable manner. On the dimensions of the phenomenon under investigation
other hand, introspection involves interpretations of cog- (Parahoo, 2006).
nitive processes by the participants themselves. With To gain the cooperation of experts in critical care, we
regard to introspection, the crucial problem is that subjec- used a purposeful sample in our study. The inclusion cri-
tive interpretations are not always related to observable teria for our study participants were as follows: voluntary,
behavior (Ericsson & Simon, 1993). In an attempt to willing, capable, and competent in thinking aloud. As
avoid these problems, scholars can use the think-aloud we were interested in clinical management decision making,
technique to collect and measure what the participant we wanted to reach experts who had at least 5 to 10 years of
verbalizes while simultaneously observing how the par- experience working in a position of responsibility in critical
ticipant processes information and performs a particular care. We also asked the unit managers to assess the par-
task. By thinking aloud, the study participant is making ticipants’ capability and competence before asking for the
interpretations, and the main aim of the analysis is an approval of the voluntary participants themselves.
objective observation of these interpretations. In our study, we decided to limit the sample size to
Scholars can use the think-aloud technique both con- approximately 20 participants and to use replacements if
currently and retrospectively. The difference between any of the participants withdrew from the study. We took
concurrent think-aloud and the retrospective technique is into consideration that the nature of the critical care can
that, in the latter, participants are videotaped while they are also dictate the sample size. Our hypothesis was that tasks
working and verbalize their thoughts only after the perfor- in critical care would be extremely complex and varied, so
mance. Scholars consider the retrospective technique more we decided to recruit more study participants than has
useful when participants verbalize their thoughts, for been recommended (see Nielsen, 1994; Yang, 2003) to
example in a foreign language, for the purposes of a multi- obtain extensive and saturated data. We evaluated satura-
cultural study (van den Haak et al., 2003). Scholars have tion of the data after each observation. The saturation of
used both methods separately and together (Nielsen, 1994). the data occurred when no new data emerged.
In our study, we used the concurrent think-aloud tec­ We carried out our observations separately during 8-hr
hnique. We recorded the talk-aloud material during the shifts. At the beginning, these observations lasted for a
performance with a portable MP3 player. This enabled us longer period, varying from 2 to 6 hr. Each of the first six
to save the material directly onto a computer. We also observations of charge nurses lasted almost 6 hr, whereas
planned and tested the use of a video camera, but, in the half of the observations of physicians lasted almost 5 hr.
early stages, it became clear that this would not elicit The courses of action and the decision-making processes
enough new information. One reason for not using the of these two expert groups seemed to be different, and

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015

Lundgrén-Laine and Salanterä 569

this affected the observation times. When charge nurses and opinions, allowed us to obtain better data. If the par-
seemed to simultaneously use many decision processes, ticipants were silent for more than 60 seconds, we
the physicians’ decisions were more linear and focused reminded them to keep on talking (“Please, keep on talk-
on one patient at a time. Dispersion of the observation time ing”). The researcher also asked some supportive
also depended on the working situation because we were questions, such as, “What are you thinking?” and “What
observing immediate, on-the-spot decisions related to are you doing?” during the observations. We kept the
the management of daily activities. For example, at times work situation as normal as possible.
the situation was calm and uneventful and at other times To this point, scholars have carried out most nursing
very hurried and intense. All together, we recorded 92 hr studies using the think-aloud technique or protocol analy-
of thinking aloud. sis with simulations or patient scenarios (see Cioffi &
Data collection is one of the crucial tasks in the design Markham, 1997; Daly, 2001; Offredy, 2002) and have
of the study, and the think-aloud technique needs to be conducted only a small number of studies in acute clini-
carefully planned beforehand because thinking aloud and cal settings (see Aitken & Mardegan, 2000 [protocol analysis
recording are not regular practices that many profession- was not used]; Greenwood and King, 1995; Greenwood,
als deal with on a regular basis. Ericsson and Simon Sullivan, Spence, & McDonald, 2000; Simmons et al.,
(1993) advise that it would be useful to carry out practice 2003). One reason for this is that researchers can easily
and warm-up tasks before the observation and thinking- control interfering variables existing in real-life situa-
aloud sessions. Such exercises can be very easy, such as tions during simulations. In our opinion, there are several
simple arithmetic problems aimed at waking up the mental advantages to simulations, including standardization,
processes. The problem is that there are no detailed guide- predetermination and preselection of a task, and the fact
lines for this in the existing literature (van Someren et al., that they demonstrate the relevance of the situation. Some
1994). scholars criticize the concurrent, real-life think-aloud ses-
Instead of arithmetical warm-up tasks, we asked our sions because they believe thinking aloud might have a
study participants to think about how they would famil- negative effect on task performance when high-complexity
iarize the researcher with their work, as they would do with tasks are being carried out (van den Haak et al., 2003).
an incoming employee. Before the observations, we gave However, content validity might only be a problem if sim-
all participants written and oral instructions. We also used ulations are unable to represent the actual situation in real
an example of what we meant by thinking aloud: “I am life (Fonteyn et al., 1993).
thinking about the admission of a new patient. I need to Scholars can analyze the think-aloud data using differ-
know how many beds we have available and how many ent methods, such as inductive content analysis (e.g., Aitken,
nurses we have on the next shift.” 2003; Curran, Campbell, & Rugg, 2006; Daly, 2001). For
Fonteyn et al. (1993) stress the importance of always this study, we chose to apply protocol analysis to think-
scheduling the sessions and arranging a quiet study setting aloud data.
to facilitate thinking aloud. In addition, scholars have defined
precise guidelines for the observer. Ericsson and Simon
(1993) recommend that researchers give reminders to the Protocol Analysis
participants to keep on talking but that they do not engage Scholars have been using protocol analysis in social sci-
in any other communication. Some researchers have criti- ence research since the beginning of the 19th century
cized these rather strict guidelines (see, e.g., Boren & (Duncker, 1926). Nonetheless, many scholars have criti-
Ramey, 2000; Yang, 2003) and have proposed that the cized the collection of verbal data and the power of
obs­erver should be allowed to communicate with the study verification as a technique and have questioned the conclu-
participants. They have justified this line of thinking by siveness of the technique. In the 1980s, scholars developed
referring to the differences among various study designs. a more systematic approach to the collection of verbal data
Moreover, other researchers have found strict guidelines to (Ericsson & Simon, 1993; Newell & Simon, 1972) and
be suitable in studies of pure cognitive processes but not began using protocol analysis as a method more widely.
suitable in, for example, usability testing in computer sci- The systematic way of collecting verbal data and the
ence (van den Haak et al., 2003). framework of the theory for how verbalization is produced
In our study, we permitted all normal communication based on memory functions and information processing
related to the working situation. Compared to Ericsson ensured the scientific footing of the analysis (Ericsson &
and Simon (1993), we allowed more freedom for interac- Simon, 1993; Smagorinsky, 1989). The studies using pro-
tion between the participants and the observer. We believe tocol analysis have evolved from testing the information
that taking the effort to make the observation as comfort- processing models of human reasoning to find a connec-
able as possible, without affecting the participants’ talking tion between thinking and verbalization to ultimately

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015

570 Qualitative Health Research 20(4)

trying to understand the decision-making processes of acquire it from the participant. This can also be called vocal-
experts (Phansalkar, Patel, Hoffman, Pharm, & Hurdle, ization (Ericsson & Simon, 1993; Nielsen et al., 2002).
2006). In our study, we applied protocol analysis with the At the second level, information has no verbal codes,
aim of finding out more about experts’ immediate, ad hoc concepts are abstract, and the person has to explain the
decisions. We were interested in decisions that needed content of his or her thoughts, or he or she will need more
rapid decision making, and we were not interested in the time for verbalization. Therefore, the thoughts must first
interpretations that the study participants gave for their be encoded into such a format that they can be verbalized.
decisions. We also analyzed the information needed to However, the focus of the information remains the same
make such decisions. (Ericsson & Simon, 1993; Nielsen et al., 2002). In our
At the first stage of the studies, scholars have reported data, an example of this level of verbalization is the con-
using protocol analysis in a highly systematic, mechani- cept of nursing intensity.
cal way in laboratory-like settings. Typically, they have At the third level, participants further process verbaliza-
performed the data analysis in a very reductive way. For tions and pay attention to additional information obtained
the purposes of research, they describe protocol analysis before verbalization. The thinking processes are no longer
roughly in terms of information technology: input, coding, essential for the immediate performance or the attention
output. Depending on the level of analysis, the input can of the participant is no longer focused on the task under
be individual words, clauses, sentences, pauses, or one observation. At this level, participants use information
complete protocol (Ericsson & Simon, 1993). Coding is mostly from their long-term memory and connect the
based on systematic, extensive, and justified classifica- thinking processes more closely to previous experience.
tion. They analyzed the categories of coding inductively Thus, participants do not compose verbalization in iso-
in random order, with the outcome in the form of the lation. Rather, other people have an influence on their
written-out protocol. However, when they then applied thinking processes (Boren & Ramey, 2000; Ericsson &
protocol analysis in more natural conditions (less struc- Simon, 1993). Traditionally, researchers consider only the
tured and more complex or knowledge-rich settings), the first- and second-level verbalizations as reliable because
strict process of analysis posed problems, especially with these levels are assumed to reveal the content of the
the coding (see Yang, 2003). working memory (Ericsson & Simon, 1993). We also
Though scholars have at times considered the different excluded the third-level verbalizations in our own study
content analysis techniques used in qualitative research because we wanted to investigate only decisions made
to be quite similar to protocol analysis, there are differ- immediately based on the working memory of the study
ences between the two (see, e.g., Corbin & Strauss, 2007; participants. Examples of the levels of verbalizations
Ericsson & Simon, 1993; Hsieh & Shannon, 2005; Miles include the following:
& Huberman, 1994). The main purposes of content anal-
ysis are to make interpretations and condense the data First level: Patient X needs dialysis today. Nurse A
and to gain insights into the phenomena under observa- is able to do the dialysis. I will choose Nurse A
tion. The main target in protocol analysis, however, is to to take care of patient X.
describe the thinking path of the participant and gain an Second level: Patient X also has Infection Y. He
insight into the decision-making process. In addition, needs a negative pressure isolation room. This
researchers carry out qualitative data analysis through will protect the other patients and staff from in-
coding, compiling themes, and constructing categories. fections. The dialysis can be carried out there.
In protocol analysis, the levels of verbalization are impor- Third level: I just remembered that the dialysis car-
tant and, after the transcription of the data analysis, usually tridges should be checked later and that Patient A’s
consist of three steps: (a) referring phrase analysis, (b) antibiotics must be collected from the pharmacy.
assertional analysis, and (c) script analysis. We address
these in more detail in the following sections.
Phases of the Analysis
With protocol analysis, it is not always necessary to tran-
Three Levels of Verbalization scribe the think-aloud material in its entirety. Sometimes,
For analysis, researchers classify verbalization in think- for example in psychological research, everything can be
aloud into three different levels (Ericsson & Simon, 1993). relevant and researchers recommend transcribing complete
At the first classification level, participants articulate protocols to discover a person’s meaning. When the objec-
direct verbalizations during the performance. No further tive of the study is to perform a task and solve problems
processing is needed at this level before verbalization, and related to the decision making of the participants, it is pos-
researchers can reproduce information as soon as they sible to ignore events that the researcher considers irrelevant

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015

Lundgrén-Laine and Salanterä 571

So. hmm. . . Meningitis patient is coming and he needs isolation. Let’s look at our
resources. Our isolation room is available. There are some sick leaves. Well, both of
Verbal data these sick leaves are on Thursday. It means 21, 19 and 12 nurses per night shift. Ok, I
- listening the should call up someone. Is this isolation really necessary anymore, I am not sure. I should
whole data ask it later.

01: Meningitis patient is coming

02: He needs isolation
03: Look at our resources
04: Isolation room is available
Transcribing
05: There are some sick leaves
- focus on the 06: Both of these sick leaves are on Thursday
study questions 07: It means 21, 19 and 12 nurses per night shift
08: I should call up someone
09: Is this isolation really necessary anymore, I am not sure
10: I should ask it later

Levels of 01: 1st level

verbalization
02: 2nd level
- 1st and 2nd
level 03-08: 2nd level
09-10: 3rd level (excluded)

Coding categories: (a) admission, (b) special treatments, (c) material resources,
(d) adverse events, (e) human resources, (f) administrative data, (g) know-how of
Coding
personnel, (h) patient information and vital signs, (i) medication, (j) laboratory tests,
- ad hoc
(k) radiology and imaging, (l) diagnosis, and (m) discharge
decisions
- information
01: Admission
02: Patient information and vital signs
03: Human resources, material resources
04: Material resources
05-08: Human resources
Descriptive
Model
Descriptive model of the ad hoc decision-making process of charge nurses and physicians
in critical care

Figure 1. Data analyzing process

and not related to the particular performance under obser- out segmentation and coding through a coding scheme, which
vation (Parahoo, 2006; van Someren et al., 1994). In our defines how the elements of the decisions can be identi-
study, we listened to all data. We did the transcribing in fied in the data. In the third phase, we examined the protocols
authentic form (*txt) and omitted those parts of the data that through the decision-making processes, how the processes
were not related to the focus of our study questions. proceeded, and in what order. Scholars also recommend
We subsequently performed the analysis through the investigating the differences among the processes (Ericsson
three phases mentioned above: (a) referring phrase analy- & Simon, 1993; Fonteyn et al., 1993; Kuiper, 2004;
sis, (b) assertional analysis, and (c) script analysis. In the ­Kuipers & Kassirer, 1984; van Someren et al., 1994).
first phase, we began the encoding by dividing the written We began the coding of our study during the transcrip-
protocol into segments, each corresponding to one sen- tion process. We wrote each sentence or clause on a separate
tence, clause, or even word. The transcription should be line. After that, we put all the typewritten data into qualita-
done by a person familiar with the language and terminol- tive data analysis software, NVivo7 (QSR International,
ogy used by the study participants. This point should be 2006). We continued the coding so that each text line was
emphasized because transcription of the data is time- read, and according to the protocol analysis, we coded
consuming, and if the context of the study is unfamiliar to first- and second-level verbalizations either as a decision
the writer, the following steps in the protocol become more or as information used in decision making. All together,
complicated. Thereon, we encoded the segments induc- we named 13 different coding categories in the analysis:
tively using the information contained in them. We carried (a) admission, (b) special treatments, (c) material

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015

572 Qualitative Health Research 20(4)

resources, (d) adverse events, (e) human resources, (f) Careful planning of data collection, advice given to the
administrative data, (g) know-how of personnel, (h) patient study participants, and short practice sessions before the
information and vital signs, (i) medication, (j) laboratory observation are important for researchers using the think-
tests, (k) radiology and imaging, (l) diagnosis, and (m) dis- aloud technique, both for the success of the collection and
charge. We conducted the assertional analysis by to make the situations as natural as possible. Typically, the
investigating the relationships between the decisions and acute clinical settings cannot be predetermined or struc-
the information requirements, that is, by considering tured beforehand, and it is impossible to verify the
what information was needed for which decisions and situations afterward. This makes guiding the participants
what decisions were made according to which informa- during the performance difficult, as it might interrupt the
tion. Finally, we described the processes of the study performance flow of the participant. If the guidelines of
participants as a whole. We give an example of the data the think-aloud technique are followed to the letter, the
analyzing process in Figure 1. researcher does not prompt the study participant and only
asks questions that define the situation and gives remind-
ers to keep on talking.
Ethical Considerations When researchers combine the think-aloud technique
We obtained approval for our study from the nursing and protocol analysis, factors such as inclusion criteria,
research governance committees of the hospital districts suitability of the study participants, and stability of the
and the chief medical officers of the hospitals. The com- sampling affect the reliability. In our study, it was not
mittees also granted ethical approval for our study, and we possible to decide on the final sample size in advance, so
conducted our study in accordance with Finnish national we decided to keep the sample size large enough while
legislation and ethical principles. The researcher provided also paying attention to the voluntariness of participants,
both a written information sheet and verbal information ability to talk aloud, work experience, and unit managers’
before the observations and obtained written informed assessments of the potential participants. Almost all par-
consent from the participants. The researcher assured the ticipants in our study verbalized their thoughts very well;
participants of the confidentiality and anonymity of the only two seemed to feel slightly uncomfortable when talk-
data and that withdrawal from the study was possible at ing aloud. This feeling of discomfort when talking aloud
any time. In our study, we recorded only the study partici- is a clear disadvantage and should be considered if the
pants’ speech, properly deleting all other conversations saturation of the data is poor. This is another reason why
and private information concerning the patient, family, or we advocate more communication and free interaction
staff members. The researcher took into account the flu- between the researcher and participant.
ency performance of the study unit and tried not to The reliability of the analysis should be evaluated
disturb the study participants as they carried out their through the different levels and phases of the analysis.
duties. The anonymity of the study participant, profes- When the researcher is aiming at an objective analysis,
sional security, and the concealment of confidential preconceptions should be recognized and shared. Together
information are esp­ecially important in these kinds of stud- with objective analysis, the coding should be defined, con-
ies, where the investigations are made in real settings. sistent, and repeatable. The reliability of the analysis can
be improved with two or more coders by assessing inter-
coder reliability (Burla et al., 2008). In our study, a
Reliability of the Think-Aloud researcher who was familiar with the substance area car-
Technique and Protocol Analysis ried out the first- and second-level codings and also defined
Evaluation of the reliability of the think-aloud technique the codes. Another researcher then evaluated the codings,
and protocol analysis is based on the researcher’s system- and after that we calculated the percentages of agreement.
atic strategy throughout the study. Equally important
factors concerning reliability are the objectivity, reflexiv-
ity, and transparency of the study process, along with the Discussion
transferability of the study findings. The whole study In this article, we have described the application of the
process should be described so that the reader is able to think-aloud technique and protocol analysis to a decision-
follow the solutions of the researcher (see Malterud, making study in acute clinical settings. Scholars rarely
2001; Miles & Huberman, 1994). Previous researchers use these methods, especially in nursing and acute
have recognized several problems related to the reliabil- health care research (Fisher & Fonteyn, 1995; Fonteyn
ity of think-aloud study when combined with protocol & Grobe, 1994; Funkesson et al., 2007; Grobe et al.,
analysis, for example, engagement that is too limited 1991; Simmons et al., 2003), and delineations of the
and unsuccessful study environments (Defeng, 2004). use and applications are lacking (Nielsen et al., 2002;

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015

Lundgrén-Laine and Salanterä 573

Yang, 2003). This makes the use of these methods contextualized point of view when applying protocol
challenging. analysis. Our experience was that the context-free coding
The decision making of professionals in critical care is proposed by Ericsson and Simon (1993) is impossible in
strongly related to the context and the nature of the work health care contexts. Because the situations in health care
with critically ill patients. The decision-making processes are not standardized, the process of decision making is
in critical care are typically complex, and rapidly chang- lost with context-free coding.
ing situations, environments, and personal and tacit We also found that knowledge of the context area and
knowledge, previous experience, and a caring culture concepts facilitated the analysis, helping us distinguish
affect decision making. Thus, the decision makers are not between the complex and overlapping processes of
just context-free information processors who base their health care professionals on the basis of their working
decisions solely on rational calculations (see Garro, 1998). memory. We utilized the qualitative data analysis soft-
In addition, we assume that a significant number of deci- ware NVivo7 (QSR International, 2006) in the analysis.
sions in critical care are too complex to model. However, In comparison to manual coding, the software facilitated
as Garro (1998) argues in her article, researchers can model data management, especially when we investigated these
certain decisions and use the model as a reasonable guide overlapping processes. Nonetheless, analysis was labori-
to understand actions taken by the decision makers. None- ous and time consuming.
theless, in certain health care–related situations, it is The findings of our study demonstrate that the think-
important that researchers make an effort to investigate aloud technique, combined with protocol analysis, functions
evidence-based decisions and the transparency of the deci- well in a decision-making study of health care professionals
sion-making process. The think-aloud technique, together in acute care settings. However, the current level of knowl-
with protocol analysis, is one promising research method edge about think-aloud and protocol analysis as research
that can be used to reveal the decisions being made and the methods into decision making in health care is still insuffi-
information that experts need in real health care situations. cient. Although many studies have considered the
In our study, we observed the immediate, ad hoc deci- phenomenon of decision making, the challenge lies in criti-
sions related to the management of daily activities of the cally assessing and evaluating the features of these methods,
unit. These decisions support the fluent actions of the unit which can then be used to reveal real performances in acute
and ensure high-quality patient care. By modeling this clinical settings.
decision making, we might be able to find those factors
that affect unit actions or even patient outcomes. With the Acknowledgments
think-aloud technique, we were able to inductively describe We express our thanks to the clinicians involved in the research.
from the data the decisions made by the acute care experts
and the information needed for those decisions. Because Declaration of Conflicting Interests
we were interested in immediate, ad hoc decisions, the The authors declared no conflicts of interest with respect to the
protocol analysis provided us with insights into factors authorship and/or publication of this article.
that health care professionals emphasized in this kind of
decision making. Funding
Throughout the different phases of the study, we expe- The authors disclosed receipt of the following financial support
rienced the recruitment of the participants, thinking aloud, for the research and/or authorship of this article: This work was
and recording as the most challenging periods for our supported by the Finnish Funding Agency for Technology and
research. This type of study requires commitment and per- Innovation, Tekes (Grants 40435/05 and 40020/07) and the
severance on the part of the participants because of the Finnish Cultural Foundation (Grant 2704/08).
time-consuming nature of data collection and intensive
observation. We found that short practice sessions before the References
observation, however clearly context related, are useful Aitken, L. M. (2003). Critical care nurses’ use of decision-making
because the complexity of the task can impede the partici- strategies. Journal of Clinical Nursing, 12, 476-483.
pants’ thinking aloud. It is also important that the participants Aitken, L. M., & Mardegan, K. J. (2000). “Thinking aloud”:
be focused on the performance, not on the conversation with Data collection in the natural setting. Western Journal of
the researcher. Good, workable technology is also crucial in Nursing Research, 22, 841-853.
data collection. In our study, a small, portable MP3 player Boren, M. T., & Ramey, J. (2000). Thinking aloud: Reconcil-
proved to be an excellent tool for data collection. ing theory and practice. IEEE Transactions on Professional
The most challenging part of our study was the analy- Communication, 43(3), 261-278.
sis, as Yang (2003) also strongly emphasized in her study. Branch, J. L. (2000). The trouble with think alouds: Generating
We maintain that the researcher should emphasize a more data using verbal protocols. In A. Kublik (Ed.), Proceedings

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015

574 Qualitative Health Research 20(4)

of the 28th Annual Conference of the Canadian Association Fowler, L. P. (1997). Clinical reasoning strategies used during
for Information Science. Dimensions of a global information care planning. Clinical Nursing Research, 6(4), 349-361.
science, Canada. Retrieved July 20, 2009, from http://www Fu, T.-V. (2001). ACT-PRO: Action protocol tracer—A tool
.cais-acsi.ca/search.asp?year=2000 for analyzing discrete action protocols. Behavior Research
Buckingham, C., & Adams, A. (2000). Classifying clinical deci- Methods, Instruments, & Computers, 33(2), 149-158.
sion making: A unifying approach. Journal of Advanced Funkesson, K. H., Anbäcken, E.-M., & Ek, A.-C. (2007).
Nursing, 32, 981-989. Nurses’ reasoning process during care planning taking pres-
Burla, L., Knierim, B., Barth, J., Liewald, K., Duetz, M., & sure ulcer prevention as an example. A think-aloud study.
Abel, T. (2008). From text to codings. Nursing Research, International Journal of Nursing Studies, 44, 1109-1119.
57(2), 113-117. Garro, L. C. (1998). On the rationality of decision making stud-
Carter, S. M., & Little, M. (2007). Justifying knowledge, jus- ies: Part 1: Decision models of treatment choice. Medical
tifying method, taking action: Epistemologies, methodolo- Anthropology Quarterly, 12(3), 319-340.
gies, and methods in qualitative research. Qualitative Health Göransson, K., Ehrenberg, A., Ehnfors, M., & Fonteyn, M.
Research, 17, 1316-1328. (2006). The use of qualitative data analysis software (QDAS)
Caulton, D. A. (2001). Relaxing the homogeneity assumption in to manage and support the analysis of think aloud (TA) data.
usability testing. Behaviour & Information Technology, 20, 1-7. Studies in Health Technology and Informatics, 122, 143-146.
Cioffi, J., & Markham, R. (1997). Clinical decision-making by Greenwood, J., & King, M. (1995). Some surprising similarities
midwives: Managing case complexity. Journal of Advanced in the clinical reasoning of “expert” and “novice” orthopae-
Nursing, 25, 265-272. dic nurses: Report of a study using verbal protocols and pro-
Corbin, J., & Strauss, A. C. (2007). Basics of qualitative research: tocol analyses. Journal of Advanced Nursing, 22, 907-913.
Techniques and procedures for developing grounded theory Greenwood, J., Sullivan, J., Spence, K., & McDonald, M.
(3rd ed.). Thousand Oaks, CA: Sage. (2000). Nursing scripts and the organizational influences on
Curran, M., Campbell, J., & Rugg, G. (2006). An investigation critical thinking: Report of a study of neonatal nurses’ clini-
into the clinical reasoning of both expert and novice podia- cal reasoning. Journal of Advanced Nursing, 31, 1106-1114.
trists. Foot, 16(1), 28-32. Grobe, S. J., Drew, J. A., & Fonteyn, M. E. (1991). A descriptive
Daly, W. M. (2001). The development of an alternative method analysis of experienced nurses’ clinical reasoning during a
in the assessment of critical thinking as an outcome of nurs- planning task. Research in Nursing & Health, 14(4), 305-314.
ing education. Journal of Advanced Nursing, 36, 120-130. Hall, M., De Jong, M., & Steehouder, M. (2004). Cultural differ-
Defeng, L. (2004). Trustworthiness of think-aloud protocols in ences and usability evaluation: Individualistic and collectivis-
the study of translation processes. International Journal of tic participants compared. Technical Communication, 51(4),
Applied Linguistics, 14(3), 301-313. 489-503.
Duncker, K. (1926). A qualitative (experimental and theoreti- Hammond, K. R. (1996). Human judgement and social policy.
cal) study of productive thinking (solving of comprehen- Irreducible uncertainty, inevitable error, unavoidable injus-
sible problems). Pedagogical Seminar, 33, 642-708. tice. New York: Oxford University Press.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis. Hoppmann, T. K. (in press). Examining the “point of frustration”:
Verbal reports as data (1st rev. ed.). Cambridge, MA: MIT The think-aloud method applied to online search tasks. Qual-
Press. ity and Quantity.
Fisher, A., & Fonteyn, M. E. (1995). An exploration of an inno- Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to
vative methodological approach for examining nurses’ heu- qualitative content analysis. Qualitative Health Research,
ristic use in clinical practice. Scholarly Inquiry for Nursing 15, 1277-1288.
Practice, 9(3), 263-276. Jones, J. A. (1989). The verbal protocol: A research technique
Fonda, S. J., Paulsen, C. A., Perkins, J., Kedziora, R. J., for nursing. Journal of Advanced Nursing, 14, 1062-1070.
Rodbard, D., & Bursell, S.-E. (2008). Usability test of Kuiper, R. A. (2004). Nursing reflections from journaling during
Internet-based informatics tool for diabetes care provid- a perioperative internship. AORN Journal, 79(1), 195-218.
ers: The comprehensive diabetes management program. Kuipers, B., & Kassirer, J. P. (1984). Causal reasoning in medi-
Diabetes Technology & Therapeutics, 10(1), 16-24. cine: Analysis of a protocol. Cognitive Science, 8, 363-385.
Fonteyn, M. E., & Grobe, S. J. (1994). Expert system develop- Lauri, S., & Salanterä, S. (2002). Developing an instrument to
ment in nursing: Implications for critical care nursing prac- measure and describe clinical decision making in different
tice. Heart & Lung: The Journal of Critical Care, 23(1), nursing fields. Journal of Professional Nursing, 18(2), 93-100.
80-87. Lewis, J. R. (1994). Sample size for usability studies: Addi-
Fonteyn, M. E., Kuipers, B., & Grobe, S. J. (1993). A descrip- tional considerations. Human Factors, 36, 369-378.
tion of think aloud method and protocol analysis. Qualita- Linden, D. E. J. (2007). The working memory networks of the
tive Health Research, 3, 430-441. human brain. Neuroscientist, 13(3), 257-267.

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015

Lundgrén-Laine and Salanterä 575

Malterud, K. (2001). Qualitative research: Standards, challenges, Salvucci, D. D., & Anderson, J. R. (2001). Automated eye-
and guidelines. Lancet, 358, 483-488. movement protocol analysis. Human-Computer Interaction,
Mercado-Martinez, F. J., Tejada-Tayabas, L. M., & Springett, J. 6, 39-86.
(2008). Methodological issues in emergent evaluations of Simmons, B., Lanuza, D., Fonteyn, M., Hicks, F., & Holm, K.
health programs: Lessons from Iberoamerica. Qualitative (2003). Clinical reasoning in experienced nurses. Western
Health Research, 18, 1277-1288. Journal of Nursing Research, 25, 720-724.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data anal- Smagorinsky, P. (1989). The reliability and validity of protocol
ysis: An expanded sourcebook (2nd ed.). Thousand Oaks, analysis. Written Communication, 6(4), 463-479.
CA: Sage. Stratman, J. F., & Hamp-Lyons, L. (1994). Reactivity in concurrent
Newell, A., & Simon, H. A. (1972). Human problem solving. think-aloud protocols: Issues for research. In P. Smagorinsky
Englewood Cliffs, NJ: Prentice Hall. (Ed.), Speaking about writing: Reflections on research method-
Nielsen, J. (1994). Estimating the number of subjects needed ology (pp. 89-111). Thousand Oaks, CA: Sage.
for a thinking aloud test. International Journal of Human- Taylor, C. (2000). Clinical problem-solving in nursing:
Computer Studies, 41, 385-397. Insights from the literature. Journal of Advanced Nursing,
Nielsen, J., Clemmensen, T., & Yssing, C. (2002). Getting 31, 842-849.
access to what goes on in people’s heads? Reflections on the van den Haak, M. J., De Jong, M. D. T., & Schellens, P. J.
think-aloud technique. Proceedings of the Nordic Conference (2003). Retrospective vs. concurrent think-aloud protocols:
on Human-Computer Interaction, Denmark, 31, 101-110. Testing the usability of an online library catalogue. Behav-
Retrieved July 20, 2009, from http://portal.acm.org/citation iour & Information Technology, 22, 339-351.
.cfm?id=572033 van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C.
Offredy, M. (2002). Decision-making in primary care: Outcomes (1994). The think aloud method. A practical guide to model-
from a study using patient scenarios. Journal of Advanced ing cognitive processes. London: Academic Press.
Nursing, 40, 532-541. Yang, S. C. (2003). Reconceptualizing think-aloud methodol-
Parahoo, K. (2006). Nursing research. Principles, process and ogy: Refining the encoding and categorizing techniques via
issues (2nd ed.). New York: Palgrave Macmillan. contextualized perspectives. Computers in Human Behavior,
Phansalkar, S., Patel, V. L., Hoffman, J. M., Pharm, D., & 19, 95-115.
Hurdle, J. F. (2006). Use of verbal protocol analysis for
identification of ADE signals [Abstract]. Proceedings of the Bios
American Medical Informatics Association, USA, 2006, 1063. Heljä Lundgrén-Laine, RN, MNS, is a doctoral student at the
Qiu, Y., & Yu, P. (2007). Nursing information systems— Department of Nursing Science, University of Turku, Finland,
Applying usability testing to assess the training needs for and serves as a nurse planner at the Turku University Hospital,
nursing students. Methods of Information in Medicine, Hospital District of Southwest Finland.
46(4), 416-419.
QSR International. (2006). NVivo7: Qualitative research com- Sanna Salanterä, PhD, RN, is a professor of clinical nursing
puter software and manual. Retrieved July 20, 2009, from science at the Department of Nursing Science, University of
http://www.qsrinternational.com/products_previous-products_ Turku, Finland, and a part-time head nurse of the Hospital Dis-
nvivo7.aspx trict of Southwest Finland.

Downloaded from qhr.sagepub.com at University of Wyoming Libraries on March 29, 2015