healthcare

Article
Emergency Nursing-Care Patient Satisfaction Scale (Enpss):
Development and Validation of a Patient Satisfaction Scale with
Emergency Room Nursing
Junpei Haruna 1, * , Naomi Minamoto 2 , Mizue Shiromaru 3 , Yukiko Taguchi 3 , Natsuko Makino 4 , Naoki Kanda 5
and Hiromi Uchida 4

1 Department of Intensive Care Medicine, School of Medicine, Sapporo Medical University,

Sapporo 060-8543, Japan
2 Department of Nursing, Sapporo City General Hospital, Sapporo 060-8604, Japan; [email protected]
3 Department of Nursing, School of Health Sciences, Sapporo Medical University, Sapporo 060-8556, Japan;
[email protected] (M.S.); [email protected] (Y.T.)
4 Department of Nursing, Sapporo Medical University Hospital, Sapporo 060-8543, Japan;
[email protected] (N.M.); [email protected] (H.U.)
5 Department of Nursing & Social Services, Health Sciences University of Hokkaido, Tobetsu 061-0293, Japan;
[email protected]
* Correspondence: [email protected]

Abstract: This study aimed to develop and validate an emergency nursing-care patient satisfaction
scale to measure patient satisfaction with emergency room (ER) nursing. Patient satisfaction scales



for ER nursing have been validated without considering the perspectives of the healthcare system or


cultural background of the country. Moreover, although nursing care is changing with COVID-19,
Citation: Haruna, J.; Minamoto, N.;
no scale has been specifically designed to assess patient satisfaction with ER nursing. The study
Shiromaru, M.; Taguchi, Y.; Makino,
N.; Kanda, N.; Uchida, H. Emergency
population included patients who visited five ERs in Japan (March to December 2021) (n = 135). The
Nursing-Care Patient Satisfaction rating scales were provided to patients who visited the ER and gave consent, and the patients were
Scale (Enpss): Development and asked to reply. In the process of validating the scale, exploratory and confirmatory factor analyses of
Validation of a Patient Satisfaction the construct and criterion validity were conducted. The confirmatory factor analysis results showed
Scale with Emergency Room Nursing. a factorial structure consisting of four factors. The domain and summary scores demonstrated
Healthcare 2022, 10, 518. https:// good-to-excellent internal reliability (Cronbach’s range = 0.81–0.89). This patient satisfaction scale
doi.org/10.3390/healthcare10030518 was designed and validated from the perspective of the Japanese healthcare system and cultural
Academic Editors: Teen-Hang Meen, backgrounds. This scale may be useful for developing assessments and interventions to improve
Chun-Yen Chang, Charles Tijus and patient satisfaction with ER nursing.
Po-Lei Lee
Keywords: emergency room; patient satisfaction scale; emergency nursing
Received: 15 February 2022
Accepted: 11 March 2022
Published: 12 March 2022

Publisher’s Note: MDPI stays neutral 1. Introduction

with regard to jurisdictional claims in
Patient response to healthcare services is one of the best ways to obtain information
published maps and institutional affil-
about the quality of healthcare [1]. In particular, patient satisfaction is an important indica-
iations.
tor for evaluating service quality and improving healthcare services, and is a commonly
used and valid indicator [2,3]. The field of medicine is divided into specialties, and spe-
cialized treatment and care are provided in each field. In measuring patient satisfaction,
Copyright: © 2022 by the authors.
the development of satisfaction scales for each specialty was reported to help deal with
Licensee MDPI, Basel, Switzerland. specific problems [4]. In nursing, patient satisfaction is defined as the extent of the gap
This article is an open access article between patient expectations of ideal nursing care and their perception of the nursing care
distributed under the terms and received [5]. We consider it important to use information obtained from patient satisfaction
conditions of the Creative Commons to improve care and enhance the quality of healthcare services.
Attribution (CC BY) license (https:// Emergency nurses have multiple challenging responsibilities, including dealing with
creativecommons.org/licenses/by/ overstressed patients and their relatives, homeless and mentally ill patients, and victims [6].
4.0/). In addition, emergency nurses play several roles, including triage [7], first aid [7], and

Healthcare 2022, 10, 518. https://doi.org/10.3390/healthcare10030518 https://www.mdpi.com/journal/healthcare

Healthcare 2022, 10, 518 2 of 20

early recognition of critically ill patients [8]. In recent years, emergency care has become
more urgent worldwide due to the increasing severity of coronavirus disease 2019 (COVID-
19) patients, increasing the burden on emergency nurses [9]. However, even during the
COVID-19 pandemic, rapid and appropriate responses to emergency patients are required.
Various models have been used to measure satisfaction with emergency care [4,10,11].
The Consumer Emergency Care Satisfaction Scale (CECSS) is one of the most widely used
patient satisfaction surveys for emergency patients in many countries [10]. Many studies
using existing satisfaction scales have been reported for triage nurses [12,13] and have been
conducted in a variety of geographic areas, ranging from urban to rural areas [14]. Factors
associated with patient satisfaction with emergency care have been consistently reported in
previous studies to be related to staff attitudes, explanations to patients, communication,
time spent waiting in the emergency room (ER) [15,16], and environmental factors in
the ER [17,18]. Satisfaction with emergency care has also been reported to be related
to organizational culture [19] and healthcare professional burnout [20,21]. Therefore,
measuring patient satisfaction is of great importance to healthcare professionals in the
ER setting, as patient satisfaction measures can be used to identify weaknesses in their
respective facilities.
However, patient satisfaction with the healthcare provided is influenced by several
characteristics, such as the cultural background of the country or region [22], race [23], the
healthcare system [24], the insurance system [25], the educational system for nurses [26],
and the economic situation of the country [27]. Even though patient satisfaction is used in
many countries, these factors limit the ability to measure patient satisfaction that reflects
country-specific characteristics. In addition, a systematic review of patient satisfaction
surveys reported that there is no gold standard for measuring patient satisfaction [28], and
that it is necessary to develop instruments that are appropriate for the healthcare system of
each country and other factors.
There are three levels of emergency hospital designation in Japan: “primary” for
patients who can be managed as outpatients, “secondary” for patients who need to be
hospitalized, and “tertiary” for patients who need to be managed in an intensive care
unit (ICU) [29]. Japan has one of the most aged populations in the world [30], and the
number of patients visiting the ER is on the rise because of the COVID-19 pandemic [31].
Moreover, Japan has had a universal health coverage system since 1961, providing equal
access to healthcare for all citizens at all times [32]. Furthermore, in terms of nursing
specialization, although there are specialists, such as certified nurses and certified nurse
specialists, they account for approximately 1% of all nurses, and the current situation
is that there is insufficient training of specialists [33]. However, the current situation in
Japan differs from that in other countries. Therefore, the patient satisfaction scale for
emergency nurses currently in use does not necessarily match the assessment in Japan.
Furthermore, the recent COVID-19 pandemic has changed patient satisfaction in the ER.
Patient satisfaction with ER staff tended to be high at the beginning of the COVID-19
pandemic. This suggests an overall appreciation for care during the early stages of the
COVID-19 pandemic [34]. Furthermore, patient satisfaction with the ER has decreased
during the COVID-19 pandemic due to increased workloads, such as infection control
measures [15]. In addition, the COVID-19 pandemic would lead to overcrowding in the
ER and lower patient satisfaction [35]. However, no scale has been developed to measure
patient satisfaction with emergency nurses in Japan to the best of our knowledge. This study
aimed to develop and validate a patient satisfaction scale to measure patient satisfaction
with ER nursing.

2. Materials and Methods

2.1. Study Design
This study had the following steps to achieve its objectives: (1) Developing items to
measure patient satisfaction with ER nursing, (2) examining content validity, (3) selecting
items, and (4) examining construct validity.
Healthcare 2022, 10, 518 3 of 20

(1) Development of items to measure patient satisfaction with ER nursing

In this phase, we generated various items according to the different categories estab-
lished by the CECSS [10] and the Patient Satisfaction Questionnaire Short Form (PSQ-18) [2].
The items were elaborated and written through four consensus meetings with consultations
from experts in developing the measuring instruments. The research team consisted of
eight experts: two certified nurses in emergency nursing, one certified nurse in operation
nursing, three certified nurse specialists in critical care nursing, and a university professor
of nursing in critical care [36]. The first version of the questionnaire was developed on
the basis of the four dimensions of satisfactory care, with 34 items distributed among the
four factors. Additionally, we conducted our review using the keywords “emergency care”,
“emergency nursing”, and “patient satisfaction” and searched CINAHL, PubMed, Medline,
and other databases in the field. As a result, 12 items were identified, and 46 items in six
categories were generated.
(2) Content validation
Previous studies have reported that it is desirable to reflect the opinions of experts in
the field as well as patients and healthy people who are involved in the content validation
process [37]. Content validation of the first questionnaire version was conducted by an
expert panel of 10 healthy individuals who had visited an ER and 9 certified nurses in
emergency nursing [38] (Table A1). We sent questionnaires to the panel of 19 people and
asked them whether the survey items were valid. The survey items were assessed on a
4-point Likert scale ranging from 1 (“not at all important”) to 4 (“very important”). We also
asked about repetition, incomprehensibility, and ease of answering. Following the method
proposed by Davis, the item-level content validity index (I-CVI) was calculated by dividing
the number of experts who gave a rating of three or four for each item by the total number
of experts [39]. Items with an I-CVI lower than 0.78 were eliminated [40]. Numerical
codes were assigned to the completed forms to ensure confidentiality and anonymity of
the questionnaires. The final version of the questionnaire was established after deleting
five items on the basis of content validation research analysis and the results obtained from
various consensus meetings between the research team and expert advisors. The 46 items
included in the first version were reduced to 39 in the final version of the emergency
nursing-care patient satisfaction scale (ENPSS).
(3) Selecting items
Participants
The emergency medical care system in Japan is classified into three categories: primary
emergency facilities that mainly treat patients who can return home without the need for
hospitalization, secondary emergency facilities that mainly treat critically ill patients who
require hospitalization, and life-saving emergency centers that treat critically ill patients
who require advanced treatment [41]. In this study, patients who visited primary and
secondary emergency facilities were included. The patients fulfilled the following criteria
to participate in the validation study:
1. Age ≥ 18 years at the time of recruitment;
2. Capable of providing consent;
3. Ability to answer questionnaires;
4. Cognitive and physical capacity to complete self-administered questionnaires without
the need for a proxy.
Participants were recruited from the emergency departments of five facilities in Japan
between March 2021 and December 2021. The questionnaire was distributed to patients
who visited the ER and met the above criteria. The research collaborator, an emergency
nurse, asked each respondent to complete the questionnaire individually.
The sample size was targeted at >100 participants based on the COnsensus-based
Standards for the selection of health Measurement Instruments (COSMIN) checklist, a
guideline for scale development [37,42].
Healthcare 2022, 10, 518 4 of 20

Survey components
The survey consists of four components. The first was a questionnaire on individual
and institutional characteristics. The second part consisted of the number of visits to the ER,
the time of day when they visited the ER, and the length of time they waited in the ER. The
third component consisted of the 6-item EuroQol 5 dimensions 5-level (EQ-5D-5L) [43,44]
to test criterion-related validity and the intensity of distress during the ER visit and overall
satisfaction in the ER. The fourth component consisted of a satisfaction survey of the nurses
in the ER.
Instruments
The EQ-5D-5L is a validated and standardized instrument that measures health-related
quality of life (QOL) [43,44]. A Japanese version of the EQ-5D-5L is available [45]. The
EQ-5D-5L consists of the following five dimensions: mobility, self-care, usual activities,
pain/discomfort, and anxiety/depression. Each dimension has five levels: no problems,
slight problems, moderate problems, severe problems, and extreme problems. Health status
is represented in 3125 combinations, and each combination of answers can be converted
into a QOL score, ranging from 0 (death) to 1 (perfect health), according to a Japanese
value set [45]. We compared the scores from our study with those of a previously reported
Japanese norm [46]. The EQ-5D-5L also uses a visual analog scale (VAS) ranging from
0 to 100, where 0 represents the worst imaginable health and 100 represents the best
imaginable health.
To determine the content of the questions related to patient satisfaction with ER
nursing, we first examined the available information from previous studies [47–50]. Second,
we extracted content related to satisfaction with ER nursing. Third, on the basis of these
contents, five certified nurses were interviewed. On the basis of these results, the following
four items were adopted in this study, and each question was measured using the VAS
with “strongly agree” as 100 and “disagree” as 0.
1. Confidence in the ER’s physician;
2. Satisfaction with the response of ER’s physician;
3. Intensity of distress at the time of ER visit;
4. Satisfaction with the outcome of treatment in the ER.
For each of the patient satisfaction surveys for nurses in the ER, respondents rated
their level of agreement on a standard five-point Likert scale (0 = “not applicable”,
1 = “strongly disagree”, 2 = “slightly disagree”, 3 = “neutral”, 4 = “slightly agree”, and
5 = “strongly agree”).

2.2. Statistical Analysis

Descriptive statistics were derived for the analysis. Categorical data were expressed
as numbers and percentages.
We examined some of the questionnaire items for possible exclusion according to the
following criteria: items with a 20% rate or higher of “not applicable” [51] and items with
an average score of 4.5, 1.5, or lower for each item. Furthermore, one of the items with a
correlation coefficient of 0.7 or higher for each item was eliminated [52].
Exploratory factor analysis (EFA) using promax rotation and maximum likelihood
extraction methods was conducted to determine the number and type of factors from 38 of
the 39 survey items, excluding the question on overall satisfaction. EFA was conducted
on the complete data for all 38 items at baseline. The factor solution from the EFA was
based on the magnitude of the factor loadings for each item. On the basis of standard
psychometric criteria, items with factor loadings of less than 0.35 were eliminated. The
researchers assessed whether the elimination or retention of specific items was meaningful
for assessing patient satisfaction.
On the basis of the results of the EFA, a confirmatory factor analysis (CFA) was used
to evaluate factor solutions. The goal of the CFA was to evaluate the model fit of the
factor structure using the root mean square error of approximation (RMSEA: where <0.09
is considered acceptable and <0.06 is considered excellent) and comparative fit index (CFI:
Healthcare 2022, 10, 518 5 of 20

where >0.9 is considered acceptable). As a result, we named each factor that represented
various aspects of patient satisfaction with nursing in the ER.
The internal consistency reliability of the ENPSS was assessed using Cronbach’s alpha.
The estimates of reliability should exceed 0.70 (0.7 ≤ α < 0.8 is acceptable, 0.8 ≤ α < 0.9
is good, and 0.9 ≤ α is excellent) [53,54]. Construct validity was assessed using Pearson
correlations of the EFA-yielded domains and the ENPSS summary score with the five
validated questionnaires, namely, EQ-5D-5L (using the VAS), anxiety on EQ-5D-5L, distress
at ER visit (VAS), confidence in physician (VAS), and satisfaction with treatment (VAS). We
hypothesized that the ENPSS would correlate more with QOL [55] since satisfaction with
treatment is associated with QOL.
Only questionnaires with complete data were included in the analysis, and there was
no imputation of missing data. Statistical significance was set at p ≤ 0.05 (two-sided).
Statistical analyses were performed using SPSS Statistics version 27 (IBM Corp., Armonk,
NY, USA) and JMP Pro software version 15 (SAS Institute Inc., Cary, NC, USA).

2.3. Ethical Considerations

The protocol for this research project was approved by a suitably constituted Ethics
Committee of Sapporo Medical University and conformed to the provisions of the Dec-
laration of Helsinki, Approval No. 1-2-51. Informed consent was obtained from all the
respondents. Participants were informed of the purpose and length of the survey, and their
participation was voluntary. Consent was obtained from respondents by checking the box
on the front page of the questionnaires that they understood the research explanation and
agreed to participate according to Institutional Review Board recommendations.

3. Results
3.1. Population
A total of 127 respondent surveys were included in the final analysis after excluding
eight surveys with missing data. The characteristics of respondents are presented in Table 1.
Patients who were still employed accounted for 66.1%, and those with underlying diseases
accounted for 63.0% of the total. Fifty-two percent of the patients visited the ER during the
day, and 40.2% visited the ER for the first time.

Table 1. Participant sociodemographic and clinical characteristics (n = 127).

Characteristic n (%)
Age means (SD) 60.2 (16.4)
Sex
Male, n (%) 63 (49.6)
Female, n (%) 64 (50.4)
Employment situation
Unemployed, n (%) 43 (33.9)
Working, n (%) 84 (66.1)
Underlying disease, n (%)
Cardiovascular disease 24 (30.0)
Respiratory tract disease 19 (23.8)
Gastrointestinal tract disease 10 (12.5)
Cancer 12 (15.0)
Diabetes 11 (13.4)
Others 21 (26.3)
None 47 (37.0)
Number of past ER visits, n (%)
First time 51 (40.2)
Two times 45 (35.4)
Three times 21 (16.5)
Healthcare 2022, 10, 518 6 of 20

Table 1. Cont.

Characteristic n (%)
Four times 5 (3.9)
Five times 4 (3.1)
Six times 1 (0.8)
Time of visit to the ER, n (%)
Daytime (9:00–17:00) 66 (52.0)
Nighttime (17:00–9:00) 61 (48.0)
ER, emergency room; SD, standard deviation.

3.2. Selecting Items

A total of 127 participants (92%) had complete data on the ENPSS and constituted the
population used for factor analysis; this was sufficient for previous EFA studies [37,42,56,57].
First, of the 38 question items, 4 items of which more than 20% were answered as not
applicable were deleted. There were 31 pairs with correlation coefficients greater than 0.7.
We eliminated one of the items of the pair, and 14 items were excluded. Second, factor
analysis was conducted using the maximum likelihood method. In the EFA using promax
rotation, one item with a factor loading of less than 0.35 was removed, and finally, 20 items
were selected (Table 2).

Table 2. Individual items and their factor resolutions in a 21-item patient satisfaction questionnaire
regarding ER nursing.

ENPSS-21 Factor Items Factor 1 Factor 2 Factor 3 Factor 4

The nurse explained everything to me before
0.973 −0.081 −0.077 −0.007
every procedure
The nurse explained to me the schedule of tests
0.839 0.078 0.073 −0.068
and procedures in the ER
Factor 1: The nurse explained my condition to me in
0.743 −0.04 −0.054 0.135
explanation and terms I could understand
response (7 items) The nurse responded appropriately to
0.582 −0.025 0.39 −0.056
my requests
The nurse gave me a chance to ask questions 0.539 0.251 −0.008 0.081
The nurse understood my symptoms 0.389 0.246 0.127 0.099
The nurse took great care of my privacy 0.38 0.024 0.319 0.19
The nurse treated me with care, respect,
−0.192 0.765 0.282 −0.168
and compassion
The nurse alleviated my anxiety and concerns 0.27 0.649 −0.232 0.127
The attention to the personal appearance of the
−0.162 0.605 0.22 0.193
Factor 2: nurses was adequate
hospitality The nurse’s language was appropriate 0.309 0.566 −0.048 −0.12
(6 items) The nurse listened to me fully about
0.198 0.523 0.218 −0.169
my symptoms
The nurse interviewed me in detail (medical
history, medications, allergy history, chief 0.22 0.504 −0.173 0.248
complaint, history before the visit)
The collaboration between the nurse and the
−0.097 0.105 0.89 0.013
physician appeared to be smooth
Factor 3: The teamwork among the nurses appeared to
0.014 0.037 0.67 0.084
teamwork (3 items) be good
The infection control measures taken by the
0.242 0.042 0.431 0.239
nurses were appropriate
Healthcare 2022, 10, 518 7 of 20

Table 2. Cont.

ENPSS-21 Factor Items Factor 1 Factor 2 Factor 3 Factor 4

The nurse performed my procedure smoothly 0.024 −0.021 −0.025 0.877
The nurse quickly responded when I was in a lot
Factor 4: 0.09 −0.16 0.185 0.828
of pain
symptom
The nurse assessed my symptoms appropriately
management −0.036 0.425 0.114 0.476
and performed the procedure
(4 items)
The nurse performed my procedure in a
0.053 0.017 0.109 0.392
safe way

The EFA yielded 20 solution items loaded into four factors representing four domains:
explanation and response (seven items), hospitality (six items), teamwork (three items),
and symptom management (four items). The global satisfaction item was not included in
the EFA, which constitutes the ENPSS-21 in English version (see Appendix B) and Japanese
version (see Appendix C). This 21-item questionnaire’s factor structure CFA (Figure 1)
showed an acceptable fit: RMSEA = 0.1 (90% confidence interval = 0.08–0.11) and CFI = 0.9.
EFA, exploratory factor analysis; ENPSS, emergency nursing-care patient satisfaction scale;
CFA, confirmatory factor analysis; RMSEA, root mean square error of approximation; CFI,
comparative fit index.

3.3. Internal Consistency Reliability

The internal consistency reliability (Cronbach’s alpha) of the four domains of the
ENPSS-21 ranged from 0.81 to 0.89 (Table 3), corresponding to good internal reliability.

Table 3. Internal consistency reliability of ENPSS-21 domains.

ENPSS-21 Domain Mean (SD) Cronbach’s Alpha

Explanation and response 4.1 (0.6) 0.89
Hospitality 4.1 (0.6) 0.88
Teamwork 4.2 (0.5) 0.81
Symptom management 4.0 (0.6) 0.88
ENPSS-21, 21-item emergency nursing-care patient satisfaction scale; SD, standard deviation.

3.4. Construct Validity

The correlations of the four domain scores of the ENPSS-21 showed that the EQ-5D-5L
VAS (0.59–0.65, p < 0.01), EQ-5D-5L Anxiety (0.55–0.62, p < 0.01), confidence in the ER
physicians (0.56–0.63, p < 0.01), satisfaction with the response of ER physicians (0.57–0.66,
p < 0.01), and satisfaction with the outcome of treatment in the ER (0.51–0.62, p < 0.01) were
in the moderate to high range. The correlation was lower for the intensity of distress at
the time of the ER visit (−0.2–−0.29, p < 0.01). As hypothesized, the summary score of
ENPSS-21 showed the strongest correlation with EQ-5D-5L (0.68) and satisfaction with the
response of the ER physician (0.68). (Figure 2 and Table 4).
Healthcare 2022, 10, 518 8 of 20
Healthcare 2022, 10, x FOR PEER REVIEW 8 of 22

Figure1.1.Confirmatory
Figure Confirmatory factor
factor analysis.
analysis.
Healthcare 2022, 10, x FOR PEER REVIEW 10 of 22
Healthcare 2022, 10, 518 9 of 20

0.8

0.6 EQ-5D-5L VAS

EQ-5D-Anxiety
Correlation Coefficient

0.4

Confidence in the
emergency room
0.2 physicians

Satisfaction with the

response of emergency
room physicians
0
Intensity of distress at
the time of emergency
room visit

-0.2 Satisfaction with the

outcome of treatment
in the emergency room

-0.4

Summary score Explanation and Response Hospitality Teamwork Symptom management

Figure 2. Construct validity of the 19-item Emergency Nursing-Care Patient Satisfaction Question-
Figure
naire 2. Construct validity of the 19-item Emergency Nursing-Care Patient Satisfaction Question-
(ENPSS-21).
naire (ENPSS-21).
Table 4. Multitrait–multimethod correlations matrix.
4. Discussion
ENPSS-21 Domain Score
In this study, we developed a patient satisfaction scale focused on ER nursing, con-
Instrument Summary Explanation Symptom
firmed its validity and reliability, Score andand concluded
Response that it could be used in clinical
Hospitality Teamwork practice.
Management
To the EQ-5D-5L
best of our VAS
knowledge, 0.68
there are no
0.65
reports on developing
0.62
a
0.59
patient satisfaction
0.61
scale that focuses on ER nursing
EQ-5D-Anxiety 0.65 in Japan. 0.62 0.62 0.55 0.56
Confidence
The factor in the ER
analysis results
0.65 were composed
0.58 of four0.62domains:0.63 “explanation0.56 and re-
physicians
sponse”, “hospitality”, ”teamwork”, and “symptom management”. This multidimen-
Satisfaction with the response
0.68 0.66
sional ofstructure
ER physicians is consistent with many reports 0.63 on patient0.57 satisfaction 0.58 analysis
Intensity of distress at the
[2,10,24,58,59]. Furthermore, −the specific dimensions obtained in this study are similar to
0.27 −0.29 −0.25 −0.24 −0.20
time of ER visit
those found in other
Satisfaction with the outcome scales [10,60], which we consider partially supportive of the construct
0.62 0.62 0.55 0.51 0.56
validity of thisintool.
of treatment the ER
A limitation
ENPSS-21, when measuring
21-item emergency nursing-care patient
patient satisfaction is that
satisfaction scale; ER, psychometric
emergency room;properties
VAS, visualmay
ana-
log
notscale.
be reflected because cultural factors from different countries and regions are not ade-
quately taken into account [61]. Translation of existing patient satisfaction measures, such
4.
asDiscussion
CECSS and others, may lead to differences in the perception of quality of care from the
In this
patient's study, wedue
perspective developed a patientdifferences
to cross-cultural satisfaction scale
[62]. focused
Because on ER
of the nursing,
specific con-
situation
firmed its validity and reliability, and concluded that it could
in ERs, where patients are more urgent than in general wards and require a variety of be used in clinical practice.
To the best of
responses, our knowledge,
a specific there arewith
scale consistent no reports on developing
the culture a patient
of the country satisfactionessen-
is considered scale
that focuses on ER nursing in Japan.
tial. Therefore, a patient satisfaction scale in the ER that takes into account the Japanese
Thecontext
cultural factor analysis
was needed. results were composed of four domains: “explanation and re-
sponse”,The “hospitality”,
ENPSS-21 domain ”teamwork”, and “symptom
“explanation and response”management”. This multidimensional
includes nurse–patient commu-
structure is consistent with many reports on patient satisfaction
nication and provision of information. In order for the nurse to meet the needs of the pa- analysis [2,10,24,58,59].
Furthermore,
tient, a natural theand
specific dimensions
constructive obtainedmust
relationship in thisbestudy are similar
established [63].toNurses
those found in
can then
other scales [10,60], which we consider partially supportive of
provide counseling and guidance to patients to improve patient satisfaction [64]. In the the construct validity of
this
ER, tool.
providing information and communication to patients is also an important factor in
A limitation
facilitating patientwhencare andmeasuring patientfactor
is an essential satisfaction
in patient is satisfaction
that psychometric
[65]. properties
may not be reflected because cultural factors from different countries and regions are not
The second domain of ENPSS-21 is “hospitality”. The nurse’s concept of compassion
adequately taken into account [61]. Translation of existing patient satisfaction measures,
and interpersonal relationships is an important element in understanding patient. The
such as CECSS and others, may lead to differences in the perception of quality of care
from the patient’s perspective due to cross-cultural differences [62]. Because of the specific
Healthcare 2022, 10, 518 10 of 20

situation in ERs, where patients are more urgent than in general wards and require a
variety of responses, a specific scale consistent with the culture of the country is considered
essential. Therefore, a patient satisfaction scale in the ER that takes into account the
Japanese cultural context was needed.
The ENPSS-21 domain “explanation and response” includes nurse–patient commu-
nication and provision of information. In order for the nurse to meet the needs of the
patient, a natural and constructive relationship must be established [63]. Nurses can then
provide counseling and guidance to patients to improve patient satisfaction [64]. In the
ER, providing information and communication to patients is also an important factor in
facilitating patient care and is an essential factor in patient satisfaction [65].
The second domain of ENPSS-21 is “hospitality”. The nurse’s concept of compassion
and interpersonal relationships is an important element in understanding patient. The
hospitality domain also included items, such as courtesy and personal appearance. In
Japan, courtesy and personal appearance strongly influence patient satisfaction [66,67], and
we consider these cultural factors unique to Japan.
The third domain of ENPSS-21 is “teamwork”. Nurses are part of the healthcare
team and are expected to collaborate with other healthcare professionals involved in
patient care [63]. Moreover, nurses have an important task to fulfill as intermediaries
between multiple healthcare professions. Organizational teamwork has been reported
to be associated with satisfaction, and nurses need to practice in the best interest of the
patient [68]. Therefore, it is considered a very important perspective for ER nurses to
collaborate with ER physicians and co-medical staff to provide medical care.
The last domain of ENPSS-21 is “symptom management”. Patients in the ER have a
wide range of distress. Appropriate analgesia affects patient satisfaction [69]. The absence
of physical pain increased patient satisfaction with nursing care [70]. Therefore, distress
relief is considered by some patients to be equivalent to good nursing care [71]. Namely,
symptom management is considered an indispensable item for providing care that is
consistent with the needs of ER patients.
Moreover, this patient satisfaction scale was developed during the COIVD-19 pan-
demic. During the COVID-19 pandemic, healthcare professionals have reported that ade-
quate infection control measures are important for reducing the risk of viral infection and
patient anxiety about the virus [72]. The questionnaire items used in this study included
items related to infection control among nurses, which may include an important domain
of recent emergency care. In addition, the compassion and interpersonal relationships of
nurses are important factors for patient satisfaction [25,73,74].
Cronbach’s alpha for all domains of the ENPSS-21 scale was greater than 0.8. It
was found that the ENPSS-21 had similar values to previous patient satisfaction scales,
which were verified for internal consistency [59,75]. This means that each factor showed
appropriate homogeneity.
Of all the hypotheses used for construct validity, the “EQ-5D-5” and “satisfaction with
the response of ER physicians” were highly correlated with each of the four domains of
the ENPSS-21 and summary score. Previous reports have shown an association between
treatment satisfaction and QOL [55,58]. Patients with anxiety were also reported to be less
satisfied with their healthcare [76], consistent with the results of this scale. In addition,
satisfaction with physicians is related to overall satisfaction with healthcare [74], consistent
with the hypothesis validation in this study. In contrast, a high ENPSS-21 score was not
associated with the intensity of distress at the time of the emergency room visit. Although
this was low compared to the intensity of distress at the emergency room visit in this
study [71], it is consistent with the reported finding that VAS pain scores in the ER do not
correlate with patient satisfaction [77].

5. Limitations
The current study has several limitations. First, test–retest reliability was not validated
in this study. Having good test–retest reliability implies internal consistency of the test and
Healthcare 2022, 10, 518 11 of 20

ensures that the measurements obtained are representative [42]. In the future, a test–retest
should be conducted to confirm reliability. Second, there are concerns about the time
when data collection took place. In this study, data were collected during the COVID-19
pandemic period. As a result, the ER system was probably different from normal, which
could have affected the assessment of patients. Third, this study used a minimum sample
size of 100 for factor analysis from previous studies [37,42]. However, for CFA, a minimum
sample of 150 is required [78]. This study did not meet that requirement, thus limiting the
results of the analysis. In the future, it will be an issue to refine the items of the scale while
taking the sample size into consideration.

6. Implications for Clinical Practice

ENPSS-21 is a brief, reliable, and valid instrument that can obtain information about
patient satisfaction with ER nursing. The instrument has direct clinical utility for improving
the quality of nursing care in the ER in Japan by providing a patient-centered perspective
on satisfaction. The ENPSS-21 also helps address the weaknesses of the organization
by measuring satisfaction regularly and comparing patient satisfaction levels relative to
each other.

7. Conclusions
This study found that the ENPSS-21 was a robust measure of patient satisfaction,
suggesting that it is possible to measure satisfaction with ER nursing. The ENPSS-21 is
designed to focus on the nursing perspective of the ER. In creating the items, the ENPSS
was designed to represent the emergency nursing care sought by patients on the basis of
an extensive literature search and content validity. The ENPSS-21 consists of 21 items in
four domains. It showed the highest correlation with the EQ-5D-5L and satisfaction with
the response of ER physicians in criterion-related validity. By measuring patient-centered
perspectives of satisfaction with ER nursing, factors lacking in each organization’s ER can
be identified and addressed to improve the quality of nursing care in the ER.

Author Contributions: Conceptualization, J.H., N.M. (Naomi Minamoto), Y.T. and N.M.
(Natsuko Makino); data curation, N.M. (Naomi Minamoto), M.S. and N.M. (Natsuko Makino);
formal analysis, J.H.; funding acquisition, M.S.; investigation, J.H.; methodology, J.H., M.S. and N.K.;
writing—original draft, J.H.; writing—review and editing, J.H., N.M. (Naomi Minamoto), M.S., Y.T.,
N.M. (Natsuko Makino), N.K. and H.U. All authors have read and agreed to the published version of
the manuscript.
Funding: This study was supported by a Research Grant from Sapporo Medical University (Mizue
Shiromaru, project CD: 2100188).
Institutional Review Board Statement: Approval of the research protocol: The protocol for this
research project was approved by a suitably constituted Ethics Committee of the Sapporo Medical
University and conforms to the provisions of Declaration of Helsinki, Approval No. 1-2-51.
Informed Consent Statement: Informed consent was obtained from all the respondents. Participants
were informed of the purpose and length of the survey, and their participation was voluntary.
Checking the box at the beginning of the questionnaire stating that they understood the study
description and agreed to participate was considered consent.
Data Availability Statement: All data from this research have been included within the manuscript.
Acknowledgments: We thank Junichi Hattori, Fumitaka Arisawa, Hideaki Okamura, Momoe Tanima,
and Keiko Shimazu for data collection.
Conflicts of Interest: The authors declare no conflict of interest.
Healthcare 2022, 10, 518 12 of 20

Appendix A

Table A1. Characteristics of expert panel, n = 19.

Characteristic n
Certified nurses in emergency nursing, n = 10
Sex, male 3
Age, mean (SD) 40.6 (4.9)
Work experience in emergency care 13.5 (3.2)
Healthy individuals who had visited an ER, n = 9
Sex, male 3
Age, mean (SD) 42.8 (9.0)
ER, emergency room; SD, standard deviation.

Appendix B. Nursing Emergency-Care Patient Satisfaction Scale—21 (ENPSS-21)

English Version
The following questionnaire asks about the patient level of satisfaction with emergency
room nurses. The goal of this questionnaire is to measure your level of satisfaction or
dissatisfaction with the nurses on the basis of what you currently feel. Please place a check
mark in the appropriate box for each question.
The following 20 questions relate to satisfaction with emergency room nurses.
1. The nurse explained everything to me before every procedure.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
2. The nurse explained the schedule of tests and procedures in the emergency room
to me.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
3. The nurse explained my condition to me in terms I could understand.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
4. The nurse responded appropriately to my requests.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
5. The nurse gave me a chance to ask questions.
 Strongly disagree
 Slightly disagree
Healthcare 2022, 10, 518 13 of 20

 Neutral
 Slightly agree
 Strongly agree
 Not applicable
6. The nurse understood my symptoms.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
7. The nurse took great care of my privacy.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
8. The nurse treated me with care, respect, and compassion.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
9. The nurse alleviated my anxiety and concerns.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
10. The attention to the personal appearance of the nurses was adequate.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
11. The nurse’s language was appropriate.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
12. The nurse listened to me fully about my symptoms.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
Healthcare 2022, 10, 518 14 of 20

13. The nurse interviewed me in detail (medical history, medications, allergy history, chief
complaint, and medical history before the visit).
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
14. The collaboration between the nurse and the physician appeared to be smooth.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
15. The teamwork among the nurses appeared to be good.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
16. The infection control measures taken by the nurses were appropriate.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
17. The nurse performed my procedure smoothly.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
18. The nurse quickly responded when I was in a lot of pain.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
19. The nurse assessed my symptoms appropriately and performed the procedure.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable
20. The nurse safely performed my procedure.
 Strongly disagree
 Slightly disagree
 Neutral
Healthcare 2022, 10, 518 15 of 20

 Slightly agree
 Strongly agree
 Not applicable
The following question is about your overall satisfaction with emergency room nurses.
Please place a check mark in the appropriate box.
21. I felt that the overall quality of nursing care I received in the emergency room
was good.
 Strongly disagree
 Slightly disagree
 Neutral
 Slightly agree
 Strongly agree
 Not applicable

Appendix C. Nursing Emergency-Care Patient Satisfaction Scale—21 (ENPSS-21)

Japanese Version
以下のアンケートは、救急外来の看護師に対するあなたの満足度を調査するもので
す。このアンケートの目的は、あなたが現在感じている、看護師に対する満足感を測定
することです。 各質問について、該当するボックスにチェックマークを入れてくださ
い。
以下の20の質問は、救急外来の看護師に対する満足度に関するものです。
1. 看護師は全ての処置の前に説明してくれましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
2. 看護師は、救急外来での検査や処置の予定について説明しましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
3. 看護師は、あなたの病状に関して理解できる言葉で説明しましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
4. 看護師は、あなたの要望に適切に対応しましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
5. 看護師は、あなたに質問する機会を与えてくれましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
Healthcare 2022, 10, 518 16 of 20

 とてもそう思う
 該当なし
6. 看護師は、あなたの症状を理解していましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
7. 看護師は、あなたのプライバシーに対して十分に考慮しましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
8. 看護師は、あなたに丁寧かつ敬意や思いやりのある対応をしましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
9. 看護師によって、あなたの不安や心配事は軽減されましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
10. 看護師の身だしなみへの配慮は十分でしたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
11. 看護師の言葉遣いは適切でしたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
12. 看護師は、あなたの症状について十分に話を聞いていましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
13. 看護師は、あなたに対して詳しく問診（既往歴、内服薬、アレルギー歴、主訴、受
診前の経緯などの聴取）をしていましたか。
 全くそう思わない
Healthcare 2022, 10, 518 17 of 20

 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
14. 看護師と医師との連携はスムーズでしたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
15. 看護師同士のチームワークは良かったですか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
16. 看護師の感染対策は十分と感じましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
17. 看護師は、あなたの処置をスムーズに行っていましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
18. 看護師は、あなたの苦痛が強いとき迅速に対応してくれましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
19. 看護師は、あなたの症状を適切に判断して処置を行なっていましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
20. 看護師は、処置を行う際、安全に配慮していましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし
Healthcare 2022, 10, 518 18 of 20

次の質問は、救急外来の看護師に対するあなたの総合的な満足度についてです。
該当するボックスにチェックマークを入れてください。
21. 救急外来受診中に受けた看護ケアの全体的な質は良いと感じましたか。
 全くそう思わない
 そう思わない
 どちらでもない
 そう思う
 とてもそう思う
 該当なし

References
1. Kamimura, A.; Ashby, J.; Myers, K.; Nourian, M.M.; Christensen, N. Satisfaction with healthcare services among free clinic
patients. J. Community Health 2015, 40, 62–72. [CrossRef] [PubMed]
2. Marshall, G.N.; Hays, R.D. The Patient Satisfaction Questionnaire Short-Form (PSQ-18); Rand: Santa Monica, CA, USA, 1994.
3. Fufa, B.D.; Negao, E.B. Satisfaction of Outpatient Service Consumers and Associated Factors Towards the Health Service Given at
Jimma Medical Center, South West Ethiopia. Patient Relat. Outcome Meas. 2019, 10, 347–354. [CrossRef] [PubMed]
4. Rahmqvist, M.; Bara, A.-C. Patient characteristics and quality dimensions related to patient satisfaction. Int. J. Qual. Health Care
2010, 22, 86–92. [CrossRef]
5. Turris, S.A. Unpacking the concept of patient satisfaction: A feminist analysis. J. Adv. Nurs. 2005, 50, 293–298. [CrossRef]
6. Ak, M.; Cinar, O.; Sutcigil, L.; Congologlu, E.D.; Haciomeroglu, B.; Canbaz, H.; Yaprak, H.; Jay, L.; Ozmenler, K.N. Communication
skills training for emergency nurses. Int. J. Med. Sci. 2011, 8, 397–401. [CrossRef]
7. Dilig-Ruiz, A.; MacDonald, I.; Demery Varin, M.; Vandyk, A.; Graham, I.D.; Squires, J.E. Job satisfaction among critical care
nurses: A systematic review. Int. J. Nurs. Stud. 2018, 88, 123–134. [CrossRef]
8. Harley, A.; Johnston, A.N.B.; Denny, K.J.; Keijzers, G.; Crilly, J.; Massey, D. Emergency nurses’ knowledge and understanding
of their role in recognising and responding to patients with sepsis: A qualitative study. Int. Emerg. Nurs. 2019, 43, 106–112.
[CrossRef]
9. An, Y.; Yang, Y.; Wang, A.; Li, Y.; Zhang, Q.; Cheung, T.; Ungvari, G.S.; Qin, Z.M.; An, F.-R.; Xiang, Y.-T. Prevalence of depression
and its impact on quality of life among frontline nurses in emergency departments during the COVID-19 outbreak. J. Affect.
Disord. 2020, 276, 312–315. [CrossRef] [PubMed]
10. Davis, B.A.; Bush, H.A. Developing effective measurement tools: A case study of the Consumer Emergency Care Satisfaction
Scale. J. Nurs. Care Qual. 1995, 9, 26–35. [CrossRef] [PubMed]
11. Abidova, A.; da Silva, P.A.; Moreira, S. The mediating role of patient satisfaction and perceived quality of healthcare in the
emergency department. Medicine 2021, 100, e25133. [CrossRef]
12. Chan, J.N.H.; Chau, J. Patient satisfaction with triage nursing care in Hong Kong. J. Adv. Nurs. 2005, 50, 498–507. [CrossRef]
13. Elder, R.; Neal, C.; Davis, B.A.; Almes, E.; Whitledge, L.; Littlepage, N. Patient satisfaction with triage nursing in a rural hospital
emergency department. J. Nurs. Care Qual. 2004, 19, 263–268. [CrossRef] [PubMed]
14. Davis, B.A.; Bush, H.A. Patient satisfaction of emergency nursing care in the United States, Slovenia, and Australia. J. Nurs. Care
Qual. 2003, 18, 267–274. [CrossRef] [PubMed]
15. Deák, A.; Fusz, K.; Kanizsai, P. Patient satisfaction during the COVID-19 pandemic in an emergency department. Orv. Hetil. 2020,
161, 1819–1823. [CrossRef] [PubMed]
16. Sonis, J.D.; Aaronson, E.L.; Lee, R.Y.; Philpotts, L.L.; White, B.A. Emergency Department Patient Experience: A Systematic Review
of the Literature. J. Patient Exp. 2018, 5, 101–106. [CrossRef] [PubMed]
17. Aaronson, E.L.; Mort, E.; Sonis, J.D.; Chang, Y.; White, B.A. Overall Emergency Department Rating: Identifying the Factors That
Matter Most to Patient Experience. J. Health Qual. 2018, 40, 367–376. [CrossRef] [PubMed]
18. Phiri, M.; Heyns, T.; Coetzee, I. Patients’ experiences of triage in an emergency department: A phenomenographic study. Appl.
Nurs. Res. 2020, 54, 151271. [CrossRef]
19. Meterko, M.; Mohr, D.C.; Young, G.J. Teamwork culture and patient satisfaction in hospitals. Med. Care 2004, 42, 492–498.
[CrossRef] [PubMed]
20. Vahey, D.C.; Aiken, L.H.; Sloane, D.M.; Clarke, S.P.; Vargas, D. Nurse burnout and patient satisfaction. Med. Care. 2004, 42,
II57–II66. [CrossRef]
21. Lu, D.W.; Weygandt, P.L.; Pinchbeck, C.; Strout, T.D. Emergency Medicine Trainee Burnout Is Associated With Lower Patients’
Satisfaction With Their Emergency Department Care. AEM Educ. Train. 2018, 2, 86–90. [CrossRef]
22. Nápoles-Springer, A.M.; Santoyo, J.; Houston, K.; Pérez-Stable, E.J.; Stewart, A.L. Patients’ perceptions of cultural factors affecting
the quality of their medical encounters. Health Expect. 2005, 8, 4–17. [CrossRef] [PubMed]
23. Woods, S.E.; Bivins, R.; Oteng, K.; Engel, A. The influence of ethnicity on patient satisfaction. Ethn. Health 2005, 10, 235–242.
[CrossRef] [PubMed]
24. Xesfingi, S.; Vozikis, A. Patient satisfaction with the healthcare system: Assessing the impact of socio-economic and healthcare
provision factors. BMC Health Serv. Res. 2016, 16, 94. [CrossRef] [PubMed]
Healthcare 2022, 10, 518 19 of 20

25. Prakash, B. Patient satisfaction. J. Public Health 2010, 3, 151–155. [CrossRef]

26. McCleery, E.; Christensen, V.; Peterson, K.; Humphrey, L.; Helfand, M. Evidence Brief: The Quality of Care Provided by Advanced
Practice Nurses. VA Evidence Synthesis Program Evidence Briefs; Department of Veterans Affairs (US): Washington, DC, USA, 2014.
27. Evans, D.K.; Welander Tärneberg, A. Health-care quality and information failure: Evidence from Nigeria. Health Econ. 2018, 27,
e90–e93. [CrossRef]
28. Almeida RS de Bourliataux-Lajoinie, S.; Martins, M. Satisfaction measurement instruments for healthcare service users: A
systematic review. Cad. Saúde Pública 2015, 31, 11–25.
29. Shimizu, K.; Hibino, S.; Biros, M.H.; Irisawa, T.; Shimazu, T. Emergency medicine in Japan: Past, present, and future. Int. J. Emerg.
Med. 2021, 14, 2. [CrossRef]
30. Ministry of Health, Labour and Welfare. The Ageing Society: Current Situation and Implementation Measures FY 2017. Available
online: https://www8.cao.go.jp/kourei/english/annualreport/2018/pdf/c1-1.pdf (accessed on 28 January 2022).
31. Katayama, Y.; Kiyohara, K.; Kitamura, T.; Hayashida, S.; Shimazu, T. Influence of the COVID-19 pandemic on an emergency
medical service system: A population-based, descriptive study in Osaka, Japan. Acute Med. Surg. 2020, 7, e534. [CrossRef]
32. Ikegami, N.; Yoo, B.-K.; Hashimoto, H.; Matsumoto, M.; Ogata, H.; Babazono, A.; Watanabe, R.; Shibuya, K.; Yang, B.-M.; Reich,
M.R.; et al. Japanese universal health coverage: Evolution, achievements, and challenges. Lancet 2011, 378, 1106–1115. [CrossRef]
33. Minarik, P.A.; Chan, G.K.; Usami, S. CNS Role and Practice in Japan. In Clinical Nurse Specialist Role and Practice: An International
Perspective; Fulton, J.S., Holly, V.W., Eds.; Springer: Cham, Denmark, 2021; pp. 171–181.
34. Grissom, M.O.; Farra, M.; Cruzen, E.S.; Barlow, E.; Gupta, S. What can COVID-19 teach us about patient satisfaction in the
emergency department? A mixed-methods approach. J. Am. Coll. Emerg. Physicians Open 2021, 2, e12436. [CrossRef]
35. King, D.M.; Vakkalanka, J.P.; Junker, C.; Harland, K.K.; Nugent, A.S. Emergency Department Overcrowding Lowers Patient
Satisfaction Scores. Acad. Emerg. Med. 2021, 28, 363–366. [CrossRef] [PubMed]
36. Romero-García, M.; de la Cueva-Ariza, L.; Jover-Sancho, C.; Delgado-Hito, P.; Acosta-Mejuto, B.; Sola-Ribo, M.; Juandó-Prats, C.;
Ricart-Basagaña, M.T.; Sola-Sole, N. La percepción del paciente crítico sobre los cuidados enfermeros: Una aproximación al
concepto de satisfacción. Enferm. Intensiv. 2013, 24, 51–62. [CrossRef] [PubMed]
37. Anthoine, E.; Moret, L.; Regnault, A.; Sébille, V.; Hardouin, J.-B. Sample size used to validate a scale: A review of publications on
newly-developed patient reported outcomes measures. Health Qual. Life Outcomes 2014, 12, 176. [CrossRef] [PubMed]
38. Bilsbury, C.D.; Richman, A. A staging approach to measuring patient-centred subjective outcomes. Acta Psychiatr. Scand. 2002,
106, 5–40. [CrossRef] [PubMed]
39. Davis, L.L. Instrument review: Getting the most from a panel of experts. Appl. Nurs. Res. 1992, 5, 194–197. [CrossRef]
40. Polit, D.F.; Beck, C.T. The content validity index: Are you sure you know what’s being reported? Critique and recommendations.
Res. Nurs. Health 2006, 29, 489–497. [CrossRef]
41. Tanigawa, K.; Tanaka, K. Emergency medical service systems in Japan: Past, present, and future. Resuscitation 2006, 69, 365–370.
[CrossRef]
42. Mokkink, L.B.; Terwee, C.B.; Patrick, D.L.; Alonso, J.; Stratford, P.W.; Knol, D.L.; Alonso, J.; Stratford, P.W.; Knol, D.L.; Bouter,
L.M. COSMIN Checklist Manual. Amsterdam: University Medical Center. 2012. Available online: http://fac.ksu.edu.sa/sites/
default/files/cosmin_checklist_manual_v9.pdf (accessed on 3 January 2022).
43. Herdman, M.; Gudex, C.; Lloyd, A.; Janssen, M.; Kind, P.; Parkin, D.; Bonsel, G.; Badia, X. Development and preliminary testing
of the new five-level version of EQ-5D (EQ-5D-5L). Qual. Life Res. 2011, 20, 1727–1736. [CrossRef]
44. Janssen, M.F.; Pickard, A.S.; Golicki, D.; Gudex, C.; Niewada, M.; Scalone, L.; Swinburn, P.; Busschbach, J. Measurement properties
of the EQ-5D-5L compared to the EQ-5D-3L across eight patient groups: A multi-country study. Qual. Life Res. 2013, 22, 1717–1727.
[CrossRef]
45. Shiroiwa, T.; Ikeda, S.; Noto, S.; Igarashi, A.; Fukuda, T.; Saito, S.; Shimozuma, K. Comparison of Value Set Based on DCE and/or
TTO Data: Scoring for EQ-5D-5L Health States in Japan. Value Health 2016, 19, 648–654. [CrossRef]
46. Shiroiwa, T.; Fukuda, T.; Ikeda, S.; Igarashi, A.; Noto, S.; Saito, S.; Shimozuma, K. Japanese population norms for preference-based
measures: EQ-5D-3L, EQ-5D-5L, and SF-6D. Qual. Life Res. 2016, 25, 707–719. [CrossRef] [PubMed]
47. Nirel, N.; Goldwag, R.; Feigenberg, Z.; Abadi, D.; Halpern, P. Stress, work overload, burnout, and satisfaction among paramedics
in Israel. Prehospital Disaster Med. 2008, 23, 537–546. [CrossRef] [PubMed]
48. Avraham, N.; Goldblatt, H.; Yafe, E. Paramedics’ experiences and coping strategies when encountering critical incidents. Qual.
Health Res. 2014, 24, 194–208. [CrossRef] [PubMed]
49. Crowe, R.P.; Bower, J.K.; Cash, R.E.; Panchal, A.R.; Rodriguez, S.A.; Olivo-Marston, S.E. Association of Burnout with Workforce-
Reducing Factors among EMS Professionals. Prehospital Emerg. Care 2018, 22, 229–236. [CrossRef]
50. Lawn, S.; Roberts, L.; Willis, E.; Couzner, L.; Mohammadi, L.; Goble, E. The effects of emergency medical service work on
the psychological, physical, and social well-being of ambulance personnel: A systematic review of qualitative research. BMC
Psychiatry 2020, 20, 348. [CrossRef]
51. Streiner, D.L.; Norman, G.R.; Cairney, J. Health Measurement Scales: A Practical Guide to Their Development and Use; Oxford
University Press: Oxford, UK, 2015.
52. Ferketich, S. Focus on psychometrics. Aspects of item analysis. Res. Nurs. Health 1991, 14, 165–168. [CrossRef]
53. Bland, J.M.; Altman, D.G. Cronbach’s alpha. BMJ 1997, 314, 572. [CrossRef]
54. Nunnally, J.C. Psychometric Theory 3E; Tata McGraw-Hill Education: New York, NY, USA, 1994.
Healthcare 2022, 10, 518 20 of 20

55. Revicki, D.A. Patient assessment of treatment satisfaction: Methods and practical issues. Gut 2004, 53 (Suppl. S4), iv40–iv44.
[CrossRef]
56. de Winter, J.C.F.; Dodou, D.; Wieringa, P.A. Exploratory Factor Analysis With Small Sample Sizes. Multivar. Behav. Res. 2009, 44,
147–181. [CrossRef]
57. Kline, P. An Easy Guide to Factor Analysis; Routledge: London, UK, 2014.
58. Wiebe, S.; Wahby, S.; Lawal, O.A.; Sajobi, T.T.; Keezer, M.R.; Nguyen, D.K.; Malmgren, K.; Tellez-Zenteno, J.; Atkinson, M.J.;
Hader, W.J.; et al. Development and validation of the Epilepsy Surgery Satisfaction Questionnaire (ESSQ-19). Epilepsia 2020, 61,
2729–2738. [CrossRef]
59. Romero-García, M.; De La Cueva-Ariza, L.; Benito-Aracil, L.; Lluch-Canut, T.; Trujols-Albet, J.; Martínez, M.A.; Juvé-Udina, M.-E.;
Delgado-Hito, P.; Benito-Aracil, L.; Lluch-Canut, T. Nursing Intensive-Care Satisfaction Scale [NICSS]: Development and
validation of a patient-centred instrument. J. Adv. Nurs. 2018, 74, 1423–1435. [CrossRef] [PubMed]
60. Naidoo, T.; Tomita, A.; Paruk, S. Burnout, anxiety and depression risk in medical doctors working in KwaZulu-Natal Province,
South Africa: Evidence from a multi-site study of resource-constrained government hospitals in a generalised HIV epidemic
setting. PLoS ONE 2020, 15, e0239753. [CrossRef] [PubMed]
61. Horne, R.; Hankins, M.; Jenkins, R. The Satisfaction with Information about Medicines Scale (SIMS): A new measurement tool for
audit and research. BMJ Qual. Saf. 2001, 10, 135–140. [CrossRef]
62. Grimshaw, J.M.; Eccles, M.P.; Lavis, J.N.; Hill, S.J.; Squires, J.E. Knowledge translation of research findings. Implement. Sci. 2012,
7, 50. [CrossRef] [PubMed]
63. Johansson, P.; Oléni, M.; Fridlund, B. Patient satisfaction with nursing care in the context of health care: A literature study. Scand.
J. Caring Sci. 2002, 16, 337–344. [CrossRef] [PubMed]
64. Morgan, A.K. Client education experiences in professional nursing practice—A phenomenological perspective. J. Adv. Nurs.
1994, 19, 792–801. [CrossRef] [PubMed]
65. Abdel Maqsood, A.S.; Oweis, A.I.; Hasna, F.S. Differences between patients’ expectations and satisfaction with nursing care in a
private hospital in Jordan. Int. J. Nurs. Pr. 2012, 18, 140–146. [CrossRef] [PubMed]
66. Elleuch, A. Patient satisfaction in Japan. Int. J. Health Care Qual. Assur. 2008, 21, 692–705. [CrossRef] [PubMed]
67. Tokunaga, J.; Imanaka, Y.; Nobutomo, K. Effects of patient demands on satisfaction with Japanese hospital care. Int. J. Qual.
Health Care 2000, 12, 395–401. [CrossRef]
68. Leiter, M.P.; Harvie, P.; Frizzell, C. The correspondence of patient satisfaction and nurse burnout. Soc. Sci. Med. 1998, 47,
1611–1617. [CrossRef]
69. Megivern, K.; Halm, M.A.; Jones, G. Measuring patient satisfaction as an outcome of nursing care. J. Nurs. Care Qual. 1992, 6,
9–24. [CrossRef] [PubMed]
70. Redmond, G.M.; Sorrell, J.M. Studying patient satisfaction: Patient voices of quality. Outcomes Manag. Nurs. Pract. 1999, 3, 67–72.
71. Lövgren, G.; Engström, B.; Norberg, A. Patients’ narratives concerning good and bad caring. Scand. J. Caring Sci. 1996, 10, 151–156.
[CrossRef] [PubMed]
72. Aashima Nanda, M.; Sharma, R. A Review of Patient Satisfaction and Experience with Telemedicine: A Virtual Solution During
and Beyond COVID-19 Pandemic. Telemed. e-Health 2021, 27, 1325–1331. [CrossRef]
73. Batbaatar, E.; Dorjdagva, J.; Luvsannyam, A.; Savino, M.M.; Amenta, P. Determinants of patient satisfaction: A systematic review.
Perspect. Public Health 2017, 137, 89–101. [CrossRef]
74. Schoenfelder, T.; Klewer, J.; Kugler, J. Determinants of patient satisfaction: A study among 39 hospitals in an in-patient setting in
Germany. Int. J. Qual. Health Care 2011, 23, 503–509. [CrossRef]
75. Laschinger, H.S.; Hall, L.M.; Pedersen, C.; Almost, J.A. Psychometric analysis of the patient satisfaction with nursing care quality
questionnaire: An actionable approach to measuring patient satisfaction. J. Nurs. Care Qual. 2005, 20, 220–230. [CrossRef]
76. Abtahi, A.M.; Brodke, D.S.; Lawrence, B.D.; Zhang, C.; Spiker, W.R. Association between patient-reported measures of psycholog-
ical distress and patient satisfaction scores after spine surgery. J. Bone Jt. Surg. Am. Vol. 2015, 97, 824–828. [CrossRef]
77. Kelly, A.M. Patient satisfaction with pain management does not correlate with initial or discharge VAS pain score, verbal pain
rating at discharge, or change in VAS score in the Emergency Department. J. Emerg. Med. 2000, 19, 113–116. [CrossRef]
78. Kyriazos, T.A. Applied psychometrics: Sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in
general. Psychology 2018, 9, 2207–2230. [CrossRef]