Heliyon 6 (2020) e03174

Contents lists available at ScienceDirect

Heliyon
journal homepage: www.cell.com/heliyon

Research article

Designing a health referral mobile application for high-mobility end users

in Indonesia
Ave Adriana Pinem a, *, Andi Yeskafauzan a, Putu Wuri Handayani a, Fatimah Azzahro a,
Achmad Nizar Hidayanto a, Dumilah Ayuningtyas b
a
Faculty of Computer Science, Universitas Indonesia, Indonesia
b
Faculty of Public Health, Universitas Indonesia, Indonesia

A R T I C L E I N F O A B S T R A C T

Keywords: Objective: The purpose of this study was to design a suitable mobile application for high mobility medical staff to
Information science support the health referral system of Indonesian universal health coverage.
Mobile application design Methods: Design science research (DSR) approach were conducted with two iterations. The first iteration was
Design science approach
conducted by designing a high-fidelity mockup and evaluating the design using the System Usability Scale (SUS)
Health referral system
framework to evaluate the overall usability of the system (n ¼ 48). The second iteration was conducted by
SUS
PSSUQ designing a running prototype and evaluating the design using the Post-Study System Usability Questionnaire
(PSSUQ) framework to evaluate the system's usefulness, information quality, and interface quality (n ¼ 31). For
designing the user interface, the eight golden rules guided the process.
Results: The results found that the design for the first iteration was good and the second iteration's design was
satisfactory to the respondents. The information quality aspects underwent several changes such as the imple-
mentation of local language as the users deemed it to be important to help them easily understand and use the
system and eventually support their mobility at the health facility.

1. Introduction (m-health) applications have been created with specific objectives, such
as monitoring health conditions and remote consultation.
The development of Information Technology (IT) in Indonesia, which The Indonesian Government has implemented several rules, pro-
is growing rapidly, is being driven by the presence of the Internet. In grams, and health efforts, which were created by the Ministry of Health,
2017, the number of Internet users in Indonesia reached more than 143 to improve health services. One such effort is the health referral system,
million (~54.68% of the total population) [1]. This growth has been and which became regulated by the Minister of Health Regulation, Republic
continues to be influenced by the utilization of smartphones. Based on a of Indonesia Regulation 001 in 2012. This health referral system is car-
2017 survey by the Association of Internet Providers in Indonesia (APJII), ried out both vertically and horizontally. Vertical referrals start from
50% of Indonesians have smartphone devices, which they mainly use for primary health facilities (Fasilitas Kesehatan Tingkat Pertama: FKTP);
Internet access. Smartphones significantly impact the number of Internet therefore, every citizen first visits his/her assigned health facility when
users in Indonesia because they provide wireless access [2]. In addition, having a health problem. A designated health facility can refer a patient
smartphones have more capabilities such as more applications, high to a higher-level health facility (Fasilitas Kesehatan Rujukan Tingkat
resolution camera and friendly screen interaction than did previous Lanjut: FKRTL) if they lack the appropriate medical staff and/or treat-
phone technology than did previous phone technology. Widespread ment options. However, patients with emergencies do not have to use the
connectivity to the Internet also provides more opportunities for appli- referral system. Per Regulation 001, participants of the National Health
cation developers to create solutions to daily activity issues [3], which Insurance program (Jaminan Kesehatan Nasional: JKN) are required to
make smartphone utilization inherent in our daily lives [4]. These ap- follow the referral system. Participants with commercial health insurance
plications can solve transportation, logistics, financial management, must follow tiered services within the referral system and rules that are
business, and health-related activity problems. Various mobile health adjusted according to the provisions of their insurance policies.

* Corresponding author.
E-mail address: [email protected] (A.A. Pinem).

https://doi.org/10.1016/j.heliyon.2020.e03174
Received 23 July 2019; Received in revised form 29 November 2019; Accepted 2 January 2020
2405-8440/© 2020 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
A.A. Pinem et al. Heliyon 6 (2020) e03174

To support the health referral process, the government has developed a M-health is a subclass of electronic health (e-health), with the dif-
web application, which is used by all health facilities in Indonesia to refer ference being that m-health requires the use of a mobile device [14].
patients to other health facilities. Medical staff members are the main However, the two share the goal of increasing both the effectiveness and
users of the system. Currently, the mobile version of this application the efficiency of medical personnel in providing health services [15].
(Mobile JKN) is only designed to help patients register themselves at M-health applications should provide immediate information to support
designated primary health facilities; the feature that will allow medical medical staff members' decisions regarding patient care both anytime
staff to send referrals has yet to be implemented. The implementation of an and anywhere, making its purpose broader than that of e-health.
m-health application for referrals is important for medical staff, given their M-health applications also seek to help patients improve their health and
high level of mobility. In addition, as mentioned earlier, most Indonesians engage in preventative care by providing information regarding their
are more accustomed to accessing the Internet using smartphones. Thus, current healthcondition.
the implementation of an m-health application for the referral system,
especially the medical staff module, is expected to increase the produc-
tivity of medical staff, impact the provision of health care, and eventually 2.2. M-Health implementation
improve the effectiveness and efficiency of the referral process. The Mo-
bile JKN patient module still has some problems, as perceived by its users, The development of mobile technology for the promotion of health
which are resulting in unsatisfactory utilization of the system. These services can be partially linked to technological accessibility, the level of
problems include usability, ease of use, and user interface [5]. This study personalization, useful location-based services, and timely access to in-
was conducted to support medical staff practice in using an m-health formation [16]. Several studies have evaluated the use of m-health for
referral system and to correct issues until the system met their expected chronic conditions, such as diabetes [17], chronic obstructive pulmonary
usability. This system was chosen for study because of the large number of disease [18], Alzheimer's disease [19], and osteoarthritis [20]. These
possible users (there are more than 200 million JKN participants) [6]. studies found that the use of m-health applications can support success in
Previous studies on designing m-health applications have been car- the management of chronic diseases and healthy behavioral changes.
ried out using a unified model language (UML) as the communication However, reaching the objectives of m-health requires significant effort,
medium because it can be explained and understood easily by the end and the process remains a challenge. Obstacles can include resistance to
user [7, 8, 9]. This allows both the developer and end user to have the change, unreliable technology, integration issues, and lack of user edu-
same perception of the application. However, we only found one study cation [21], and documentation of these obstacles regarding the design,
that explained what methodology would be suitable for designing an implementation, and evaluation of m-health is still quite limited. Based
m-health application [9]: used the Design Science Research (DSR) on a survey conducted by the WHO, only 12% of member countries have
method in one iteration to describe the design process, and the authors reported the results of evaluations of m-health services, and only a few
also evaluated the design's results via the Post-Study System Usability knew how to effectively evaluate solutions [22]. Hence, the imple-
Questionnaire (PSSUQ) to measure the usefulness of the design. mentation of successful m-health applications is difficult because many
In the current study, we used the DSR methodology with a UML to obstacles are encountered late in the process, and there is little reference
systematically design an m-health application for a referral system in material for guidance.
Indonesia that specifically focused on the medical staff module. To do so,
we conducted two iterations. In the first iteration, we used a high-fidelity 2.3. Health referrals in Indonesia
mockup to test the first design draft. The evaluation of the first iteration
was conducted using the System Usability Scale (SUS) model to measure The referral system in Indonesia has been regulated by the Minister of
the level of usability and learnability. The SUS is one of the most widely Health's Regulation Number 001 since 2012. Generally, there are three
used and well-studied measurement models [10]. After an evaluation of types of health referrals: patient, specimen, and medical personnel [23].
the initial design, we improved the prototype and reevaluated it in the When a patient is registered for the JKN, they will be assigned to the
second iteration via the PSSUQ model to measure the levels of system nearest FKTP. A patient's health referral begins with his/her admission to
usefulness, information quality, and interface quality. The PSSUQ was an FKTP. When that facility lacks the staff expertise, medical equipment,
also chosen because it provides additional data regarding the design, and/or services needed to treat the patient, he/she is referred to the next
such as ease of repair, assistance, and feedback on quality. Through this closest facility that is deemed capable of providing proper treatment
method, the results were expected to capture users' needs and expecta- (FKTRL). Such referrals aim to decrease the number of patients coming to
tions, produce the finest application design, and provide a good under- hospitals when their conditions can be handled by either clinics or public
standing of how to design an accurate health referral system. health centers. However, in cases of emergency, patients should go to the
nearest health facility. This referral system must be followed by all
2. Literature review people who participate in the JKN and other insurance participants who
make adjustments per the regulations of their insurance providers. In
2.1. M-Health addition to regulating how the referral system in Indonesia runs and who
is required to follow it, Regulation 001 also regulates the data and in-
M-health is defined as the use of mobile communication technology formation that are needed and the procedures that must be carried out in
and networks for health-related activities [11]. Per the Global Observa- the referral process.
tory for eHealth, an organization that is owned by the World Health
Organization (WHO), m-health is a form of public health and medical
practice that is supported by mobile devices, such as mobile phones, 2.4. The eight golden rules
surveillance devices, personal digital assistants, and other wireless de-
vices [12]. M-health applications are innovative in that they can provide The design stage of an application requires preparation of the user
health services at any time and in any location; thus, they overcome interface. To design a user-friendly interface, a designer might need to
existing restrictions, such as geographical boundaries [13]. The devel- follow existing design rules, which can be used as directives to ensure
opment of mobile-based communication is done through smart cellular ease of use of the application [24]. Of the available rules and directives
devices that currently support 3G and 4G networks for exchanging in- for creating a good interface, many need to be simplified. Therefore,
formation and data. M-health is a breakthrough in the health industry collections of rules that provide useful summaries of the design process
because it can help health workers provide optimal services via smart were created. A popular guideline is Shneiderman's eight golden rules
cellular device technology in any location. [25]:

2
A.A. Pinem et al. Heliyon 6 (2020) e03174

1. Strive for consistency: All things that are displayed on the interface

Design an m-health application system that allows self-monitoring

must be consistent in terms of both layout and terminology.

self-monitoring and self-prevention based on food consumed.

2. Enable frequent users to use shortcuts: These should be available for

e-health, mobile health, BMI, Android, mobile application

and self-prevention based on user food consumption data.

any important, recurring activity.
3. Provide informative feedback: This helps users complete actions.
4. Ensure dialogs to yield closure designs: Users should know that their
activity has been completed.

Design of m-health for Android applications

An m-health application mockup design for

5. Offer error prevention and simple error handling: The interface

American Journal of Engineering Research

should have an optimal design to help users avoid mistakes and easily
fix any that occur.
6. Provide easy reversal of actions: Make it easy to return to the previous
page.
7. Offer support for the internal locus of control: Users should feel that
they have control over the application and not vice versa

Not mentioned
Not mentioned
Rao & Krishna
8. Reduce short-term memory load: The interface must make the
application easy to use and not require the memorization of extensive

Study 3

Patient
information.

2014
2.5. Previous studies

We searched the keywords “design m-health referral,” “m-health

can autonomously carry out home-based health management.

applications for self-management to enhance telecare systems

Design an m-health application for telecare, where patients

referral,” and “design m-health” in the online databases ScienceDirect,
IEEE Xplore, Scopus, and Emerald Insight with a minimum publication

mobile health application, self-management, telecare,

The final design of the m-health application helped

year of 2014 (Table 1). We found limited number of papers on the
development of mobile application design especially in health area. Most
of the publications studied the acceptance, impact and conceptual design
of the mobile health application. Before 2018, the studies regarding m-

be carried out autonomously at home.

health design have not used a standardised methodology and the eval-
uation of the design also missing. Thus, this study will enrich the previous

Kao, Wei, Yu, Liang, Wu, & Wu

studies in the area of the design implementation of mobile health. Three Telematics and Informatics

design, science, usability

of the most relevant previous studies are summarized in Table 1.
Integrating m-health

The noted previous studies did not define their complete methodol-

the telecare process

ogy or their framework evaluation, except for the study by [9], which
only evaluated one iteration. This made it difficult to evaluate the im-
provements that were made after the first user evaluation. In this study,
Study 2

Patient
PSSUQ
we used the DSR approach in two iterations for the design and evaluation
2018

DSR
stages. We also used two different types of usability testing frameworks
to measure different objectives. The first iteration would have resulted in
high-fidelity mockups while, in the second iteration, the result would

in health facilities and connect services at health facilities to patients.

have been in the form of a prototype that could test features that could
not be tested in the first iteration and was uploaded on Google Playstore.
while, in the second iteration, the result would have been in the form of a
prototype that could test features that could not be tested in the first
iteration. Therefore, we used SUS testing for the first iteration and PSSUQ
for the second iteration. This study also used UML diagrams to assist in Both e-health and m-health applications can perform
designing the m-health applications, which was similar to the work of [7,
mobile health, patient, smartphone, socket
Design and develop e-health and m-health

8, 9].
Mobile-health application software design

International Journal of Computer Science

viewing prescription drugs, test results,

electronic-health, healthcare, intranet,

applications with important functions,

such as managing short messages and

functions to manage health services

3. Methodology
health tips, and health documents,

3.1. Design Science Research

and Information Security
Ayangbekun & Kasali

Information systems—an applied research discipline—uses several

in a health facility.
and development

staff and patient

theories from other scientific disciplines, such as economics, computer

Not mentioned
Not mentioned

science, and the social sciences. Results from the use of information
systems are still under debate, and they cannot always be applied to
Medical
Study 1

2014

problems in research and practice [26]. Therefore, DSR is considered

relevant in information systems research because it can solve two key
Table 1. Previous studies.

issues: the role of IT artefacts [27] and information systems research that
is considered less relevant to professionals [28]. In general, information
Framework Evaluation

systems research that uses DSR is better suited to problem solving.

Notably, DSR differs from Professional Design regarding the problems
Publication Year

faced and the solutions offered. Professional Design solves problems that
Methodology
Attributes

Keywords

already exist and can be solved using artefacts of best practices. By

Research
Purpose
Authors
Journal

Object
Result

contrast, DSR solves an important new problem that must be solved in a

Title

unique, effective, and efficient way [29]. In the initial stages, every new

3
A.A. Pinem et al. Heliyon 6 (2020) e03174

smartphones that had the Android operating system. The design success
rate was measured using the SUS usability testing framework in the first
iteration to discern the values of the dimensions of usability and learn-
ability regarding the application's design. In the second iteration, the
PSSUQ was used to see evaluate three dimensions: system usefulness,
information quality, and interface quality.

3.3. Research instrument

Figure 1. Research procedure.
For the evaluation and communication processes, two measures were
chosen. The first iteration used the SUS, and the second iteration used the
PSSUQ. These measures were chosen because they are well-known us-
ability testing frameworks that are widely used for testing and evaluation
[10, 31]. As previously mentioned, the PSSUQ was used in the second
iteration because it evaluated the ease of repair, assistance, and feedback
regarding system quality that were created by the SUS. It was not possible
to use the PSSUQ in the first iteration because the initial design was in the
form of a high-fidelity mockup.
The instrument used in this study was a questionnaire that was
divided into two parts. The first part was related to the respondents'
demographics, the obstacles faced, and the expectations of the current
referral process. The second part included a statement that was related to
the research, both the SUS and PSSUQ usability testing frameworks, and
criticisms and suggestions regarding the application's design. The
research statement was answered via a 5-point Likert scale for the SUS
Figure 2. Use case diagram. and a 7-point Likert scale for the PSSUQ. Readability tests were con-
ducted with five respondents on February 17, 2019 to ensure that all
artefact made by DSR for a discipline is an “experiment” that can still be questions and statements could be understood. Feedback from this
questioned. Existing knowledge can be used at the appropriate time, even testing guided the revision of the questionnaire. Generally, improve-
though it is often found that the required knowledge does not exist at the ments were made regarding word selection and highlighting keywords
time that the research is conducted. Creativity and trial and error are for each statement.
characteristics of DSR efforts [29].
The activities of DSR projects are generally divided into five stages: 3.4. Population and data collection
awareness of problems, suggestions, development, evaluation, and con-
clusions [30]. However, this approach can be modified to suit the Data collection in this study was conducted by distributing ques-
research [26]. In the model created by [26], they divided awareness of tionnaires offline to a sample of a predetermined population. The data
problems into two processes: identify the problem and motivate and collection was permitted and supported by Health Department of Tan-
define the objectives of a solution. They also united suggestions and gerang Selatan City. Prior filling out the questionnaires and conducting
development with the design and development process. In addition, they the demonstration, all participants were asked to read the agreement at
divided the evaluation process into demonstration and evaluation, and the beginning of the questionnaires. If the participants agreed, then the
they changed conclusions to communications [26]. Considering the survey was carried out.
above, DSR clearly has a considerable impact on research in the disci- The application was designed for medical staff in health facilities
pline of information systems regarding problem solving. whose tasks are related to patient health referrals. Purposive sampling,
which is a method that is done randomly on a population with charac-
3.2. Procedures teristics that are suitable for the research [32], was used. Data were
collected by attending meetings of medical staff at a hospital, which
There were five stages carried out in this study: problem identifica- allowed direct interaction between the respondents and the researchers.
tion and objective definition, design and development, demonstration,
evaluation and communication, and conclusions (Figure 1). The design 4. Results and analysis
and development, demonstration, and evaluation and communication
processes were conducted in two iterations. In the first iteration, the 4.1. Problem identification and objective definition
design was made in the form of a mockup, where users could try the flow
process and evaluate the design interface. In the second iteration, the As mentioned previously, the medical referral system in Indonesia
design was made in the form of a prototype for use by participants with was implemented in 2012 for JKN users and other insurance participants,

Figure 3. Flowchart of the GUI

4
A.A. Pinem et al. Heliyon 6 (2020) e03174

Figure 4. Example of a user interface for referring the patient (Step 1, Step 2 and Step 6).

which, when combined, exceeded 200 million people in 2018. Due to the ICD-9 CM code. Then, user can submit the referral and receive a success
great number of people involved, the system must be run effectively and message. The interface's design was conducted by following the eight
efficiently. One way to ensure this is to use IT to assist in the referral golden rules. All but two rules were implemented. The offer error pre-
process. Some health applications, such as SISRUTE and P-Care, have vention and simple error handling rule was left out because users were
been made for several platforms. These web-based applications are used not allowed to fill in the data. An example of the interface that was
by medical personnel to conduct referrals, and Mobile JKN allows par- designed to refer patients can be seen in Figure 4.
ticipants to register as members and view their history. However, there is
currently no mobile-based application that was designed for use by 4.2.1. Demonstration
medical staff to offer referrals. A mobile application for health referrals is For the first population, the evaluation of the first iteration was done
important for medical staff due to their high mobility while completing offline by attending a meeting of medical personnel on February 26, 2019
tasks. Therefore, the objective of this study was to design an m-health at a district hospital. The number of respondents in the first iteration was
application for the medical referral system in Indonesia that met users' 48 (n ¼ 48) (Table 2). For the first demonstration, the respondents were
expectations. asked to complete one use case which was to refer the patient following
the flowchart in Figure 3. The mockup of the mobile application was built
4.2. First Iteration with marvelapp.com and available on bit.ly/rujukanPetugas. Re-
spondents, with their own smartphones, were asked to access the mockup
Design and development. The process at this stage started with through the link.
creating a UML diagram. This study used two UMLs: the use case diagram
(UCD) and the activity diagram. Both diagrams resulted from previous 4.2.2. Evaluation and communication
research [23] with a few additions. Activity diagrams a used to describe The results of the questionnaires after the demonstration were used to
the activity or process of the current reference. The activity diagrams evaluate the design's draft. The data obtained by the questionnaire were
guided the design of this application and were used as a reference to then processed for interpretation. The calculation was made following
ensure that all the referral processes could be implemented through the the SUS evaluation method (Table 3).
application. The results of the activity diagram supported the creation of Prior to evaluating the SUS value, reliability of the instrument was
the UCD (Figure 2). The use case that will be evaluated in this study is conducted to evaluate the Cronbach's Alpha (CA) and the Composite
‘Refer the Patient’. The flow of the use case is shown in Figure 3. The first Reliability (CR). The values of the CA and the CR were 0.736 and 0.68,
step, user has to determine the referral type which are emergency or respectively. The SUS values were measured via a method developed by
policlinic. The next steps (step 2 until step 5) steps are fill out the data [33]. The measurement was done with words called “Adjective Ratings”
including patient profile, referral schedule, referral data, ICD-10 and instead of numbers. We chose this measurement so that the value that
was produced by the SUS would be easier to understand. Based on the
measurement, the application module designs were in the “OK” range,
Table 2. Respondents' profile for the medical staff module in the first iteration. meaning that the results of the design were good, but many changes
could be made to make it more fun to use and easier to learn. To deter-
Variables Total
mine what improvements should be made, we utilized an open-question
Gender Female 41 (85%)
questionnaire, which was designed to elicit criticism and suggestions.
Male 7 (15%)
The suggestions were grouped to distinguish between improvements to
Age 24–30 years old 19 (40%)
31–40 years old 23 (48%)
>40 years old 6 (12%)
Table 3. SUS value.
Occupation Nurse 24 (50%)
Doctor 23 (48%) Design for module SUS value (average) Note
Other Medical Staff 1 (2%) Medical staff 62.7 OK

5
A.A. Pinem et al. Heliyon 6 (2020) e03174

Table 4. Suggestions for the medical staff module in the first iteration.

No. Usability Number of Occurences Functionality Number of Occurences

1 Use of Indonesian in the ICD-9 and ICD-10 field 4 Provide reasons why a referral is rejected 2
by health facilities
2 Display the number of empty rooms 5 Looking for another referral to a health 1
facility if the first one is rejected
3 Indicate that the data are already 2
registered in the system
4 Changes to the sequence of the 3
field for emergency referrals
5 Display the rooms' amenities when 2
making emergency referrals

usability and functionality, which are shown in Table 4. In addition, the The communication process that was carried out in the second iter-
grouping determined the priority of the design improvements, which ation was not significantly different from that of the first. Communica-
were the usability groups items. tions phase was also carried out during demonstrations and the
implementation of the design evaluations. In addition, the final design of
4.3. Second iteration this study was published in the Playstore so that it can be tried by users.

4.3.1. Design and development 5. Discussion

The design of this iteration focused on improving the initial design
based on the results of the evaluations from the first iteration. Those This study aimed to design an m-health application for a health
results found that some information was not in accordance with the referral system in Indonesia that could meet user expectations for us-
current conditions. The design of the second iteration was technically ability. The application was designed to cover the scope of the system
different from the first iteration because the design was built into a and for use by medical staff. The research process was carried via two
prototype, which is available in the Google Playstore. However, the iterations with different measurements. Respondent in the first and
design was still implemented by using the eight golden rules to assist in second iteration are different. This is because the demonstration was
directing the design process. Table 5 shows the improvements that were carried out on hospital staffs who were actively working in the different
made in the second iteration. Examples of the design improvements are health facility, making it difficult to meet the same respondents in both
shown in Figure 5. Notably, all fields on the referral form were updated to iterations. However, in the both iterations there was no significant
the local language. difference between the distribution of gender and age. But, the occu-
pation of respondents is quite different. While in first iteration the
4.3.2. Demonstration, evaluation, and communication percentage of doctors and nurses is 50% and 48% respectively, the
The second iteration's evaluation was carried out offline at a meeting second iteration has doctors and nurses each by 35% and the rest are
with medical staff members. The number of respondents in this evalua- other hospital staff. However, this might be seen as opportunity to
tion was 31 (n ¼ 31) (Table 6). In the second iteration, the respondents evaluate the application by other staffs at hospital, because eventually,
were asked to download the application via Google Playstore in their not only doctor and nurse, other staff such as admins would also use the
own smartphone, and complete the use case to refer a patient with the mobile application.
task shown in Figure 3. The first evaluation used the SUS to measure the level of usability and
The second iteration was evaluated using the PSSUQ usability testing learnability of the design, while the second iteration used the PSSUQ to
framework, which was chosen for its ability to measure the task related to measure system usefulness, information quality, and interface quality.
error handling, which was implemented in this iteration. The values of The purpose of the first iteration evaluation was to ensure that the initial
the CA and the CR for each variable are shown in Table 7. The maximum design was in accordance with the objectives of usability and learn-
value of each dimension was 7.0. The average value for all the di- ability, which help users to complete tasks effectively and efficiently and
mensions was 4.6, meaning that, overall, the respondents were satisfied satisfy their perceived utilization level [34]. Thus, the first iteration
with the results of the design. The values of system usefulness, infor- focused on the overall design, which was expected to help users carry out
mation quality, and interface quality can be seen in Table 8. the intended functionalities.

Table 5. Second iteration improvements.

No Improvement Module Related rule Related use case Action

1 Uses Indonesian in Medical staff Reduce short-term memory load Refer the patient Changing the language
the ICD-9
and ICD-10 field
2 Displays the number of Medical staff Reduce short-term memory load Refer the patient Update the interface to view the empty rooms'
empty rooms information
3 Indicates that the data Medical staff Offer error prevention Refer the patient Adding information when the user has already
are already and simple error handling completed the form
registered in the system
4 Changes to the Medical staff Reduce short-term memory load Refer the patient Updating the order and position of the field on
sequence of the the patient referral form
field for emergency
referrals
5 Displays the rooms' Medical staff Reduce short-term memory load Refer the patient Changing the placement and appearance
amenities when of the rooms' amenities
making emergency referrals

6
A.A. Pinem et al. Heliyon 6 (2020) e03174

Figure 5. Design improvements to the referral form in the second iteration. (Step 3, Step 4, Step 5).

Table 8. PSSUQ values.

Table 6. Respondents' profile for the medical staff module in the second
iteration. Design System Information Interface Average Note
Module Usefulness Quality Quality Score
Variables Total
Medical 4.7 4.5 4.8 4.6 Above the median (4) is
Gender Female 27 (87%)
Staff considered good
Male 4 (13%)
Age 17–23 years old 2 (6%)
displaying rooms information, success message) and interface quality
24–30 years old 12 (39%) aspect (the sequence of information) that would be evaluated in second
31–40 years old 14 (45%) iteration using PSSUQ.
>40 years old 3 (10%) The second iteration was developed using the Android platform,
Occupation Nurse 11 (35%) which did allow the user to enter data. The application was designed to
Doctor 11 (35%) prompt an error message and to indicate which field will require
Other Medical Staff 9 (30%) correction. For easy reversal of actions, the design includes a return
button for each view. To implement the support of internal locus of
control, a reference history was made available to the user. In the second
The implementation of usability and learnability in the first iteration
iteration, the search field was updated to increase user awareness
was accomplished by using the eight golden rules during the design
regarding their power to navigate the application. Finally, for the reduce
process. For example, the strive for consistency rule was implemented by
short-term memory load rule, the first evaluation found that the design
designing the main menu at the bottom of the display but having it
was lacking in this area; a field had been implemented in a foreign lan-
disappear if the user enters something else. Another rule was enabling
guage (ICD-9 and ICD-10 field). This was therefore corrected in the
frequent users to use shortcuts, which was implemented by allowing
second iteration.
users to easily access other features on the home menu display. To
Based on the results of the first evaluation, reducing short-term
implement the informative feedback rule, the user will be sent infor-
memory load is quite important, especially for busy medical personnel.
mation on actions that he/she has carried out. For the dialogs to yield
Improvements were made based on the results of the first iteration's
closure rule, the display will tell the user whether a task has been suc-
evaluation, including changes in the use of language in the ICD-9/ICD-10
cessfully completed. For example, if a user has completed the referral
field, which also improved information quality. Per [35], information
process, then the system will convey that the information has been
quality is defined as when information that is provided by a system can
created and saved. The offer error prevention and simple error handling
be easily understood and can effectively assist in the completion of tasks.
rule was only implemented in the second iteration because the user could
The information quality dimension underwent many improvements
not enter data in the first iteration's design. The result form the first
based on the results of the first iteration evaluation, including changing
iteration using SUS shows that overall value for the usability were OK
the order of the data that refer to emergencies, adding information on
with some change suggestion to be made. There are five improvements
actions that have been given, changing the language in the ICD-9/ICD-10
regarding the usability of the system. The improvement can be catego-
field, and adding information on patient data automatically if it has been
rized as information quality aspect (implementation of local language,
registered in the system previously.
The interface quality dimension is concerned with whether a system
Table 7. Cronbach's alpha and composite reliability. is capable of satisfying its users [35]. From the first iteration, this
dimension was also improved. For example, the display was changed to
Variables Cronbach's Alpha Composite Reliability include certain information, such as amenities in rooms, amenities in
System Usefulness 0.937 0.951 health facilities, and the number of empty rooms, so that it can be more
Information Quality 0.925 0.942 easily seen by users. The three dimensions of the PSSUQ were important
Interface Quality 0.844 0.898 for achieving usability that met user expectations.

7
A.A. Pinem et al. Heliyon 6 (2020) e03174

Based on the results of our calculations, the application's designs F. Azzahro: Performed the experiments; Analyzed and interpreted the
obtained SUS values at the level of “OK” [33], meaning that the design data.
results were acceptable, but improvements could be made. The PSSUQ A. N. Hidayanto, D. Ayuningtyas: Conceived and designed the ex-
value in the second iteration evaluation mimicked the first iteration, periments; Analyzed and interpreted the data.
where users found the application's design satisfactory, with the lowest
value found in the information quality dimension. Funding statement

6. Implications This work was supported by the Directorate General of Higher Edu-
cation for the Program Dasar Penelitian Unggulan Perguruan Tinggi
The theoretical implications of this research are the contributions to (PDUPT) 2019, grant No. NKB-1495/UN2.R3.1/HKP05.00/2019, as well
the designing of an m-health application, especially when considering as by the Directorate of Research and Community Engagement at the
that there is little research in this area. This study also provides more Universitas Indonesia.
insight into the use of the DSR methodology. In the previous research of
[9], the procedure was only carried out in one iteration; therefore, their Competing interest statement
evaluation results focused only on the value of the usability testing
framework (PSSUQ). In this research, we used the SUS to evaluate the The authors declare no conflict of interest.
first iteration design and the PSSUQ to evaluate the second iteration
design. Additional information
From a practical standpoint, this research is expected to give practical
implications to relevant policy-making institutions, such as the Ministry No additional information is available for this paper.
of Health and the Health Insurance Institution, regarding information
quality, interface quality, and system quality when implementing m- References
health applications. It will also support the referral process in Indonesia,
given the high level of mobility of medical staff. In addition, this appli- [1] APJII, Penterasi & Perilaku Pengguna Internet Indonesia, Teknopreneur, Jakarta,
cation is expected to be a benchmark in the process of designing and 2017.
[2] F.V. Haucke, Smartphone-enabled social change Evidence from the Fairphone case,
further development of m-health applications for the referral system in
J. Clean. Prod. (2018) 1719–1730.
Indonesia. [3] M. Anshari, Y. Alas, Smartphones habits, necessities, and big data challenges,
J. High Technol. Manag. Res. (2015) 177–185.
[4] N. Bento, Calling for change? Innovation, diffusion, and the energy impacts of
7. Conclusion
global mobile telephony, Energy Res. Soc. Sci. (2016) 84–100.
[5] P.W. Handayani, D.A. Meigasari, A.A. Pinem, A.N. Hidayanto, D. Ayuningtyas,
This research was conducted to design an m-health application for a Critical success Factors for mobile Health Implementation in Indonesia, Heliyon,
health referral system in Indonesia. This study used the DSR methodol- 2018.
[6] Agustiyanti, Peserta BPJS Kesehatan Tembus 200 Juta Jiwa, 16 August 2018
ogy in the design process, which was carried out in two iterations. The [Online]. Available: https://www.cnnindonesia.com/ekonomi/20180816113007-
evaluation process used the SUS usability testing framework for the first 78-322721/peserta-bpjs-kesehatan-tembus-200-juta-jiwa.
iteration and the PSSUQ for the second iteration. In the first iteration, [7] O.J. Ayangbekun, O.M. Kasali, Mobile - health application software design and
development, Int. J. Comput. Sci. Inf. Secur. 12 (7) (2014).
there were 48 respondents. The results from the SUS value in the first [8] D.V.S. Rao, D.T.M. Krishna, A design of mobile health for android applications, Am.
iteration placed the design in the category of “OK,” meaning that the J. Eng. Res. (2014).
results of the design were good, but many improvements could be made. [9] H.-Y. Kao, C.-W. Wei, M.-C. Yu, T.-Y. Liang, W.-H. Wu, Y.J. Wu, Integrating a mobile
health applications for self-management to enhance Telecare system, Telematics
Therefore, in the second iteration, improvements were made to meet the Inf. (2018).
suggestions from the first iteration. There were 31 respondents in the [10] N. Harrati, I. Bouchrika, A. Tari, A. Ladjailia, Exploring user satisfaction for e-
second iteration. Based on the results of the evaluation, the final design learning systems via usage-based metrics and system usability scale analysis,
Comput. Hum. Behav. 61 (2016) 463–471.
produced system usefulness, information quality, and interface quality
[11] R. Istepanian, S. Laxminarayan, C.S. Pattichis, M-health: Emerging Mobile Health
that satisfied the respondents. Systems, 2006.
Medical staff have the expectation of usability of the system's design, [12] S. Ryu, mHealth: new horizons for health through mobile technologies: based on
the findings of the second global survey on eHealth, in: Global Observatory for
which requires local language for the needed forms (ICD-9 and ICD-10)
eHealth Series, 2012.
to improve their effectiveness. In addition to system usefulness, infor- [13] B.M. Silva, J.J. Rodrigues, I.d. l.T. Díez, M. L
opez-Coronado, K. Saleem, Mobile-
mation quality and interface quality must be implemented in the appli- health: a review of current state in 2015, J. Biomed. Inform. (2015) 265–272.
cation. The respondents required improvements and additions in the data [14] J.J.P. Rodrigues, S.S. Compte, I.d. l.T. Diez, e-Health Systems, Theory, Advances
and Technical Applications, Elsevier Inc., 2016.
sequence when making emergency referrals and improvements in the [15] W. Boonchieng, E. Boonchieng, W. Tuanrat, C. Khuntichot, K. Duangchaemkarn,
appearance of the nearest health facilities. These three dimensions must Integrative system of virtual electronic health record with online community-based
have a minimum value above the middle Likert value (4) to obtain good health determinant data for home care service: MHealth development and usability
test, in: IEEE Healthcare Innovations and Point of Care Technologies, 2017.
results, which would place the design in accordance with the needs of its [16] S. Akter, P. Ray, mHealth - an ultimate platform to serve the unserved, Yearb. Med.
users. In addition, the values of usability and learnability that were ob- Inf. (2010) 94–100.
tained from the SUS must be at least 51 to place the design in accordance [17] S. Lim, S.M. Kang, H. Shin, H.J. Lee, J.W. Yoon, S.H. Yu, S.-Y. Kim, S.Y. Yoo,
H.S. Jung, K.S. Park, J.O. Ryu, H.C. Jang, Improved glycemic control without
with the expectations of its users. hypoglycemia in elderly diabetic patients using the ubiquitous healthcare service, a
new medical information system, Diabetes Care (2011) 308–313.
Declarations [18] W.-T. Liu, C.-H. Wang, H.-C. Lin, S.-M. Lin, K.-Y. Lee, Y.-L. Lo, S.-H. Hung, Y.-
M. Chang, K. Chung, H.-P. Kuo, Efficacy of a cell phone-based exercise programme
for COPD, Eur. Respir. J. (2008) 651–659.
Author contribution statement [19] V. Faucounau, M. Riguet, G. Orvoen, A. Lacombe, V. Rialle, J. Extra, A.-S. Rigaud,
Electronic tracking system and wandering in Alzheimer’s disease: a case study, Ann.
Phys. Rehabil. Med. (2009) 579–587.
A. A. Pinem: Conceived and designed the experiments; Performed the
[20] N. Bellamy, C. Wilson, J. Hendrikz, S.L. Whitehouse, B. Patel, S. Dennison, T. Davis,
experiments; Analyzed and interpreted the data; Wrote the paper. Osteoarthritis Index delivered by mobile phone (m-WOMAC) is valid, reliable, and
A. Yeskafauzan: Performed the experiments; Analyzed and inter- responsive, J. Clin. Epidemiol. (2011) 182–190.
preted the data; Contributed reagents, materials, analysis tools or data. [21] V.P. Gurupur, T.T.H. Wan, Challenges in Implementing mHealth Interventions: a
Technical Perspective, Mhealth, 2017.
P. W. Handayani: Conceived and designed the experiments; Per- [22] World Health Organization, mHealth, New Horizons for Health through Mobile
formed the experiments; Analyzed and interpreted the data. Technologies, 2011.

8
A.A. Pinem et al. Heliyon 6 (2020) e03174

[23] P.W. Handayani, A.A. Pinem, Q. Munajat, F. Azzahro, A.N. Hidayanto, [30] B. Kuechler, V. Vaishnavi, On theory development in design science research:
D. Ayuningtyas, A. Sartono, Health referral enterprise architecture design in anatomy of a research project, Eur. J. Inf. Syst. (2008) 489–504.
Indonesia, Healthc. Inf. Res. 25 (1) (2019) 3–11. [31] T.-l. Lai, P.-y. Chen, C.-y. Chou, A user experience study of a web-based formative
[24] F.K. Mazumder, U.K. Das, Usability guidelines for usable user interface, IJRET: Int. assessment system, in: International Conference on Applied System Innovation,
J. Renew. Energy Technol. (2014) 79–82. Sapporo, 2017.
[25] P.A. Chalmers, The role of cognitive theory in human–computer interface, Comput. [32] S.J. Fetzer, Considering the sample, J. PeriAnesthesia Nurs. (2017) 379–381.
Hum. Behav. 19 (2003) 593–607. [33] A. Bangor, P.T. Kortum, J.T. Miller, An empirical evaluation of the system usability,
[26] K. Peffers, T. Tuunanen, M.A. Rothenberger, S. Chatterjee, A design science Intl. J. Human Comput. Interact. (2008) 574–594.
research methodology for information systems research, J. Manag. Inf. Syst. (2007) [34] I. Maramba, A. Chatterjee, C. Newman, Methods of usability testing in the
45–77. development of eHealth applications: a scoping review, Int. J. Med. Inform. (2019)
[27] I. Benbasat, R. Zmud, The identity crisis within the IS discipline: defining and 95–104.
communicating the discipline’s core properties, MIS Q. 27 (2003) 183–194. [35] N. Pereira-Azevedo, L. Os orio, A. Fraga, M.J. Roobol, Rotterdam prostate cancer
[28] R. Hirschheim, H. Klein, Crisis in the IS field? A critical reflection on the state of the risk calculator: development and usability testing of the mobile phone app, J. Med.
dicipline, J AIS 4 (2003) 237–293. Internet Res. Canc. (2017).
[29] A. Hevner, S. Chatterjee, Design science research in information systems, in: Design
Research in Information System, 2010, pp. 9–22.