(IJACSA) International Journal of Advanced Computer Science and Applications,

Vol. 14, No. 1, 2023

e-Government Usability Evaluation: A Comparison

between Algeria and the UK
Mohamed Benaida
Faculty of Computer and Information Systems
Islamic University of Madinah, Saudi Arabia

Abstract—e-Government holds the keys to improving Algerian government seeks to establish an electronic system
government services provided to citizens and the private sectors through which citizens can benefit from the e-services
within their countries. Although Algeria is the largest country in provided by their government in various fields. However, the
Africa and has one of the most thriving economies in the e-government project encountered many problems regarding
continent, it is remarkable that the Algerian EGDI ranking was infrastructure, use of technology in a sound manner, the quality
120th according to the latest UN e-government survey. This of services, and ease of access for citizens. The ultimate aim of
inspired the researcher to investigate the relationship between e-governments is to improve the usability of their services for
the success factors of e-services in developed countries and their their users, thus increasing user interaction. There are several
counterparts in developing countries. The main aim of this study
definitions of e-government based on its use, however this
is to explore the factors that influence the level of usability of e-
government services between developing and developed countries
research focuses on e-government as a system rather than from
against a set of specific guidelines to provide means for a technical or social aspect. One definition describes e-
improving these services in developing countries. The researcher government as "the use of information technology to enable
selectively extracted three guideline categories from Research- and improve the efficiency of government-provided services to
Based Web Design and Usability Guidelines as a means for citizens, employees, businesses and agencies" [2]. Most
expert evaluation of 10 Algerian e-government services compared governments in developing countries have recently engaged in
to British e-government services. Our results show that Algerian e-services that aim to conserve effort, time, and money for the
e-services lack mostly in Use Frames when Functions Must population. Many various usability guidelines exist today (such
Remain Accessible, Highlighting Information, and Graphics as ISO/CD 9241-151). Thus, it is hard to point out one
Should Not Look like Banner Ads (belonging to Page Layout, guideline as the model since every audience around the world
Text Appearance, and Graphics, Images & Multimedia has its own culture, background, symbols, language, and
respectively), whereas UK e-services scored highly across all political system [3]. Furthermore, the level of the country's
three categories. These findings further enhance the UN e- economy affects their perception and judgments of everything
government survey and identifies the sub-categories that around them, and therefore each audience has different needs
developing countries need to pay more attention to in order to [4], [5].
provide a more reliable and robust e-service to its users and
citizens. Furthermore, this study proposes that the Research- This study decided to utilise the Research-Based Web
Based Web Design & Usability Guidelines can be converted into Design & Usability Guidelines (RBED&UG) due to its
an evaluation tool to be used by evaluators to easily assess the exclusivity and popularity in government agencies, educational
usability of a website. The combination of relative importance, programs, and various independent sectors. These guidelines
chapters of the guidelines, and their respective guidelines were established by the U.S. Department of Health and Human
gathered from Research-Based Web Design & Usability Services initially in 2003 and further developed in 2006 with
Guidelines, along with the evaluation of these individual the aim of helping designers develop a large range of websites,
guidelines by evaluators will serve as an integral tool for but mainly information filled government services. The
developers when developing e-government services to reach the exclusivity of these guidelines lies in their relative importance
satisfaction of the users.
in rankings whilst also providing references and expert
Keywords— Human computer interaction; usability evaluation;
feedback to support their argument. Another reason this study
web design; e-Government; user satisfaction decided to use guidelines is due to the recommendations
provided by the authors of these guidelines, which motivates
I. INTRODUCTION researchers to explore avenues with little or almost no research,
as is the case is with Algerian e-government. The latest edition
The internet officially launched in Algeria in 2001, Seven
is made up of 18 main chapters, which are divided into
years later in 2008, e-government services in Algeria were
multiple guidelines enhanced by the relative importance and
established [1]. Only recently has the Algerian government
strength of evidence providing a total of 209 guidelines across
tried to keep up with modern technology by using online
these chapters [6]. This research focused on three chapters of
services. However, the shift from centralisation to using
this guideline (Page Layout, Text Appearance, and Graphics,
electronic services has not been an easy process. Design,
Multimedia, & Images) as it aimed to provide accurate and in-
logistics infrastructure, internet speed, and adoption of new
depth results rather than a broad and general set of results
services; are all factors that are considered great challenges
which may not be as useful to the web designers of these
facing the government as it moves into the world of electronic
websites.
services. Like other governments in developing countries, the

680 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

The main reasons for picking Algerian e-government chapters and to show that it can indeed be implemented
services as the subject of this study is that there is a lack of efficiently and effectively to good use.
research in this field regarding less developed countries
generally [7], and according to our knowledge, even after more This research paper is split into six parts including the
than a decade since its establishment, there are no studies that introduction which provides an overview of the main
analyse the usability of e-government services in Algeria. components of this research. Then, the literature review dwells
Furthermore, this simultaneously fulfils the recommendations on previous research and identifies their key findings and
of the authors of the RBED&UG of researching into new areas shortcomings which need to be addressed. The methodology
and providing relevant research results. Therefore, the main details the stages of this research and the demographic data of
contribution of this research is that it is the first study to shed the participants that were involved. Results and discussion
light on the usability of Algerian e-government services and include a detailed analysis which breaks down the various
provide web designers with a deep insight into how to address attributes that are suffering or excelling in the usability of e-
the flaws (lack of online services, lack of visual consistency, government services. The conclusion provides an overview of
etc.) which the experts in this study have identified in order to our research along with the main results and contributions it
help the government create usable, learnable, and effective e- brings forward to the domain of e-government usability.
services that will increase user interaction. Finally, the limitations and future work marks the end of this
research paper.
The e-government development index (EGDI) measures a
country's capability to deliver online public services and is II. LITERATURE REVIEW
calculated using three variables including telecommunications Information technology is growing rapidly; government
infrastructure index (TII), human capital index (HCI), and services must also be up to date with these technological
online service index (OSI). This study compares the Algerian advancements. In developing countries, this challenge is far
government e-services with the government e-services in the greater, since many still suffer from poor IT infrastructure and
United Kingdom, as Britain is a developed country and ranks financial problems, which hinder their progress in providing
high in the EGDI rankings published in the latest United advanced electronic services to their citizens. Besides, building
Nations e-government survey so that a comparison can be trust between the citizens and using electronic services is still
made between Algeria as a developing country and Britain as a in development [9]. This section discusses e-government
developed country, allowing for better understanding of websites and usability, usability evaluation methods and e-
deficiencies and focusing on the factors that are likely to have governments Algerian websites.
an impact on improving the quality of service provided to
citizens in developing countries, which facilitates the lives of A. e-Government Websites and Usability
citizens and makes them more efficient and of better quality. In the last decade, e-government services have become
The British are 7th in the EGDI rankings whereas Algeria is more important than ever before. Many governments have
ranked 120th in the world [8]. The aim is to explore why there launched at least one or more systems that provide citizens
are such big gaps in the e-services provided between developed with the necessary information and adequate services. It
and less developed countries. Unlike the British e-government summarises and stores relevant information based on the
services which are all available on one website, Algerian e- quality of the internet and the level of ability of the users to use
government services are all independent and each website their internet services. Many factors can affect the performance
serves a different purpose, for example, the ministry of interior, of these e-government systems such as the educational
the ministry of sports etc. Therefore, this study selected ten background of the users, culture, and simplicity of the system,
Algerian e-government websites to compare with the British e- efficiency, robustness and reliability [3]. There are various
government website to serve as a realistic representation of definitions of e-government depending on the field of interest
Algerian e-services. The null hypothesis in this study is that and the perception of experts [10]. The World Bank (2015)
there is no significant difference between Algerian and British believes that “E-Government refers to the use by government
e-government services, whereas the alternative hypothesis agencies of information technologies (such as Wide Area
states that there is a significant difference between Algerian Networks, the Internet, and mobile computing) that have the
and British e-government services. ability to transform relations with citizens, businesses, and
other arms of government” [11]. However, a comprehensive
Another contribution of this study is that it converts the
study on multiple definitions of e-government established six
RBED&UG into a questionnaire to be used by web designers
main elements that serve as a widely shared definition. These
to help evaluate and gather relevant data for improving
variables consist of; “(1) the major initiatives of management
electronic web services. Researchers realised the importance of
and delivery of information and public services. (2) Taken by
these guidelines in developing and initiating web systems in
all levels of governments (including agencies and sectors). (3)
various fields and the high potential that it can be used by web
On behalf of citizens, business. (4) Involving using multi-ways
designers for evaluating the level of a system and addressing
of internet, web site, system integration, and interoperability.
certain issues that may present. Researchers believe that the
(5) To enhance the services (information, communication,
same concept of converting guidelines into a questionnaire to
policy making), quality and security. (6) As a new key (main,
be used as an evaluating tool for web designers can be
important) strategy or approach” [10].
implemented with any of the chapters of the RBED&UG. The
study only used three chapters as a representative of all Usability is defined by the ISO as the “extent to which a
system, product or service can be used by specified users to

681 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

achieve specified goals with effectiveness, efficiency and quality provided to their citizens. e-Government services in
satisfaction in a specified context of use” [12]. Any new Mauritius Island showed that factors like expected performance
system or product should achieve its goals and satisfy its end and facilitating conditions have a positive relationship with the
users by ensuring that the system is more interactive and users' behavioural intentions. On the other hand, computer self-
attractive to maximise the benefits of the system. Most e- efficacy is negatively related to the users' behavioural
government projects (35% total failures and 50% partial intentions. The authors suggest essential steps the government
failures) in less developed countries are unsuccessful [7]. They could take to help the citizens adopt their e-government
are unable to fulfil the satisfaction of their users due to the services effectively and efficiently [32]. In Turkey, factors that
widespread usability issues found on their systems [13]. could help citizens adopt e-government services are
Nielsen’s description of usability via the implementation of Performance Expectancy, Social Influence, Facilitating
various metrics is widely accepted, thus ten usability guidelines Conditions, and Trust on the Internet. Authors [33] have
were initiated to identify errors, evaluate the level of initiated a new guideline for policymakers to initiate attractive
effectiveness and efficiency and establish user control, thus, e-government services by considering the highlighted priorities
ultimately fulfilling user satisfaction.[3]. of the citizens. The authors proposed a new analysis technique
that could potentially increase and maximise e-government
Some studies [14] argue that e-government can benefit portals' potential to benefit individuals from various cultural
from usability in two aspects; satisfaction is key due to backgrounds. This analysis technique incorporates various
improved user performance on e-government services, on the factors such as content analysis, user perception, and
other hand, the first impressions are also crucial for attracting persuasive quality gap. Culture is another factor that can affect
users to using e-government online services. e-Government the usability satisfaction level [34], [16]. For example, [35]
relates to usability in multiple ways correlating to satisfaction; argue that their suggested approach provides an estimation
internet speeds [15] loading speed [16] usefulness and ease of method to measure the contribution of e-government quality
use [17]. The abovementioned variables are one of the most attribute to cross-cultural quality gap. They also examine the
influential in the usability of e-government according to satisfaction level of Jordanian citizens when using these e-
previous studies. Poor design interface can affect not only the government services. Several factors influencing satisfaction
perception of users, but in some cases can become more were identified; security and privacy, trust, accessibility,
serious and influence various factors such as user safety [18]. awareness, and quality. They claim that their findings have
Therefore, users should be an essential part in the design stages proven very useful for industry practitioners and policymakers
to minimise the errors that end users can face [7] explore the within the government.
value of e-government and the public value it holds. Through
this research, six values were identified; Open Government B. Usability Evaluation Methods
capabilities, improved ethical behaviour and professionalism, Usability evaluation becomes more crucial in testing
improved public services, improved administrative efficiency, usability issues including content and page layout, to create a
improved social value and well-being, and improved trust and system that suits the requirements of the user and meets their
confidence in government. In least developed countries and needs. There are two main usability evaluation methods used to
developing countries, there was a lack of research in attempting assess usability websites: heuristic evaluation and usability
to figure out the public value of e-government and the user testing.
perception of these services.
Many studies use this type of evaluation (see Table Ⅰ) via
Many studies in other countries have attempted to figure Nielsen’s ten heuristics due to its low cost, time efficiency, and
out solutions and identify factors that can influence the quality how it requires only a few experts to give their opinion about
and usability of e-governments services. Studies have shown the problems that face certain applications or websites. On the
that usability is the heart of an e-government system that other hand, “usability testing refers to evaluating a product or
attempts to fulfil the satisfaction of its users. Furthermore, it service by testing it with representative users” [36]. Usability
can directly affect the trust of the users [19]. The meaning of testing gathers a group of users to complete several tasks on the
usability may not be clear enough. However, usability is not system to assess the usability level and identify any issues
just how to make the system easy to use; instead, it is the ease within it. The users assessing the system are the same users that
of use based on efficiency, effectiveness, learnability, and user the designer is creating a usable system for, thus any issues
satisfaction [20]. Some studies argue that western e- identified by these participants are crucial and provide a real-
government websites fail to deliver services to their citizens life representation of the end user feedback after the official
[21]. There were several usability issues identified in launch of the system. Consequently, designers can take their
developing countries such as Jordan [22], Malaysia [23], China feedback on board and make the appropriate adjustments to
[24], and Sub-Saharan Africa [25] such as South Africa [26] improve the overall system to make it more usable and
and Tanzania [27]. The usability problems identified included attractive. For these reasons, this study proposes that the new
accessibility [28] poor design [29] learnability [30] and text questionnaire is to be used as a heuristic evaluation method
appearance [31]. Furthermore, poor usability design affects the used by evaluators in web design.
quality of the website and the adoption of users, which drives
them away from revisiting the website [5].
Each country faces its own specific usability issues, and
many studies have found different factors that can positively or
negatively evaluate the level of e-government services and the

682 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

TABLE I. PREVIOUS STUDIES OF USABILITY EVALUATION METHODS electronic services, and accordingly, a comparison was made
Study Method Type between these ten sites. After this information was evaluated
by experts, the average was collected from each of the ten sites
Fonseca and Peñalvo
Usability testing Gaming and compared with the standard British site in order to
(2019) [37] facilitate the evaluation process.
Huang and Chen (2019) Digital learning
Usability testing B. Proposed Usability Questionnaire Model
[38] environments
After realising the effectiveness of the RBED&UG,
Fuller-Tyszkiewicz et al researchers believed that these guidelines could be effectively
Usability testing Health app
(2018) [39] used during usability of e-government services to identify
Krzewińska et al (2018) flaws within the usability of these systems. Whilst guidelines
Usability testing Web systems are used by web designers when creating a web page,
[40]
researchers propose that the RBED&UG is transformed and
De Souza Filho et al Collaborative used as a questionnaire to evaluate the system after it is created
Gaming
(2017) [41] heuristic evaluation so that web designers can brush up and improve on the areas
that are lacking thus providing a more robust and reliable
Diaz et al (2017) [42] Heuristic Evaluation e-Commerce
service. This study decided to selectively examine the
Jucá et al (2017) [43] Heuristic Evaluation Gaming effectiveness of three chapters (Page Layout, Text Appearance,
Graphics, Images & Multimedia) from the RBED&UG to
Alhadreti and Mayhew Examine think-aloud
Usability testing serve as a strong representative for the other 15 chapters. Since
(2017) [44] methods the original guidelines follow a five-point scale, researchers
Usability testing and believed it would be most appropriate to be consistent and
Murillo et al (2017) [45] Web-based systems implement a five-point scale in the questionnaire where one
Heuristic Evaluation
represents "very poor" to five which represents "very strong".
Atashi et al (2016) [46] Heuristic Evaluation User Interface Tables Ⅱ, Ⅲ and Ⅳ display the questionnaires that were
Falkowska et al (2016) Usability testing (Eye provided to the experts before the start of the experiment. They
Web applications were provided with two copies of each one, totalling to six
[47] Tracking)
copies of these questionnaires to evaluate both Algerian and
Smartphones and British e-Government services relative to the three selected
Inostroza et al (2015) [48] Heuristic Evaluation
application guidelines. The experts marked their selected scores for the
Chynał and Sobecki relative e-service and guidelines of these categories. These
Usability testing User Interface results will be collated by the researcher at the end of their
(2015) [49]
evaluation and utilised for statistical analysis of the data.
Bezerra et al (2014) [50] Usability testing Ubiquitous systems
TABLE II. TABLE TYPE STYLES
Ko et al (2013) [51] Heuristic Evaluation Mobile app
Usability Page Layout Questionnaire
Boothe et al (2013) [52] Usability testing Interface medium
1 2 3 4 5
e-Government
Sivaji et al (2011) [53] Heuristic Evaluation 1. Avoid Cluttered Displays
websites
Usability testing and 2. Place Important Items Consistently
Thyvalikakath (2009) [54] Comparison study
Heuristic Evaluation 3. Place Important Items at Top Centre
Garcia et al (2005) [55] Heuristic Evaluation e-Government 4. Structure for Easy Comparison

5. Establish Level of Importance

III. METHODOLOGY
6. Optimize Display Density
A. e-Services Evaluated
7. Align Items on a Page
Ten Algerian e-Government websites were selected for
evaluation in this study. They include the following: Minister 8. Use Fluid Layouts
of Defence, Minister of Interior and Local Government, 9. Avoid Scroll Stoppers
Minister of Foreign Affairs, Minister of Finance, Minister of
Energy, Minister of War Veterans (Moudjahidine), Minister of 10. Set Appropriate Page Lengths
Religious Affairs and Endowments (Wakfs), Minister of 11. Use Moderate White Space
Vocational Education and Training Professionals, Ministry of
Culture and Arts, and Minister of National Solidarity, Family 12. Choose Appropriate Line Lengths
and Women's Affairs. There is no unified website in Algeria 13. Use Frames when Functions Must Remain
that can provide more government electronic services, so the Accessible
researcher had to collect the ten most used Algerian websites to
evaluate electronic government services, while in the United
Kingdom there is only one website that collects government

683 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

12. Introduce Animation

TABLE III. USABILITY TEXT APPEARANCE QUESTIONNAIRE
13. Emulate Real-World Objects
Usability Text Appearance Questionnaire
1 2 3 4 5 14. Use Thumbnail Images to Preview Larger
1. Use Black Text on Plain, High-Contrast Images
Backgrounds
15. Use Images to Facilitate Learning
2. Format Common Items Consistently
16. Using Photographs of People
3. Use Mixed-Case for Prose Text

4. Ensure Visual Consistency

C. Procedure
The volunteering experts were provided with all the
5. Use Bold Text Sparingly websites required for this experiment and a brief was given
before the beginning of the evaluation process along with
6. Use Attention-Attracting Features when copies of the questionnaires they would have to fill out. The
Appropriate brief very simply explained the goals of our research and that
the participants are testing the guidelines as a questionnaire for
7. Use Familiar Fonts evaluation to explore the issues in Algerian and British e-
Government websites. The results would also simultaneously
8. Use at Least 12-Point Font help assess the validity of the proposed questionnaire. The
researcher decided to allocate some time for the experts before
9. Color-Coding and Instructions evaluation to get familiar with these e-services so that they can
easily and effectively evaluate them once the process begins.
10. Emphasize Importance After that the experts evaluated the e-services according to the
three dimensions (Page Layout, Text Appearance, and
11. Highlighting Information Graphics, Images, & Multimedia) from our usability
questionnaire accordingly, freely with no time constraints.
Experts were asked to return the questionnaire once complete.
TABLE IV. USABILITY GRAPHICS, IMAGES, & MULTIMEDIA
QUESTIONNAIRE D. Participants
Usability Graphics, Images, & Multimedia Questionnaire Since some e-services are only available in English, Arabic,
1 2 3 4 5 or French, the researcher decided to select seven experts who
are confident with all three languages to help gather the results
1. Use Simple Background Images
of our study across all platforms with no issues and full
understanding. Each expert evaluated the ten selected Algerian
2. Label Clickable Images
e-Government websites and the British e-Government website.
3. Ensure that Images Do Not Slow
The participants who contributed to this research voluntarily
Downloads are from different educational levels and gender backgrounds.
Most participating experts were males aged between 36 and 50
4. Use Video, Animation, and Audio years who are of a postgraduate degree status. Table Ⅴ shows
Meaningfully the expert demographics.

5. Include Logos TABLE V. EXPERT DEMOGRAPHICS

Number of Participants
6. Graphics Should Not Look like Banner Ads Demographic Category %
(7)

7. Limit Large Images Above the Fold Age 25-35 1 14

36-50 4 57
8. Ensure Web Site Images Convey Intended
Messages 50+ 2 29

Gender Women 2 29
9. Limit the Use of Images
Men 5 71
10. Include Actual Data with Data Graphics Undergraduate
Education Level 2 29
Degree
11. Display Monitoring Information
Postgraduate
Graphically 5 71
Degree

684 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

E. SPSS Analysis 5
The results of the expert evaluation gathered via the 4
proposed questionnaire were inserted into the SPSS program
and results of importance to this study were extracted such as 3
the paired samples statistics mean and standard deviation, the 2
paired samples test mean and standard deviation, t-values, and 1
p-values. The paired samples statistics mean shows us the
0
average score for each guideline of each chapter, whereas the
paired samples statistics standard deviation uses the results of Page Layout Text Appearance Graphics, Images &
the experts to identify the level of consensus between the set of Multimedia
results. The paired samples test mean is the difference between
Algerian e-government British e-government
the paired samples statistics mean of the two tested variables.
Furthermore, the p values show us the level of significant
differences between our tested samples in each element of the Fig. 1. Chart comparing the mean results of Algerian and British e-
Government services across the three selected categories of the RBED&UG
three categories selected. For the data to be considered
significantly different, the value of p must be less than 0.05, A. Page Layout
thus rejecting the null hypothesis which in this study is that
there is no difference between the Algerian and British e- Page layout is the main factor in the user interface; it
Government services, therefore the alternative hypothesis outlines the position of elements and their dimensions [56].
would be accepted instead. However, if the value of p is greater The most important aspect when designing a page layout is to
than 0.05, then the data is deemed to be insignificantly include necessary information only and to avoid overwhelming
different, and the null hypothesis is accepted. This analytical the user with unnecessary information that will only lead to
technique helps researchers identify where the main flaws lie in confusion and over complication.
a certain set of data, and in this case, it shows us which With the help of the SPSS software, the researcher was
guideline suffers the most in the usability of Algerian and able to extract relevant information as seen in Table Ⅵ which
British e-Services and where the biggest difference lies presents all this data along with the 13 guidelines of Page
between the two. Layout that are ranked according to relative importance
according to the RBED&UG. The results show that all the
IV. RESULTS AND DISCUSSION
guidelines of the Page Layout when comparing the Algerian
Experts evaluated Algerian and British e-Government and British services show significant differences due to the
services via our proposed questionnaire derived from the three value of p being less than 0.05. Therefore, the British e-
categories of the RBED&UG. The results of each of the seven Government services according to the experts in this study are
experts were gathered and inserted into the statistical software significantly different to the Algerian e-Government services.
SPSS. This software provided us with some important data As a result, the null hypothesis for this study is rejected, and
which will be analysed later on. As expected from the results, the alternative hypothesis is accepted. These results
the British e-Government services scored higher across all compliment the research conducted by the UN which displays
three categories overall as seen in Fig. 1. The Algerian e- the ranking difference regarding the EGDI, where Algeria falls
Government services scored 2.748, in contrast to the British e- behind the UK in every attribute of the page layout. Algeria
Government services which scored 4.758 in the Page Layout must focus on the page layout as a whole, according to the
category. For Text Appearance, the scores for the Algerian and relative importance of its guidelines to make sure that the most
British e-services were 2.785 and 4.562 respectively. Finally, impactful factors of web usability are approached first.
in the Graphics, Images & Multimedia, the results followed a According to the experts, the two strongest attributes in the
similar pattern, where the Algerian e-Government services Algerian e-Government services include Establish Level of
were inferior to the British services (2.594 and 4.143 Importance and Choose Appropriate Line Lengths which both
respectively), however this category suffered the most scored 3.571. On the other hand, the experts believed that
compared to the other two categories. Furthermore, each Choose Appropriate Line Lengths (paired samples statistics
category will be discussed independently later on, and an in- mean = 4.929) was the strongest attribute of the British e-
depth analysis will be provided. The null hypothesis and Government services. Furthermore, the two lowest scored
alternative hypothesis will be accepted or rejected relevantly in variables in this section for the Algerian e-Government
each of the three categories. services include Use Frames when Functions Must Remain

685 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

Accessible and Use Moderate White Space which scored 2.429 Stoppers
4.786 (0.149)
(0.189) p=0.000
and 2.500 respectively. Whereas, for the British e-Services, the
lowest scored guidelines include Use Frames when Functions 10. Set 2.929 (0.202)
Must Remain Accessible and Establish Level of Importance Appropriate -2.000
t=-8.198
(3.356 and 4.429, respectively). The two highest scoring Page 3 (0.646)
4.929 (0.714) p=0.000
categories of paired samples test mean values are Structure for Lengths
Easy Comparison and Use Moderate White Space (both scored
-2.143). This variable represents the difference in the level of 11. Use 2.500 (0.110)
Moderate -2.143
guidelines between the Algerian and British e-services as t=-15.000
3 (0.378)
White Space p=0.000
identified by the experts. Therefore, this result shows that the 4.643 (0.180)
biggest difference between the two tested subjects in this study
12. Choose 3.571 (0.202)
lie in the two abovementioned sections of Page Layout. These
Appropriate -1.357
results display the areas that require the most attention whilst t=-5.203
Line 2 (0.690)
also simultaneously adhering to the level of importance of Lengths 4.929 (0.714) p=0.002
these factors to prioritise the level of importance needed to
identify which attributes must be first approached or 13. Use Frames
approached with the most attention. 2.429 (0.130)
when
Functions -0.929
TABLE VI. RESULTS OF THE SPSS ANALYSIS FOR PAGE LAYOUT t=-2.931
Must 1 (0.838)
3.356 (0.340) p=0.026
Page Layout Remain
Paired Accessible
Samples
Paired
Statistics
Samples B. Text Appearance
Relative Mean (SD)
Guidelines Test Significance
Importance Algerian e- Text is the backbone of any website as it expresses and
Mean
Government
& British e-
(SD) infers a lot about the website whilst also playing a major role in
Government user satisfaction [57]. Therefore, designers must pay more
1. Avoid 2.714 (0.184)
attention to the main points that affect web design usability
Cluttered -1.929 t=-6.485 [58]. Designers must also research the location of their target
5
Displays 4.642 (0.143) (0.787) p=0.001 audience and study their culture as each culture's needs differ
[3]. Therefore, further investigation into the level of
2. Place 3.071 (0.202) infrastructures and the level of user adoption must be
Important undergone. Consequently, creating a website with the
-1.571 t=-5.680
Items 5
(0.731) p=0.001 appropriate font, font size etc. will allow users to become more
Consistently 4.642 (0.180) comfortable while effectively surfing the website. Consistency,
and the level of the quality attributes of the content, which
3. Place 2.790 (0.184) include understandability, relevance, accuracy and finally
Important coverage, should also be focused on accordingly as it is crucial
-1.786 t=-6.250
Items at Top 5 when evaluating the usability of text appearance [59], [60].
4.571 (0.170) (0.756) p=0.001
Centre
Table Ⅶ which presents the results of the SPSS analysis
4. Structure for 2.571 (0.202) for Text Appearance also rejects the null hypothesis of this
Easy -2.143 t=-10.190 research as none of the p values of its relative guidelines
4
Comparison 4.715 (0.184) (0.556) p=0.000 appeared to be greater than 0.05. The analysis also shows that
the Algerian e-services matched the British e-services in one
5. Establish 3.571 (0.170) category (Emphasis Importance) which holds a relative
Level of
4
-0.857 t=-3.286 importance of 2 according to the RBED&UG. The paired
Importance 4.429 (0.170) (0.610) p=0.017 samples statistics mean simply represents the average score of
all seven experts in their assessment of the level of the
6. Optimize 3.286 (0.214) attributes within the websites. The results show that for the
Display -1.000
t=-3.464 Algerian e-Government services, use at Least 12-Point Font
4 (0.764)
Density 4.786 (0.149) p=0.013 (4.357) and Use Similar Fonts (4.071) scored the highest.
Whereas, for the British e-Government services, the experts
7. Align Items 2.642(0.180) believed that Ensure Visual Consistency and Highlighting
-2.143 t=-9.045
on a Page 4 Information were the strongest attributes of this category, as all
4.786(0.149) (0.627) p=0.000
experts unanimously agreed to their perfect score of five. On
8. Use Fluid 2.714 (0.240) the contrary, the lowest scoring factors of Text Appearance
-2.000 t=-7.483
Layouts 3 regarding the Algerian e-services belonged to Highlighting
4.714 (0.149) (0.707) p=0.000
Information (1.856) and Use Attention-Attracting Features
9. Avoid Scroll 3 2.857 (0.922) -1.929 t=-27.000 when Appropriate (2.071). The experts believed that for the
British e-Services, the weakest score belonged to Emphasize

686 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

Importance (2.786). In addition, the two highest scoring C. Graphics, Images, and Multimedia
guidelines regarding the paired samples test mean are Graphics, Images & Multimedia was the densest category
Highlighting Information and Ensure Visual Consistency, amongst the three tested in this study, as it contained a total of
where they scored -3.143 and -2.857, respectively. 16 guidelines. Graphics, images, and multimedia individually
or collectively, play a major role in user interface design. The
TABLE VII. RESULTS OF THE SPSS ANALYSIS FOR TEXT APPEARANCE
proper use of graphics can enhance the usability of websites
Text Appearance [61]. Studies show that images are a crucial aspect when
Paired designing a website [5], [62]. Images also influence the
Samples
Paired
interactive systems of enhancing the positive or negative
Statistics reaction of the users of websites [63]. On the other hand,
Samples
Relative Mean (SD) multimedia can attract more users and efficiently summarize
Guidelines Test Significance
Importance Algerian e-
Government
Mean the message that the designers and the website owners want to
(SD) deliver to the users [64]. The designer should know the
& British e-
Government background of the target users to create a website that respects
1. Use Black 2.142 their culture and values [5]. Furthermore, the designer should
Text on Plain, (0.210) look to get familiar with the internet speed of the target region,
-2.143 t= -7.071
High-Contrast 4 which will dictate the use or lack of graphics, images and
4.186 (0.801) p=0.000
Backgrounds (0.149) multimedia as they can take a while to upload/download or
may not upload/download if the internet speed is not enough or
2. Format 2.714 if the file is too big. Therefore, designers must ensure that all
Common (0.240)
-2.143 t=-7.579 these factors are considered to avoid failing their target users
Items 4 and avoid providing them with an unattractive website that
4.857 (0.748) p=0.000
Consistently (0.092) their internet connection speed is too slow to handle that may
potentially disrespect their culture, which ultimately causes the
3. Use Mixed- 3.714
(0.101)
whole website to fail. The abovementioned factors are crucial
Case for Prose -0.714 t=-2.970
4 to the success of any website.
Text 4.429 (0.636) p=0.000
(0.170) The results from this category (Table Ⅷ) follow a similar
4. Ensure Visual 2.143 pattern to the previous categories, where according to the
(0.210) -2.857 t= -13.587 experts; the British e-Services are a lot superior to the Algerian
Consistency 4
5.000 (0.556) p=0.000
e-services. Across all guidelines, there is a significant
(0.000)
2.286 difference between the two tested subjects. However, for
5. Use Bold Text
(0.184) -2.714 t=-14.717 Introduce Animation, the value of p is 0.726 which is greater
Sparingly 3
5.000 (0.488) p=0.003 than 0.05 which means that there is no significant difference
(0.000) between the Algerian and British e-Government services in
6. Use 2.071 relation to this guideline. Thus, for this category alone which
Attention- (0.202) has a relative importance of 2 according to the RBED&UG, the
Attracting -2.429 t=-14.283 null hypothesis is accepted, and the alternative hypothesis is
3
Features when 4.500 (0.450) p=0.000 rejected. The strongest attributes according to the experts
Appropriate (0.154)
regarding the Algerian e-services are Using Photographs of
People (4.429) and Include Logos (4.071), whereas the weakest
4.071
7. Use Familiar
(0.130)
attributes include Graphics Should Not Look like Banner Ads
-0.571 t=-2.828
Fonts 3 and Ensure Web Site Images Convey Intended Messages
4.643 (0.535) p=0.000
(0.143) which both scored 1.643 according to the average rating of the
8. Use at Least 4.357 experts. The Graphics Should Not Look like Banners guideline
-0.429
12-Point Font 3
(0.922)
(0.345)
t=-3.286 had the highest paired samples test mean (-3.357) amongst all
4.786 p=0.000 guidelines of this category. Overall, the results of this category
(0.101) illustrate the shortcomings of the Algerian e-services and how
9. Color-Coding 2.500
(0.154)
they should approach each category according to its evaluated
and -2.500 t=-16.202
2
(0.408) p=0.000
level by the experts and the relative importance determined by
Instructions 5.000
(0.000) the guidelines.
10. Emphasize 2.786
(0.184) -1.857 t=-6.569
Importance 2
2.786 (0. 748) p=0.011
(0.184)
11. Highlighting 1.856
(0.210) -3.143 t=-14.946
Information 2
5.000 (0.556) p=0.000
(0.000)

687 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

TABLE VIII. RESULTS OF THE SPSS ANALYSIS FOR PAGE LAYOUT 12. Introduce 2.571(0.170) -0.143
t=-3.67
Animation 2 (1.029)
2.714 p=0.726
Graphics, Images, and Multimedia
(0.240)
Paired 13. Emulate 2.286
Samples Real-World (0.185) -2.071 t=-9.021
Paired 2
Statistics Objects 4.357 (0.607) p=0.000
Samples (0.143)
Relative Mean (SD)
Guidelines Test Significance
Importance Algerian e- 14. Use
Mean 1.857(0.210)
Government Thumbnail
(SD)
& British e-
Images to
Government -0.786 t=-3.667
Preview 2 2.642
1. Use Simple 3.429 (0.567) p=0.010
(0.202) Larger (0.210)
Background -1.285 t=--6.971
4 Images
Images 4.714 (0.487) p=0.000
(0.149)
15. Use Images 2.000
2. Label 2.286
to Facilitate (0.218) -3.000 t=-13.748
Clickable (0.185) -2.429 t=-12.021 1
4 Learning 5.000 (0.577) p=0.000
Images 4.714 (0.535) p=0.000
(0.000)
(0.101)
16. Using 4.429
3. Ensure that 1.856
Photographs (0.130) 2.642 t=8.721
Images Do (0.261) 1
-3.071 t=-11.103 of People 1.787 (0.801) p=0.000
Not Slow 4
4.929 (0.732) p=0.000 (0.1184)
Downloads (0.714)

4. Use Video,
V. CONCLUSION
2.143(0.143)
Animation, This study sheds light on e-government websites that
-1.929 t= -9.546
and Audio 4 4.071 (0.535) p=0.000
provide electronic services which facilitate communication
Meaningfully (0.170) between government agencies and citizens. User satisfaction
influences web design regardless of the domain, therefore their
5. Include 4.071 needs and expectations must be met by the developers. The
Logos (0.170) -0.714 t=-4.804 main subject of this study is the Algerian e-Government which
4
4.786 (0.394) p=0.003 according to the UN has an EGDI Global Rank of 120. This
(0.101) rank was surprising for the researcher considering the
6. Graphics 1.643 population size, infrastructure, and economy of the largest
Should Not (0.143)
-3.357 t=-23.500 country in Africa. Therefore, this study attempted to identify
Look like 4 the reasons that may be rooted to the lack of success of
(0.378) p=0.000
Banner Ads 5.000(0.000)
Algerian e-Government services at a global level. For this
reason, the UK which ranks 7th in the EGDI Global Rank was
7. Limit Large 3.429(0.170) selected as the second test subject to serve as a comparative
Images
-1.571 t=-9.242 tool to help the researcher identify the underlying reasons for
Above the 4 5.000 (0.450) p=0.000 the underperformance of these services. However, this study
Fold (0.000) had to decide on an evaluation tool to use to gather relevant
8. Ensure Web 1.643
data. Consequently, the researcher identified an ideal set of
Site Images (0.210) guidelines known as the RBED&UG which was selected in
Convey -3.214 t=-13.367
this study due to its exclusivity and popularity in government
Intended
4
4.857 (0.636) p=0.000 agencies as the domain of this topic is e-government usability.
Messages (0.092) The interesting factor regarding these guidelines is that it
provides relative importance to each guideline. The researcher
9. Limit the Use
was really interested by this feature as the researcher believes
2.214(0.149)
of Images 3
-2.786 t=-18.735 that the most impactful guidelines must be addressed before the
5.000 (0.393) p=0.000 less important ones. Although these guidelines are used as an
(0.000)
10. Include
evaluation when designing websites, the researcher identified
3.786(0.184) the potential of this to be converted into a questionnaire that
Actual Data
with Data 3
-0.929 t=-3.653 could be used by experts when evaluating a system. The
4.714 (0.667) p=0.011 complexity of certain aspects of these guidelines, or now
Graphics (0.101)
proposed questionnaire, suggests that it is not appropriate to be
11. Display 1.857 evaluated by normal users, but instead by experts within the
Monitoring (0.210) field. The researcher believes that this proposal can prove to be
-2.857 t=-15.894 very effective as experts are able to rate the chapters and
Information 3
4.714 (0.475) p=0.000 guidelines accordingly, thus the developers can really benefit
Graphically (0.149)
from their evaluation. There are a total of 18 guidelines in
RBED&UG; however, after thorough research and information

688 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

gathering, the researcher selected only three categories (Page [10] G. Hu, W.Pan, M. Lu, and J. Wang, "The widely shared definition of
Layout, Text Appearance, and Graphics, Images & e‐Government: An exploratory study." The Electronic Library (2009).
Multimedia) due to their proven role in web usability. This [11] Word bank. Accessed 14 August,2022 Retrieved from
https://www.worldbank.org/en/topic /digitaldevelopment/brief/e-
research ultimately aims to explore the reasons that influence government.
the usability of Algerian e-Government services via comparing [12] ISO 9241-210, International Standard: Ergonomics of Human–System
it with UK e-Government services against a set of guidelines. Interaction −Part 210: Human-Centred Design for Interactive Systems,
The results were gathered after an experiment which 1st edition 2010-03-15, Reference number ISO 9241-210:2010(E).
constituted of an expert evaluation of the two selected systems [13] J. V. Dijk, W. Pieterson, A.V. Deuren, and W. Ebbers, "E-services for
via completing the proposed questionnaires handed out to citizens: The Dutch usage case." International Conference on Electronic
them. Their results were inputted into SPSS which provided Government. Springer, Berlin, Heidelberg, 2007.
statistical analysis of the set of data. This data helped identify [14] Z. Huang, and M. Benyoucef. "Usability and credibility of e-government
the mean values of each guideline, their standard deviation, and websites." Government information quarterly 31.4 (2014): 584-595.
the significance value of each one. The results display the sheer [15] M. Benaida, "Cross-cultural web design and education: a comparison
between arab universities and us universities based on Hofstede cultural
difference in quality between the Algerian and British e- dimensions." IJCSNS 18.10 (2018).
Government services across all three guideline categories. The [16] P. Verdegem, and G. Verleye, "User-centered E-Government in practice:
three weakest guidelines of each chapter for the Algerian e- A comprehensive model for measuring user satisfaction." Government
Government services include Use Frames when Functions information quarterly 26.3 (2009): 487-497.
Must Remain Accessible, Highlighting Information, and [17] V.Venkatesh,, M. G. Morris, G.B. Davis, and F.D. Davis, (2003). User
Graphics Should Not Look like Banner Ads (belonging to Page acceptance of information technology: Toward a unified view. MIS
Layout, Text Appearance, and Graphics, Images & Multimedia quarterly, 425-478.
respectively). The Algerian e-Government services [18] J. Kaipio, T. Lääveri, H. Hyppönen, S. Vainiomäki, J. Reponen, A.
representatives and developers should focus on these results Kushniruk, E. Borycki, and J. Vänskä, “Usability problems do not heal
by themselves: National survey on physicians’ experiences with EHRs
and improve their systems according to the results of this in Finland." International Journal of Medical Informatics 97 (2017):
research, giving most attention to the weakest areas as 266-281.
identified by this study whilst also paying attention to their [19] V. Venkatesh, H. Hoehle, and R. Aljafari. "A usability evaluation of the
relative importance. The combination of these two integral Obamacare website." Government information quarterly 31.4 (2014):
factors will help provide effectiveness and efficiency during 669-680.
the evaluation and development process of a system. [20] T. Stewart, "Websites–quality and usability." Behaviour & Information
Technology 31.7 (2012): 645-646.
VI. LIMITATION AND FUTURE WORK [21] N.E. Youngblood, and J. Mackiewicz. "A usability analysis of municipal
government website home pages in Alabama." Government Information
This study provides solutions specific to the three usability Quarterly 29.4 (2012): 582-588.
guideline sections researched. Therefore, there is an [22] A. F. Al-Bataineh, and S. H. Mustafa. "How Jordanian e-Government
opportunity for further research to expand on the remaining 15 websites respond to the needs of people with disabilities." 2016 7th
guidelines of the RBED&UG. In addition, only two countries International Conference on Computer Science and Information
were examined in this research. Technology (CSIT). IEEE, 2016.
[23] M.Z. Yazid, A.H, Jantan, A.A. Abd Ghani, A. Kamaruddin, and N.
REFERENCES Admodisastro, "Accessibility design issues with Malaysian news
[1] N, B. Mohammed, and H. B Abdelhakim. "The success factors of e- websites: a case study using a checker and WAVE." Int J Eng Technol
government strategy in North Africa: A comparative study between 7.4 (2018): 69-73
Algerian and Tunisian digital strategy." 2014 4th International [24] C. Yingnan, C., Y. Xiaoping, Y, and X. Wang, "The research and
Symposium ISKO-Maghreb: Concepts and Tools for knowledge realization of the system of the timeliness assessment of E-government
Management (ISKO-Maghreb). IEEE, 2014. web sites based on the semantic analysis." 2011 International
[2] F. Bélanger, and L. Carter, "Trust and risk in e-government adoption." Conference on E-Business and E-Government (ICEE). IEEE, 2011.
The journal of strategic information systems 17.2 (2008): 165-176. [25] Zhao, C., Marghitu, D., & Mou, L. (2016). An Exploratory Study of the
[3] M. Benaida. "Significance of culture toward the usability of web design Chinese Provincial Government Portals Accessibility. In E-Learn:
and its relationship with satisfaction." Universal Access in the World Conference on E-Learning in Corporate, Government,
Information Society 21.3 (2022): 625-638. Healthcare, and Higher Education (pp. 1093-1101). Association for the
[4] N. Bevan, Nigel, and S. Lonneke, "Are guidelines and standards for web Advancement of Computing in Education (AACE).
usability comprehensive?" International Conference on Human- [26] S. F. Verkijika, and L, De Wet. "E-government adoption in sub-Saharan
Computer Interaction. Springer, Berlin, Heidelberg, 2007. Africa." Electronic Commerce Research and Applications 30 (2018): 83-
[5] M. Benaida, Developing Arabic usability guidelines for e-learning 93.
websites in higher education. University of Salford (United Kingdom), [27] M. Dollie, and S. Kabanda. "e-Government in Africa: perceived
2014. concerns of persons with disabilities (PWDs) in South Africa."
[6] N. Bevan, "Guidelines and standards for web usability." Proceedings of Proceedings of the European Conference on e-Government, ECEG.
HCI International. Vol. 2005. Lawrence Erlbaum, 2005. 2017.
[7] J. D. Twizeyimana, and A. Annika, “The public value of E- [28] S.H. Said, Investigation of usability shortcomings on the user interfaces
Government–A literature review." Government information quarterly of web based e-Government systems: a case of Tanzania. Diss. The
36.2 (2019): 167-178. University of Dodoma, 2015.
[8] United Nations e-government, E-Government Survey, The Future of [29] W. Yaokumah,, S. Brown, and R. Amponsah. "Accessibility, quality and
Digital Government, 2020. performance of government portals and ministry web sites: a view using
diagnostic tools." 2015 Annual Global Online Conference on
[9] S.E. Colesca, and D. Liliana. "Adoption and use of e-government Information and Computer Technology (GOCICT). IEEE, 2015.
services: The case of Romania." Journal of applied research and
technology 6 (2008): 204-217.

689 | P a g e
www.ijacsa.thesai.org
 (IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 14, No. 1, 2023

[30] M. J. Albers, "Tapping as a measure of cognitive load and website [47] A. Atashi, R. Khajouei, A. Azizi, and A. Dadashi,. "User Interface
usability." Proceedings of the 29th ACM international conference on problems of a nationwide inpatient information system: a heuristic
Design of communication. 2011. evaluation." Applied clinical informatics 7.01 (2016): 89-100
[31] N.S. Aziz, A. Kamaludin, N.S. Sulaiman, and A. Yacob, "Measuring the [48] J. Falkowska J. Sobecki,, and M. Pietrzak. "Eye tracking usability
Role of Satisfaction in Website Usability Model." Advanced Science testing enhanced with EEG analysis." International Conference of
Letters 24.10 (2018): 7762-7768. Design, User Experience, and Usability. Springer, Cham, 2016.
[32] M. H. Miraz, A. Maaruf, and P. Excell. "Multilingual website usability [49] R. Inostroza, C. Rusu, S. Roncagliolo, V. Rusu,, and C.A. Collazos,
analysis based on an international user survey." arXiv preprint "Developing SMASH: A set of SMArtphone's uSability Heuristics."
arXiv:1708.05085 (2017). Computer Standards & Interfaces 43 (2016): 40-52.
[33] M.Z. Lallmahomed, N. Lallmahomed, and G.M. Lallmahomed, [50] P. Chynał, and J. Sobecki. "Statistical verification of remote usability
"Factors influencing the adoption of e-Government services in testing method." Proceedings of the Mulitimedia, Interaction, Design,
Mauritius." Telematics and Informatics 34.4 (2017): 57-72. and Innnovation. 2015. 1-7.
[34] M. Kurfalı, A. Arifoğlu, G. Tokdemir, and Y. Paçin, (2017). Adoption [51] Bezerra, C., Andrade, R. M., Santos, R. M., Abed, M., de Oliveira, K.
of e-government services in Turkey. Computers in Human Behavior, 66, M., Monteiro, J. M. & Ezzedine, H. (2014, October). Challenges for
168-178. usability testing in ubiquitous systems. In Proceedings of the 26th
[35] C. Lodge, "The impact of culture on usability: designing usable products Conference on l'Interaction Homme-Machine (pp. 183-188).
for the international user." International Conference on Usability and [52] S. M. Ko, W. S. Chang, and Y.G. Ji, "Usability principles for augmented
Internationalization. Springer, Berlin, Heidelberg, 2007. reality applications in a smartphone environment." International journal
[36] A. M. Aladwani, (2013). A cross-cultural comparison of Kuwaiti and of human-computer interaction 29.8 (2013): 501-515.
British citizens' views of e-government interface quality. Government [53] C. Boothe, L. Strawderman, and E. Hosea. "The effects of prototype
Information Quarterly, 30(1), 74-86. medium on usability testing." Applied ergonomics 44.6 (2013): 1033-
[37] Usability testing, Accessed 20 August 2022. Retrieved from 1038.
https://www.usability.gov/how-to-and-tools/methods/usability [54] A. Sivaji, A. Abdullah, and A.G. Downe, "Usability testing
testing.html. methodology: Effectiveness of heuristic evaluation in E-government
[38] D. Fonseca, and F. J. arcía-Peñalvo. "Interactive and collaborative website development." 2011 fifth Asia modelling symposium. IEEE,
technological ecosystems for improving academic motivation and 2011.
engagement." Universal Access in the Information Society 18.3 (2019): [55] T. P. Thyvalikakath, V. Monaco, H. Thambuganipalle and T. Schleyer,
423-430. "Comparative study of heuristic evaluation and usability testing
[39] H. Huang, and C. W. Chen. "Creating different learning experiences: methods." Studies in health technology and informatics 143 (2009): 322.
assessment of usability factors in an interactive three-dimensional [56] A. C. B. Garcia, C. Maciel, and F. B. Pinto, "A quality inspection
holographic projection system for experiential learning." Universal method to evaluate e-government sites." International Conference on
Access in the Information Society 18.3 (2019): 443-453. Electronic Government. Springer, Berlin, Heidelberg, 2005.
[40] M. Fuller-Tyszkiewicz, B. Richardson, B. Klein, H. Skouteris, H. [57] I. Prazina, E. Kovač, D. Pozderac, Z. Ramić, V. Okanović, and E.
Christensen, D. Austin, ... and A. Ware, "A mobile app–based Cogo, "Testing Web Page Layouts Using Galen Framework Tests
intervention for depression: end-user and expert usability testing study." Generated by The Mockup Tool." 2019 27th Telecommunications
JMIR mental health 5.3 (2018): e9445. Forum (TELFOR). IEEE, 2019.
[41] J. Krzewińska, A. Indyka-Piasecka, M. Kopel, E. Kukla, Z, Telec, and [58] R. Volentine, A. Owens, C. Tenopir, and M. Frame, “Usability testing to
B. Trawiński, "Usability testing of a responsive web system for a school improve research data services." Qualitative and Quantitative Methods
for disabled children." Asian Conference on Intelligent Information and in Libraries 4.1 (2017): 59-68
Database Systems. Springer, Cham, 2018. [59] E. Mwangi, S. Kimani, and A. Mindila. "A Review Of Web-Based GIS
[42] F.J. C. De Souza, I.T. Monteiro, and P. M. Jucá, "Game for aNy Usability Elements." J. Inform. Technol 4 (2019): 2-13.
heuristic evaluation (G4NHE): a generalization of the G4H gamification [60] G. W. Tan, and K.K. Wei, "An empirical study of Web browsing
considering different sets of usability heuristics." Universal Access in behaviour: Towards an effective Website design." Electronic Commerce
the Information Society 18.3 (2019): 489-505. Research and Applications 5.4 (2006): 261-271.
[43] J. Díaz, J., C. Rusu, C, and C.A. Collazos, "Experimental validation of a [61] T. Yang, J. Linder, and D. Bolchini. "DEEP: design-oriented evaluation
set of cultural-oriented usability heuristics: e-Commerce websites of perceived usability." International Journal of Human-Computer
evaluation." Computer Standards & Interfaces 50 (2017): 160-178. Interaction 28.5 (2012): 308-346.
[44] P. M. Jucá, I. Teixeira Monteiro, and J. C. D Souza Filho, "Game for [62] S. Djamasbi, M. Siegel, and T. Tullis. "Generation Y, web design, and
heuristic evaluation (G4H): a serious game for collaborative evaluation eye tracking." International journal of human-computer studies 68.5
of systems." International Conference on Human-Computer Interaction. (2010): 307-323.
Springer, Cham, 2017.
[63] A. M. Fiore, and H. J. Jin,. "Influence of image interactivity on
[45] O. Alhadreti, and P. Mayhew. "To intervene or not to intervene: An approach responses towards an online retailer." Internet Research
investigation of three think-aloud protocols in usability testing." Journal (2003).
of Usability Studies 12.3 (2017): 111-132.
[64] S. Ram, and N. Paliwal. "Design and Development of Multimedia Based
[46] B. Murillo, S. Vargas, A. Moquillaza, L. Fernández, and F. Paz, User Education Program: The Advantages of YouTube." (2013).
"Usability testing as a complement of heuristic evaluation: A case
study." International conference of design, user experience, and
usability. Springer, Cham, 2017.

690 | P a g e
www.ijacsa.thesai.org