Nova Southeastern University

NSUWorks
CCE Theses and Dissertations College of Computing and Engineering

2019

Examining the Impact of Learning Management

Systems in Computer Programming Courses
Mohammed Naif Alatawi
Nova Southeastern University, [email protected]

This document is a product of extensive research conducted at the Nova Southeastern University College of
Computing and Engineering. For more information on research and degree programs at the NSU College of
Computing and Engineering, please click here.

Follow this and additional works at: https://nsuworks.nova.edu/gscis_etd

Part of the Computer Sciences Commons

Share Feedback About This Item

NSUWorks Citation
Mohammed Naif Alatawi. 2019. Examining the Impact of Learning Management Systems in Computer Programming Courses. Doctoral
dissertation. Nova Southeastern University. Retrieved from NSUWorks, College of Engineering and Computing. (1084)
https://nsuworks.nova.edu/gscis_etd/1084.

This Dissertation is brought to you by the College of Computing and Engineering at NSUWorks. It has been accepted for inclusion in CCE Theses and
Dissertations by an authorized administrator of NSUWorks. For more information, please contact [email protected].
Examining the Impact of Learning Management Systems in Computer
Programming Courses

By

Mohammed Naif Alatawi

A Dissertation Submitted in Partial Fulfillment of the Requirements

for the Degree of Doctor of Philosophy
in
Information Systems

College of Computing and Engineering

Nova Southeastern University
2019
An Abstract of a Dissertation Submitted to Nova Southeastern University in Partial Fulfillment of
the Requirements for the Degree of Doctor of Philosophy

Examining the Impact of Learning Management Systems in Computer

Programming Courses

By
Mohammed Naif Alatawi
2019

The development of software and communication technologies in education has led the majority
of universities worldwide to integrate the functions of Learning Management Systems (LMSs)
into their learning environments. LMSs offers several features that encourage their use by
universities and other educational institutions, such as unlimited access to course content, easy
tracking of learners’ progress and performance, and reduced costs in terms of both money and
time. Most existing LMS studies have been focused on experienced LMS users who are familiar
with its functions, with little consideration given to new users. Furthermore, although previous
researchers have identified various means of enhancing the effectiveness of LMS use, no
consensus has yet been reached on which of these features most successfully improve the learning
outcomes of new learners enrolled in programming courses.

The purpose of this study, therefore, was to examine the usability of particular LMS features and
their impact on learning outcomes for freshman students enrolled in programming courses.
Through the Virtual Programming Lab (VPL) and discussion forums, particular LMS features
have been considered. For this study, a quantitative quasi-experimental design was employed,
including experimental and control groups of new students enrolled in an introductory
programming course that involved different LMS features. These features have been considered
in the place of treatment in this experiment, in which the level of difference between participants
in the two groups was compared.

This study involved two main dependent variables: LMS features’ usability and learning
achievement. For the first dependent variable, LMS usability, the participants completed a survey,
based on the components of Shackel’s usability model (1991), to evaluate the effectiveness of the
LMS features’ usability. Four constructs underpin this model: effectiveness, learnability,
flexibility, and attitude. For the second dependent variable, learning achievement, the final grade
was used to measure the impact of these two LMS features on learning achievement between the
two groups.

The results revealed significance differences related to LMS features’ usability and learning
achievement between the experimental group and the control group. Participants in the
experimental group reported greater LMS usability than did those in the control group, and
overall course scores indicated improved learning performance in members of the experimental
group who applied the VPL and discussion forms features of programming courses.
 Acknowledgements

I am grateful to all who supported me in the completion of this dissertation. I am particularly

thankful to my wonderful parents and to my wife, Khulud, and my kids, Naif, Reema, and Renad,
for their patience and endless encouragement during this time.
I also thank my advisor, Dr. Ling Wang, for her guidance and support throughout the dissertation
process. She was always ready to answer a question, respond to an inquiry, or otherwise point me
in the right direction. Special thanks go also to my committee members, Dr. Junping Sun, and Dr.
Inkyoung Hur, for all their valuable advice and guidance.
 Table of Contents

Abstract ii
List of Tables v
List of Figures vi

Chapters

1. Introduction 1
Background 1
Problem Statement 2
Dissertation Goal 5
Research Questions 6
Hypothesis 6
Relevance and Significance 6
Barriers and Issues 6
Definition of Terms 7
List of Acronyms 7
Summary 8

2. Review of the Literature 9

Overview 9
Theoretical Foundation 9
Past Literature and Identification of Gaps 11
Analysis of the Research Methods Used 12
Synthesis of the Literature 14
Summary 17

3. Methodology 18
Research Approach 18
Participants 18
Weekly Groups Assessment 19
LMS Environment Setup 21
 Data Collection 24
Data Analysis 25
Resources 25

4. Results 27
Introduction 27
Pre-Analysis Data Screening 28
Demographic Data Analysis 28
Data Analysis and Results 31
Research Hypothesis 1 31
Research Hypothesis 2 33
Summary 35

5. Conclusions, Implications, Recommendations, and Summary 36

Conclusions 36
Study Limitations 39
Implications 40
Recommendations for Future Research 40
Summary 41

Appendices 42
A. Invitation to Participate 42
B. Informed Consent 43
C. Study Participants’ Survey Instrument 45
D. Participating Institution Study Approval 51
E. IRB Approval Letter 52

References 53
 List of Tables

Tables

1. Summary of Participants’ Gender (N=59) 29

2. Summary of Participants’ Current Education Level (N=59) 29

3. Summary of Participants’ Level of Programming Knowledge (N=59) 30

4. Summary of Participants’ Learning Style (N=59) 30

5. Summary of Participants’ Prior LMS Utilization (N=59) 31

6. Independent Sample t-Test Results of Usability (N=59) 32

7. Descriptive Statistics Results of Usability Model for Treatment Group (N=29) 32

8. Descriptive Statistics Results of Usability Model for Control Group (N=30) 33

9. Independent Sample t-Test Results of Overall Scores (N=59) 34

10. Summary of Chi-Square Tests Results N (59) 39

 List of Figures

Figures

1. Weekly Groups Assessment Plan 20

2. Types of Programming Classes 21

3. Integration LMS with VPL 22

4. VPL Instructor Dashboard View 22

5. VPL Student Dashboard View 23

6. Discussion Forums in LMS 24

7. The Normality and Linearity Test Outputs 28

8. Independent Sample t-Test Results of Overall Scores Comparison 34

 1

Chapter 1

Introduction

Background

During the past decade, information technology (IT) has made a number of advances that

have significantly benefited the field of education, perhaps most notably via integrating the LMS

into the education sector (Lasrado, 2009). As one of today’s most rapidly developing

technologies, LMSs see wide use in educational institutions the world over and, like other types

of software, are available in either free and open-source or commercial versions (Al-Busaidi &

Al-Shihi, 2010).

An LMS, sometimes also called a learning platform, comprises a broad slate of

subsystems designed to help instructors and students alike as they use online learning services

(Paulsen, 2002). Recognizing the importance of this learning platform, universities around the

world are modifying their strategies by adopting technologies that they believe will help them

achieve their goals in this age of ready information (Alharbi & Drew, 2014). Indeed, the adoption

of such information systems does much to advance the collaborative classroom, opening a new

frontier of pedagogy (Clapp & Swenson, 2013).

Alassaf, Harfoushi, Obiedat, and Hammouri (2014) defined LMSs as education software

applications that allow both instructors and learners to complete a range of tasks, including

managing student documents, monitoring student activities, and administering exams, quizzes,

and assignments. Furthermore, institutions can use LMSs to organize training programs, control

course content, facilitate online communication between instructors and learners, and manage
 2
learning progress events. Accordingly, their benefits and functions make them highly important

applications for any educational institution.

However, according to Chughtai, Zhang, and Craig (2015), the user experience (UX) of

LMSs is a serious cause for concern because a good level of usability can ensure that learners will

use the LMS’s features more effectively. Moreover, a consideration of learners’ attitudes toward

LMSs and the use of their experience in the development process is key to reducing the negative

impact of LMS use during the learning procedure. This is because the main goal of using any

LMS is to find an effective learning environment that supports learners to achieve their learning

outcomes effectively and easily (Chaffar & Frasson, 2004). Furthermore, the rapid growth in the

variety of online programs in higher education has spurred researchers and stakeholders to

identify dynamic LMS features that can deliver in particular courses considered challenging for

learners, such as computer science and healthcare (Azevedo & Feyzi-Behnagh, 2010).

Accordingly, the main contribution of this study is an examination of the usability of LMS

features and their influence on students’ learning outcomes in programming courses.

Problem Statement

The existing literature is focused on experienced LMS users, with comparatively little

attention paid to newcomers (i.e., freshman-level university students). Furthermore, although

previous researchers have identified various ways of enhancing the effectiveness of LMS use, no

consensus has yet been reached regarding which of these features most successfully improves

learning outcomes for new learners enrolled in programming courses.

However, users’ experience levels in using LMSs play a significant role in determining

the degree to which the desired benefits are realized. Based on this factor, Al Hamad (2016)

evaluated students’ perceptions of LMS implementation based on use of the Moodle platform at

Fujairah College, United Arab Emirates. Examining a population comprising a small group of
 3
students with strong computer skills and experience of using an LMS, the researcher found that a

positive perception of such use corresponded to an existing high level of technological skill

among the participants.

Moreover, when new learners struggle to understand the content of a course, they are less

likely to use the LMS effectively, or they show a negative response toward it. For instance,

Rahman et al. (2010) adopted Shackel’s usability model (1991) and its four

principles⎯effectiveness, learnability, flexibility, and attitude⎯while seeking to gauge the

effectiveness of the Open University Malaysia (OUM) LMS using the following four measures:

ease of mastery, error tolerance, speed, and quality. The authors evaluated the effectiveness of the

LMS used by learners enrolled in different academic courses with varying levels of difficulty,

before randomly selecting students to complete a survey. Their findings revealed that new

learners were negatively affected by an LMS when using the system for the first time in their

courses (Rahman et al., 2010).

Nevertheless, a few researchers have evaluated the usability of LMS features for freshman

students in challenging courses. Alhazbi (2016), for example, investigated the effects of students’

use of LMS features as part of programming courses, hoping to increase levels of engagement and

motivation among those taking introductory courses in this area at Qatar University. Although the

author created an LMS environment designed to support students’ collaborative learning by

increasing levels of interaction between them and instructors, the latter faced challenges when

they sought to use class activities to involve learners in programming courses during the semester.

The findings indicated that students had a low level of the knowledge required to use certain LMS

features; in particular, learners in these courses needed continuous guidance and support, such as

communication tools intended to allow them to express their thoughts about class topics through

LMS discussion forums (Alhazbi, 2016).

 4
Furthermore, the VPL tool is one of the most important tools that could be integrated to

LMSs in programming courses to support learners. For instance, Kaunang (2016) examined

students’ perceptions of a VPL tool that was integrated with an LMS at the University of Sam

Ratulangi. The aim of this study was to reveal the weaknesses and strengths of VPL tools when

used for teaching and delivering online programming classes in an engineering department in the

academic year 2015–16. In his methodology, the author designed an online course for an

electrical engineering program, making it available to students through the Moodle LMS portal.

For the final evaluation, he conducted a survey that was focused on the use of VPL activities with

Moodle LMS to complete the programming assignments and evaluate the coding exercise. The

results showed that, in general, students had a strong VPL in most respects, but the author found a

degree of weakness in students using VPL as a coding editor. In addition, he noted that some of

the participants’ feedback, in the section on VPL questions, was negative in terms of VPL

availability.

What is more, no previous studies have successfully demonstrated exactly which LMS

features are most useful for improving students’ learning outcomes. Mwalumbwe and Mtebe

(2017), for example, evaluated student performance at Tanzania’s Mbeya University of Science

and Technology (MUST) using the Moodle LMS. The researchers found no significant enhancing

effect associated with a number of factors related to LMS use, including frequency of access,

length of time spent on the platform, and quantity of materials downloaded. However, they did

find a notable effect pertaining to other factors associated with LMS use, such as level of class

interaction and engagement in discussion posts and course exercises.

Additionally, in an empirical study of the effectiveness of learning enhanced by LMS

technology, Chowdhry, Sieler, and Alwis (2014) investigated the relationship between LMS

features and students’ academic performance. They compared student data for the 2013–14
 5
academic year at Edinburgh Napier University, Scotland, using three different models to inform

their analyses of students’ final grades and levels of access to LMS features. The authors

concluded that student access to material features did not directly affect academic performance in

the Low Module (LM); however, they did recommend that future researchers compare constant

factors between models to evaluate different LMSs. Moreover, they suggested that scholars

investigate LMS access for specific activities with a view to evaluating student performance,

which would allow them to identify LMSs that are more and less effective, with the ultimate goal

of improving the student learning process (Chowdhry et al., 2014).

The studies described thus far reveal a lack of research in which efforts are made to

evaluate the usability of LMS features in terms of impact on effective learning outcomes for

freshman students enrolled in programming courses at higher education institutions. In fact, most

studies have been focused on LMS features’ usability in general terms and among users with a

high LMS skills level (Alhazbi, 2016). Furthermore, no agreement has yet been reached regarding

which LMS features most successfully improve learning outcomes in challenging courses

(Rahman et al., 2010).

Dissertation Goal

The goal of this study was to evaluate the usability of LMS for freshman-level students

who employ VPL and discussion forum features and its impact on their learning outcomes in

programming courses. Whereas previous researchers have investigated the effects of LMS

features in general terms, the aim here was to examine a specific combination of LMS features to

determine which ones are likely to enhance learning environment outcomes for new learners

enrolled in programming courses. Accordingly, the focus was on those features previously

identified by researchers as most important for the delivery of such courses.

 6
Research Questions

RQ1: What differences exist in LMS usability between freshman students who are using, or not

using, LMS features (VPL and discussion forum) in programming courses?

RQ2: What differences exist in learning achievement between freshman students who are using,

or not using, LMS features (VPL and discussion forum) in programming courses?

Hypothesis

Hypothesis 1: There are significance differences in LMS usability between freshman students

who are using, or not using, LMS features (VPL and discussion forum) in programming courses.

Hypothesis 2: There are significance differences in learning achievement between freshman

students who are using, or not using, LMS features (VPL and discussion forum) in programming

courses.

Relevance and Significance

The significance of this study was threefold. First, the findings added to the body of

knowledge related to LMS research by evaluating the usability of users’ experience and LMS

functions that may increase the efficiency of learning environments in difficult courses. Second,

gaining mastery over the learning process, especially for new students, could increase the quality

of university education programs’ outcomes. Thus, universities around the world could tailor

LMS features to their students’ needs, easing new starts’ transitions into challenging courses. In

such an environment, LMS use can motivate and support learners to increase their own level of

knowledge.

Barriers and Issues

The aim of this study was to investigate which features influence the effectiveness of LMS

use, with a view to improving students’ levels of academic achievement in difficult courses and
 7
enhancing their outcomes in said courses. To this end, the target population was freshmen

majoring in computer science; thus the findings are not generalizable to students of different

majors or attending any college. Furthermore, students’ ability to use LMSs could be limited by

their varying levels of computer skills and different technological backgrounds.

Definition of Terms

Effectiveness – Defined as users’ performance in the completion of systems tasks within some

environments; it can be measured by characteristics of interaction, such as time and errors

(Rahman, Ghazali, & Ismail, 2010; Alabbadi, M. M., 2010).

Flexibility – Defined as the adaptation to change in tasks and environments that can be more

convenient for specific design (Rahman et al., 2010; Alabbadi, M. M., 2010).

Learnability – Defined as the level of users’ ability to learn and complete systems tasks and learn

the system’s details, whether or not given training (Rahman et al., 2010; Alabbadi, M. M., 2010).

Usability– Defined as the degree to which the system can be used to achieve a specific goal by

specific users with efficiency, effectiveness, and satisfaction in a certain task of use (Bevan,

Carter, Earthy, Geis, & Harker, 2016).

Users’ Attitude – Defined as users’ satisfaction with the system, whether they continue using the

system or improve their use of the system in terms of discomfort, tiredness, and personal effort

(Rahman et al., 2010; Alabbadi, M. M., 2010).

List of Acronyms

IT – Information Technology

LMS – Learning Management System

VPL – Virtual Programming Lab

 8
Summary

In Chapter One of this study, the following research areas are discussed: problem

statement, study goals, research questions, significance, barriers and issues, definition of terms,

and list of acronyms. The existing literature is focused on experienced LMS users, with little

attention paid to freshman-level university students, and although previous researchers have

identified several ways of enhancing the effectiveness of LMS use, no consensus has yet been

reached regarding which LMS features most successfully improve learning outcomes for new

enrollees in programming courses. The main goal of this study was thus to examine the impact of

LMSs’ usability on such outcomes for freshman-level students who employ particular LMS

features in said courses. A specific combination of LMS features was also examined to determine

which ones enhance learning environment outcomes for new learners enrolled in courses in which

VPL and discussion forum features are applied. Chapter One also features an introduction to the

research study’s significance, barriers, and issues. Finally, this chapter includes a list of

definitions of key terms for the reader to understand.

 9

Chapter 2

Review of the Literature

Overview

In this study, the literature review is focused on whether the effective usability of certain

LMS features can improve a freshman’s learning performance in challenging courses.

Specifically, the underpinning strategy is the investigation of a unique population—in this case,

freshman students or novice learners in programming courses. The purpose of combining these

features is to enable a better understanding of how LMS tools could effectively be used to

improve freshmen’s learning performance in said courses. This chapter will begin with the

theoretical foundation of the study, followed by a review of the previous literature and

identification of gaps, an analysis of the research methods used, and finishing with a synthesis of

the literature.

Theoretical Foundation

The usability has multi-dimension constructs that can be evaluated from various

perspectives. Moreover, the usability concept can be determined by the user experience (UX), the

products (systems), environments, and tasks. The examination of any software use’s effectiveness

is one of the usability perspectives. From this point, we can clearly understand that the usability

factor has a theoretical principle, based on the human-computer-interaction (HCI) field (Jeng,

2005). Usability, one of the dependent variables of this study, has been defined as the degree to

which the system can be used to achieve a specific goal by specific users with efficiency,

effectiveness, and satisfaction (Bevan, Carter, Earthy, Geis, & Harker, 2016).
 10
However, in this study, Shackel’s usability model (1991) is adapted to measure LMS

features’ usability and to review the identified literature. This model was developed by Brian

Shackel as a framework to enable a better understanding of usability evaluation and to measure

users’ tasks based on four criteria: learnability, flexibility, effectiveness, and user attitude (Joo,

Lin, & Lu, 2011). Because the aim of this study was to investigate the effectiveness of certain

LMS features’ usability among learners enrolled in programming courses, the survey design was

based on the components of Shackel’s usability model (1991) in the interests of examining the

effectiveness of VPL and discussion forums’ usability as LMS features in terms of the four

criteria mentioned. According to Preece (1993), usability can be measured and tested via several

different approaches and techniques, such as expert/heuristic, observation, survey evaluation, and

experimental.

Moreover, in previous studies, these four elements have been defined and classified in

various contexts, including that of virtual learning environment or LMS use. For example,

Koohang and Du Plessis (2004) defined the components of Shackel’s usability model (1991) in

their investigation, integrating usability methods and education to design an advanced LMS

framework that could help create desirable learning environments. In so doing, the researchers, as

in prior works, defined the four components as follows:

Effectiveness is defined by Koohang and Du Plessis (2004) as effective in an LMS when

learners use it to successfully help themselves and accurately complete all tasks they have been

assigned. Shackel (1991) defined it as the efficiency of tasks’ completion in terms of the number

of errors and the time taken. Nielsen (1993) described it by saying that the effectiveness of a

system is directly related to its output; if the effectiveness of the system is high, the productivity

will be high.
 11
Flexibility is defined by Koohang and Du Plessis (2004) as flexible in an LMS when it

can be adapted to reflect differences among, and revisions to, tasks and can help learners

familiarize themselves with these alterations. Shackel (1991) defined it as accepting the

adaptations and changes made to specific environments and tasks.

Learnability is defined by Koohang and Du Plessis (2004) as learnable in an LMS when

learners can understand its features quickly and use it to execute tasks easily. Shackel (1991)

defined it as the relationship between the frequency of system use and users’ training

performance. According to Nielsen (1993), learnability means that the system must be easy for

users to learn or use and they should find it easy to execute their tasks.

Attitude: Students’ attitude to an LMS is evidenced in their feedback on how it conforms

to their requirements, such as those of agreement, happiness, and enjoyment (Koohang & Du

Plessis, 2004). Shackle (1991) defined it as users’ feedback and opinions on their use of a system

in the aspects of personal effort, tiredness, and discomfort. According to Nielsen (1993), attitude

is defined as users’ satisfaction or feelings during, or after, using the system to complete a task.

Past Literature and Identification of Gaps

In previous studies, freshman-level students have been defined as “novice learners” or

“first-year college students” and then categorized according to their learning attitudes. For

instance, Perkins et al. (1989) classified novice learners into two main types of student based on

how they responded to difficult aspects of a course: “stopper students” and “mover students.”

The former cannot move on or continue in the learning process when they experience difficulty

learning content, whereas the latter can learn from instructor feedback and use it to further the

learning process (Perkins et al., 1989).

Moreover, according to Koorsse (2015), novice programmers require greater skills to use

programming languages and work in a programming environment. This is because the absence of
 12
such skills can affect both their understanding of coding concepts and their development of

programming skills. Novice programmers also need increased assistance to develop their ability

to execute programming tasks and to better understand the principles of programming concepts.

The literature indicates that programming courses offer the most significant challenge

facing novice learners enrolled in a computer science major. Such students may struggle to

abstract the main concepts of programming topics, which are themselves a prerequisite for

understanding program syntax and, ultimately, writing program code. Consequently, this

unfortunate situation has encouraged researchers worldwide to conduct additional investigations

with the aim of finding ways of reducing the difficulty inherent in programming courses. For

example, Alakeel (2015), based on curiosity about the factors causing difficulties over learning

computer programming in Saudi Arabia, designed a survey with which to examine learners’

feedback based on three main indices related to learning environments and teaching methods. The

researcher then distributed this survey randomly to students in three different universities in Saudi

Arabia (Alakeel, 2015). The results of the survey showed, first, that students spend too little time

practicing while learning programming, as well as too little time in laboratory sessions, and

second, a lack of class activities, such as assignments and exercises, during which students could

receive instructor feedback by 91% (Alakeel, 2015). However, the author’s objective was to

identify the factors causing the difficulty, not to find ways of decreasing it or to identify effective

learning tools with which to fill the gap between this difficulty and the learning process, such as

LMS features.

Analysis of the Research Methods Used

The literature shows that various authors have underlined the importance of researchers’

and educational institutions’ identification of an LMS learning method that can help learners

improve their progress. In particular, several studies have shown that implementation and
 13
integration of an LMS approach positively affects the learning process and learners’ performance

in higher education institutions around the world. For example, Davies and Graff (2005)

investigated whether LMS-environment interactions could aid students in the learning process.

These researchers examined 122 students enrolled at the University of Glamorgan, Wales, who

had used the Blackboard LMS in their courses for 12 months; the authors also sought to identify

LMS features’ use for level one students (Davies & Graff, 2005). To do so, they used the

Kruskal-Wallis test to correlate overall Blackboard use for each group of participants, using final

grades as a measuring instrument for a range of courses. Their findings revealed that those

students who failed their courses had used Blackboard the least and that those who passed with

high marks had used the application the most (Davies & Graff, 2005).

According to the literature, most studies have focused on students experienced in LMS use

and who are familiar with its functions. Notable among such articles is Holbl and Welzer’s (2010)

attempt to measure students’ feedback and communication habits using Moodle LMS. These

authors designed a survey in the form of a questionnaire for a sample of participants comprising

136 students from various courses. This survey targeted three main LMS functions: collaboration

features, communication features, and wiki use in the Moodle LMS (Holbl & Welzer, 2010).

Most participants were second- or third-year college students. Data analysis showed that, in

general, students were not keen on using the communication tools or other Moodle features, with

only 11% of them using discussion forums and 30% preferring to use the LMS communication

feature (Holbl & Welzer, 2010). The feedback received as part of this study revealed that most

students did not initially know how to use the LMS communication feature, with 70% preferring

to use regular email to connect with their instructors. Furthermore, 40% preferred to submit their

assignments in person or to send them by email (Holbl & Welzer, 2010). Notably, the results of
 14
this examination of students’ communication habits as regards the Moodle LMS failed to prove

the effectiveness of LMS usability among this sample of participants.

Most researchers have examined the usability of LMS features in general terms, with few

investigating the effectiveness of certain features that can help improve learners’ performance in

particularly challenging courses. In a general examination of LMS features, a study conducted by

Sahid, Santosa, Ferdiana, and Lukito (2016) was aimed at evaluating and measuring the LMS user

experience (UX) through application of a questionnaire tool. In this evaluation, LMS was

examined in several categories such as attractiveness, dependability, and novelty. In their

approach, the authors observed the LMS for two different semesters and applied two LMS types,

which they called LMS1 and LMS2.

The data were collected from 38 learners who had experience in LMS use from taking two

courses in an information technology program, which were a project management class and a risk

management class. The results showed that, first, the user perception of LMS 1 and LMS 2 varied

among the learners; and second, the quality level between the two LMS types was different. The

only weakness of this study was that the LMS’s features remained unknown during the LMS

evaluation phase. The comparison between LMS 1 and LMS 2 was, in general terms, to identify

which had a good level of quality.

Synthesis of the Literature

Previous studies indicate that the VPL is an essential pedagogical tool that may integrate

with the LMS to implement the practical activities in programming courses remotely. For

instance, Cavus, Uzunboylu, and Ibrahim (2006) carried out a pilot study at Near East University,

Turkey, during the 2004–2005 fall semester. These researchers sought to create a virtual learning

environment in which to teach programming languages using LMS features. An additional focus

involved use of the Moodle LMS as a collaborative tool in two programming courses (for the Java
 15
and Pascal programming languages) through its integration with other programming learning

tools such as GREWPtool (Cavus et al., 2006). This integration between virtual learning features,

dubbed the Near East University Virtual Learning Environment (NEU-VLE), was supported by

the Moodle LMS.

Cavus et al. (2006) divided the learners into two groups according to programming

language studied, hoping to discover whether the LMS features supported the objectives of these

two courses. Thereafter, the researchers used a questionnaire-based survey containing 5-point

Likert scale-type questions. In their analysis, they noted certain disadvantages of the NEU-VLE

system and recommended that LMS systems include live video sessions to promote instructor-

student and student-student interactions (Cavus et al., 2006). Importantly, participants reported

that, when using the LMS, instructors were unable to perceive, through the students’ body

language, whether they understood the subjects (Cavus et al., 2006). The authors also observed

that LMS use alone was not sufficient to deliver a realistic class environment or to facilitate

exchange of feedback between instructors and learners.

The researchers also noted that the students in the Pascal programming language group

were less satisfied with NEU-VLE than were their counterparts in the Java programming

language group (Cavus et al., 2006). The reason given was that, because the compiler features in

GREWPtool did not support Pascal syntax and code, students in this class could not run their

programming exercises in the same way as could the members of the Java group. Accordingly,

Cavus et al. (2006) concluded that, in general, an LMS integrated with other collaborative tools is

most likely to recreate the characteristics of a real classroom.

One of the most important functions of VPL is the automatic assessment function. This

feature helps students to improve their programming learning process by practicing their coding

exercise before receiving automatic feedback on the same, including grading system instructors’
 16
comments and their code evaluation. According to Prieto-Blazquez et al. (2009), the VPL

automatic assessment process is an essential resource in programming because it supports the

content’s accessibility and availability, allowing students to execute their code remotely, anytime

and anywhere. Moreover, it gives students feedback on their class programming activities, such as

exercises and exams, in real time, which lets them know whether their code execution was

successful.

The literature indicates that the use of LMS online discussion forums affords learners an

opportunity to enhance their learning process through collaborative classes’ discussion topics. A

study conducted by Cho and Tobias (2016) was aimed at examining the effect of said forums on

students’ learning process as measured by three factors: learners’ time on discussion forums,

community of inquiry, and user satisfaction. The authors argued for a lack of empirical studies

constituting investigations of the influence of the online discussion feature on learners’

educational achievement. In the methodology of this study, the researchers used the same course

for three consecutive semesters under three conditions. The first was that students take the course

without using the discussion forums; the second was that students use the forums without

instructor participation; and the third was that students use the forums with active instructor

participation. During this experiment, the Blackboard platform was used as the LMS with which

to deliver this course. The results showed that the condition of using the discussion forums with

active instructor participation yielded excellent results in three factors: the time spent on the LMS,

the course satisfaction, and students’ academic achievement.

Furthermore, an empirical study conducted by Powell, Wimmer, Kilgus, and Force (2017)

was aimed at investigating the effects of discussion forums in regards to learners’ achievement of

web-based assessments. In their approach, the authors applied the Aplia LMS to deliver

Business Administration courses at the University of Pennsylvania, USA, before dividing learners
 17
enrolled in business and technology courses into two groups (control/treatment). The results

showed that the overall grades in the treatment group were higher than those in the control group

and that the discussion forums motivated the learners in the treatment group to discuss the class

topics.

Summary

As shown by these studies, various researchers have investigated and measured the

effectiveness with which learners can use LMS features in a wide range of contexts and variables.

Some have divided novice learners facing a difficult course into two types, stoppers and movers,

with the former needing direct instructor support to continue the learning process effectively

(Perkins et al., 1989). The theoretical foundation of this study is based on Shackel’s usability

model (1991), which measures the LMS’s features’ usability as a framework to enable a better

understanding of usability evaluation via four criteria: learnability, flexibility, effectiveness, and

user attitude. A review of the literature indicated that programming courses pose the greatest

challenge for freshman-level students enrolled in a computer science major (Alakeel, 2015).

Although some studies have failed to demonstrate that the usability of specific LMS features

supports learners in this area, most have been focused on experienced students who had already

used LMSs in their learning environment (Sahid et al., 2016). Furthermore, the VPL is one of the

most important features that can be integrated with LMS functions to deliver programming

courses. It offers both a collaborative tool for programming activities and an automatic

assessment tool, which can enhance students’ learning process (Blazquez et al., 2009; Cavus et

al., 2006). In addition, the online discussion forums can augment said process by offering a

collaborative feature with which to discuss the class topics and share knowledge (Cho & Tobias,

2016; Powell et al., 2017).

 18

Chapter 3

Methodology

Research Approach

The methodology used in this study was a quantitative research-based approach to

examining the usability of an LMS for freshman-level students who employ VPL and discussion

forum features as well as these features’ impact on the learning outcomes of programming

courses. This involved the use of a quantitative quasi-experimental design including experimental

(treatment) and control groups of new starts enrolled in an introductory programming course. The

LMS’s features were considered as a substitute for treatment in this experiment between

participants in both groups who either used or did not use these particular features. At the end of

the experiment, participants completed a survey that included questions about the post-test and

Shackel’s usability model (1991); these items were used to measure the LMS features’ usability.

The students’ final grades were used to measure the effectiveness, in terms of learning

outcomes, of the use of certain LMS features between the two groups (Davies & Graff, 2005).

The overall course grade included class activities such as programming exercises, class topics

discussion, and the final exam. These activities were therefore identical for both groups.

Participants

This study targeted freshman-level students enrolled in an introductory programming

course. These students were divided into two groups based on their use of LMS features. Each

group included 30 participants. The treatment group used the LMS features selected for the study,

including the VPL and discussion forums, while the control group used basic LMS features such

as downloading class materials, sending messages and communications, and submitting exercise
 19
files. The same course content and subject matter was presented to both groups. Such an approach

allowed the evaluation of students’ level of effectiveness at using certain LMS features based on

participants’ learning performance outcomes.

Weekly Groups Assessment

The groups had engaged in the programming course’s activities, which would also be part

of their assessment. The participants in the treatment group had used both LMS features—VPL

and discussion forums—every week. They had used the former for practice and for submitting

their programming exercises and the latter to reply to the weekly class discussion topic (Figure 1).

Meanwhile, the control group had a traditional programming class style and the same weekly

assessment based on their programming exercises and participation in the class discussion topic.

However, class activities for both groups were graded based on the following criteria: The

total course score was 100%, which included the programming exercises with a weight of 35%,

and all participants had seven exercises, distributed over seven weeks, each of which was worth

5%. The class discussion topics’ responses were weighted at 35%; the seven class topics, also

distributed over seven weeks, were each worth 5%; and the final exam had a weight of 30%.
 20

Figure 1: Weekly Groups Assessment Plan

 21
LMS Environment Setup

Because the aim of this study was to investigate the effect of particular LMS features’

usability on freshman-level students’ learning outcomes in challenging courses. In one instance,

Mhashi and Alakeel (2013) conducted a case study in which they investigated the complications

facing students learning computer programming skills at the University of Tabuk, Saudi Arabia.

The results showed that such difficulties included issues such as the use of programming

languages. As such, the current study featured the use of an introductory programming course

taught in the computer science department of the University of Tabuk to examine certain LMS

features’ usability’s impact on freshman-level students’ learning outcomes. Typically, the

programming courses are offered in two different sections, one theoretical and the other

practical; for the purposes of this study, in the practical section, the instructor used the VPL to

deliver course content (Figures 2–3).

Figure 2: Types of Programming Classes

 22

Figure 3: Integration of VPL into LMS

According to previous studies, a VPL plays a key role in practical programming courses’

sections because it uses an automatic assessment system for programming activities in such

classes, as well as an automatic grading code for class assignments, which facilitates instructors’

timely evaluation of students’ work and offering of feedback (Prieto-Blazquez et al., 2009).

Put simply, the VPL offers two access levels, one for instructors and another for students.

Specifically, it allows instructors to design and administer programming class events by

evaluating learners’ programming activities, exercises, tests, or assignments. Instructors can also

give students feedback in the form of written comments and edited and graded code. They can

also see a list of timestamped student submissions and grade them accordingly (Figures 4–5).

Figure 4. VPL Instructors’ Dashboard View

 23
After an instructor has finished designing a class module, students can join the class using

their individual credentials. In each student’s profile, he or she can view the student VPL

dashboard, enabling tasks such as reviewing the instructor’s guidelines for each programming

task, submitting a work file, running the file via the VPL compiler, and waiting for the

instructor’s feedback (Figure 5).

Figure 5: VPL Students’ Dashboard View

However, the literature has shown that one of the most important features of an LMS is its

ability to facilitate online participation via discussion forums. According to Shaw (2012), this

feature allows the instructor to foster collaboration in class by posting a topic for learners’ review

and eventual discussion. Furthermore, said author classified the learners’ attitudes, based on their

participation level when using the discussion forums, as “replier user,” “asker user,” “no activity

user,” or “watcher user.” In this study, discussion forums were the second LMS feature used to

cover the theoretical topics in a programming course. The in-class topics were divided into eight

sections, and each week, the learners had a specific programming topic that was discussed

individually (Figure 6).

 24

Figure 6: Discussion Forums in LMS

Data Collection

In this study, two groups of participants were followed for about eight weeks, during

which time the participants in the experimental group used integrated LMS features (VPL and

discussion forums) in their programming class activities. The main goal of this phase was to

measure the research’s dependent variables, which were the usability of LMS features and

students’ learning outcomes. For the first dependent variable, learning outcome, the final

examination took place in the last week of the experiment, and the final grades for both groups

were then compared to measure the students’ rate of success (Chowdhry et al., 2014). In the

control group (which used LMS without any features), the only data considered were the final

grades. The overall grade data from both groups were then collated for comparison; the aim of

this phase was to assess the course outcomes by comparing participants’ final grades for this

programming course.

For the second dependent variable, usability of LMS features, the participants completed a

survey instrument to measure the components of Shackel’s usability model (1991) on a 5-point
 25
Likert-type scale (1 = strongly disagree to 5 = strongly agree). The survey was distributed via an

online form to all participants. The survey design was based on the components of Shackel’s

usability model (1991) in the interests of examining the usability of the VPL and discussion

forums features via the four constructs underpinning the model: effectiveness, learnability,

flexibility, and attitude.

Data Analysis

Pre-analysis data screening used the SPSS software package to eliminate outliers from the

data collected and to review said data for missing elements. Normality and linearity tests were

also performed to evaluate the necessity of data transformations. This study featured the use of

two types of analyses—descriptive analysis and t-test analysis—to analyze the data obtained from

the surveys, and the resulting scores could thereby identify any statistically significant difference

between these groups.

Resources

Although many resources were needed to conduct this study, specific hardware and

software were not required. Any common LMS platform would have sufficed, whether

Blackboard, Canvas, Moodle, or the like. Google survey forms were used to collect data, with the

link to the survey made available to all participants. A data analysis application, such as SPSS

Statistics or Smart PLS Statistics, was used to analyze the collected data. A VPL software

package, with functions tailored to meet the requirements of programming courses, was made

available to all students participating in this experiment.

The data used for this study were collected through an electronic survey administered to a

sample of freshman-level students in the computer science department at the University of Tabuk.

The final grade data for both groups of participants were required at the end of the experiment.
 26
Considering these requirements, certain procedures had to be completed and permissions obtained

before this study could proceed, chief among them the receipt of Institutional Review Board

(IRB) study approval from Nova University. Additionally, the Participating Institution Study

Approval form of the University of Tabuk had to be completed to apply for permission to conduct

this study and its associated experiment and to obtain survey data and final grades of students

from both groups of participants.

As noted, this study required the use of a data analysis application such as SPSS Statistics

or Smart PLS Statistics. Similarly, the VPL tool was required to be used by participants for two

months during the course of this experiment. Accordingly, the researcher needed to create a VPL

environment suitable for programming courses before making it available to all students

participating in the experiment. To make this possible, logistical support was required from the

computer science department of the University of Tabuk to identify a programming course

suitable for this experiment and to integrate the VPL and LMS platform that was used.
 27

Chapter 4

Results

Introduction

This chapter contains the results that were obtained from the survey, as well as the

students’ reports and the statistical analysis outcomes. In this study, the participants were

freshman students enrolled in an introductory programming language course (programming 101

C#) in the spring 2019 semester at the University of Tabuk. The study was conducted during eight

weeks of the spring 2019 semester, and 60 students from three sections of the programming 101

course participated in the experiment. This experiment examined the effectiveness of LMS

usability for freshman-level students who employ VPL and discussion forum features, as well as

its impact on students’ learning outcomes in programming courses. In a quasi-experimental

design, the 60 freshman students were divided into two groups; 30 students were in the control

group and 30 were in the treatment group. This chapter also addresses the results based on the

research hypotheses as follows:

1. There are significance differences in LMS usability between freshman students who are

using LMS features (VPL and discussion forums) in programming courses and those who

are not.

2. There are significance differences in learning achievement between freshman students

who are using LMS features (VPL and discussion forums) in programming courses and

those who are not.

This study applied a five-point Likert scale to measure participants’ responses regarding

effectiveness, learnability, flexibility, and attitude. The Likert scale points were assigned

correspondingly: Strongly Disagree = 1, Disagree = 2, Neutral = 3, Agree = 4, Strongly Agree =

 28
5. The participants’ responses for each question were calculated to determine the highest mean

for each response.

Pre-Analysis Data Screening

Pre-analysis and data screening identified one incomplete response that had been

removed from the experimental group. Between both groups, 59 freshman students completed the

experiment procedures: 29 students in the experimental group and 30 in the control group.

Normality and linearity tests indicated that the data were normal, with data transformation figures

were not necessary (Figure 7).

Figure 7: The Normality and Linearity Test Outputs

Demographic Data Analysis

This section discusses the demographic variables that were gathered from survey (Q1 -

Q5) for both groups. The demographic information is presented in Tables 1-5. Table 1 shows the

gender of the students who participated in the study. The data showed that in the treatment group,
 29
51.7% of the participants were male and 48.3% were female. In the control group, 96.7% of the

participants were male and 3.3% were female.

Table 1: Summary of Participants’ Gender (N=59)

Group Male # Male % Female # Female %

Treatment 15 51.7 14 48.3

Control 29 96.7 1 3.3

Total 59

Table 2 shows the education level of the students who participated in the study. The data

indicated that in the treatment group, 100% of the students were freshmen. On the other hand,

96.7% of the students in the control group were freshmen; only one participant was at the

sophomore education level.

Table 2: Summary of Participants’ Current Education Level (N=59)

Group Freshman # Freshman % Sophomore # Sophomore %

Treatment 29 100

Control 29 96.7 1 3.3

Total 59

In addition, Table 3 shows the participants’ level of programming knowledge. The

resulting data indicated that in the treatment group, 100% of the students were novices. In the

control group, 90% of the students were novices, and 10% had an intermediate level of

programming knowledge.
 30
Table 3: Summary of Participants’ Level of Programming Knowledge (N=59)
Group Novice # Novice % Intermediate # Intermediate %

Treatment 29 100

Control 27 90 3 10

Total 59

Participants were also asked about their learning style, and Table 4 shows the learning

style for the students who participated in this study. The results indicated that 29 of the

participants in the treatment group, or 100%, had an online learning style. In the control group, 30

of the participants, or 100%, reported a traditional learning style.

Table 4: Summary of Participants’ Learning Style (N=59)

Group VPL style # VPL style % Traditional style # Traditional style %

Treatment 29 100

Control 30 100

Total 59

Furthermore, in question 5, the participants were asked about their prior use of the LMS,

and Table 5 shows the results from both groups. In the treatment group, 29 participants, or 100%,

answered NO to question 5, saying that they had not used the LMS before. On the other hand, in

the control group, 27 participants answered NO, and only three of the participants selected YES.
 31
Table 5: Summary of Participants’ Prior LMS Utilization (N=59)
Group YES # YES % NO # NO %

Treatment 29 100

Control 3 10 27 90

Total 3 56

Data Analysis and Results

In this study, two types of analysis were conducted: descriptive analysis and independent

samples t-test. Descriptive analysis was used to analyze the data collected using survey questions

7-25 in order to address the first research hypothesis. The independent samples t-test was used to

determine the statistically significant difference in the data collected from the usability survey.

Furthermore, the independent samples t-test was used to address the second research hypotheses

in order to define the statistical differences in the overall scores resulting from both groups.

Research Hypothesis 1

Research hypothesis 1 stated: “There are significance differences in LMS usability

between freshman students are using LMS features (VPL and discussion forums) in programming

courses and those who are not.” The data were obtained for the purpose of considering the

usability of the LMS, which was measured based on the participants’ responses to the components

of Shackel’s (1991) usability model: effectiveness, learnability, flexibility, and attitude (Appendix

C). The effectiveness of LMS usability was gauged from the answers of the participants in both

groups to (Q7 -Q12). The participants answered (Q13 - Q17) to gauge the learnability of LMS

use, and (Q18 - Q20) measured the flexibility of LMS use among the participants. (Q21 - Q25)

gathered information about the attitudes of the participants toward LMS use.
 32
However, regarding the first hypothesis, the results of the independent samples t-test

shown in Table 6 and Figure 8 indicate that there were significant differences in usability means

in the treatment group’s responses (M = 4.25, SD = 0.25) when compared to the control group for

(M = 3.22, SD = 0.70) conditions, t(57) = 6.72, p < .0001

Table 6: Independent Sample t-Test Results of Usability (N=59)

Group N Mean Std. Deviation t-value p-value

Treatment 29 4.25 0 .25

6.72 < .0001

Control 30 3.22 0.70

Total 59

Table 7 shows the results of the descriptive statistics test for usability model constructs for

the treatment group responses: effectiveness (M = 4.24, SD =.262), learnability (M = 4.20, SD

=.329), flexibility (M = 4.21, SD =.482), and attitude (M = 4.35, SD =.425).

Table 7: Descriptive Statistics Results of Usability Model for Treatment Group (N=29)
Construct Minimum Maximum Mean Std. Deviation

Effectiveness 3.67 4.67 4.24 .262

Learnability 3.8 5.0 4.20 .329

Flexibility 3.33 5.0 4.21 .482

Attitude 3.5 5.0 4.35 .425

Table 8 shows the results of the descriptive statistics test for usability model constructs for

the control group survey feedback. The results indicate that there was a significant difference in

means between groups. The effectiveness response for the control group was less than the
 33
treatment group by (M = 3.25, SD =.742), and in the learnability response, it was less than the

treatment group by (M = 3.26, SD =.819). The flexibility response was less for the control group

than the treatment group by (M = 3.18, SD = 1.38), and the attitude response in the control group

less than the treatment group by (M = 3.20, SD =.856).

Table 8: Descriptive Statistics Results of Usability Model for Control Group (N=30)
Construct Minimum Maximum Mean Std. Deviation

Effectiveness 2.00 5.0 3.25 .742

Learnability 1.8 5.0 3.26 .819

Flexibility 1.00 5.0 3.18 1.38

Attitude 1.4 5.0 3.20 .856

Research Hypothesis 2

The purpose of this study was to examine the usability of the LMS for freshman-level

learners who employ VPL and discussion forum features, as well as the impact of these two

features on their learning outcomes in programming courses. For the dependent variable, the

learning outcomes, the participants’ final grades were measured to indicate learning outcomes for

the participants in the two groups (Davies & Graff, 2005). Also, the total overall rate was 100%,

which included seven programming exercises weighted 35%, seven class discussion topics

weighted 35%, and a final exam weighted 30%. The data related to these activities were collected

from both groups for seven weeks during the spring 2019 semester and the data from the final

exam were collected during the eighth week of the experiment.

Research hypothesis 2 stated: “There are significance differences in learning achievement

between freshman students who are using LMS features (VPL and discussion forums) in
 34
programming courses and those who are not.” For this research hypothesis, the independent

samples t-test results in Table 9 and Figure 8 indicate that there were significant differences in

overall score means, and the treatment group (M = 82.28 SD = 6.01) was greater than the control

group by (M = 60.17, SD = 3.34) conditions, t(57) = 18.31, p < .0001

Table 9: Independent Sample t-Test Results of Overall Scores (N=59)

Group N Mean Std. Deviation t-value p-value

Treatment 29 82.28 6 .01

6.72 < .0001

Control 30 60.17 3.34

Total 59

Figure 8: Independent Sample t-Test Results of Overall Scores Comparison

 35
Summary

This chapter provides the results that were obtained from the participants in the

experiment, all of whom are freshman students enrolled in (programming 101 C#) course in the

spring 2019 semester at the University of Tabuk, Saudi Arabia. The data included the study

survey responses, class activities reports, and statistical analysis outcomes with consideration for

the two study hypotheses that were presented in chapter three. In the survey demographic

questions, the participants were asked about their gender, their level of education, their level of

programming knowledge, their learning style, and their prior LMS experience.

The participants in both groups answered (Q7 - Q25) to indicate their responses to

Shackel’s (1991) usability model components: effectiveness, learnability, flexibility, and attitude.

After the data collection and the data pre-analysis had been performed, and normality and

linearity tests were performed on the data and the results showed that the collected data was

normal. Then, the descriptive analysis and independent samples t-test were performed to analyze

the data collected from the treatment and control groups. The analysis output showed that there

was a significant difference between the two groups in terms of LMS usability and learning

achievement.
 36

Chapter 5

Conclusions, Implications, Recommendations, and Summary

Conclusions

This chapter describes the conclusions, implications, and recommendations arising from

the results of this study, based on the research problem and the study’s goals. This study’s

limitations, and recommendations for future research, are also outlined. The chapter concludes

with a summary of the findings of this study.

The research problem selected for this study addressed a gap in the literature left by

previous studies’ focus on experienced LMS users, with little attention paid to newcomers (i.e.,

freshman-level university students). Furthermore, although previous researchers have identified

various ways of enhancing the effectiveness of LMS use, no consensus has yet been reached on

which most successfully improves learning outcomes for new learners enrolled in programming

courses.

The literature review revealed that numerous researchers have investigated and assessed

the effectiveness with which learners can use LMS features in a wide range of contexts,

considering many different variables. This research was grounded in Shackel’s usability model

(1991), which measures the usability of LMS features in terms of four criteria: learnability,

flexibility, effectiveness, and user attitude.

The goal of this study was to evaluate LMS usability for freshman-level students who

employ VPL and discussion forum features, as well as the effect on students’ learning outcomes

in programming courses. Accordingly, a quantitative research methodology in the form of an

 37
independent-samples t-test statistical analysis was used to address two research hypotheses listed

below:

1. There are significance differences in LMS usability between freshman students who

are using, or not using, LMS features (VPL and discussion forum) in programming

courses.

2. There are significance differences in learning achievement between freshman students

who are using, or not using, LMS features (VPL and discussion forum) in

programming courses.

The research experiment was performed over the course of eight weeks on freshman

students in a programming course at University of Tabuk, Saudi Arabia. For the LMS usability

data were obtained from the participants through a survey instrument. In addition, the data were

collected through participants’ class activities records in both groups. These activities were

designed to measure the participants’ learning achievements in both groups, which were

distributed based on the experimental period as follows: seven programming exercises, seven

class topics discussion, and final exam for all participants.

The first research question was amid to measure the first dependent variable, which was

the LMS usability, based on Shackel’s model elements including 19 survey questions. Questions

(Q7–Q12) measured the effectiveness of LMS usability for participants in treatment and control

group, whereas questions (Q13–Q17) measured the learnability with which participants learned to

use the LMS. Questions (Q18–Q20) measured the flexibility of LMS use, and questions (Q21–

Q25) measured the attitudes toward LMS use. In this study’s first key finding, the independent-

samples t-test showed significant differences in LMS usability among freshman students using

VPL and discussion forum capabilities in the treatment group (M = 4.25, SD = 0.25) conditions,

t(57) = 6.72, p < .0001, which supports research hypothesis 1. These results are consistent with
 38
the findings of Alhazbi (2016), who found significant differences in LMS usability, user attitude,

and satisfaction among learners using the LMS features in their programming courses.

For example, Alhazbi (2016) found that LMS use motivates learners in programming

course environments to collaborate with class activities and to increase their level of interaction

with their instructors. Furthermore, his study indicated that learners in programming courses are

more comfortable when using LMS discussion forums to participate in class discussion topics.

Moreover, the study’s results were consistent with the second finding of Prieto-Blazquez et al.

(2009), who recorded a significant difference among participants’ feedback regarding the

usability of VPL resources in programming courses. For example, LMS resources such as VPL

can support learners taking programming courses in executing their code remotely and allow them

to access their class content anytime and from anywhere. In addition, they found that LMS use

helps students receive instructors’ feedback on their programming work via an automatic

assessment feature in real time, allowing students to track the success of their programming

exercises.

The second research question aimed at measuring the second dependent variable—

students’ learning achievement. The measurement was based on participants’ class activity data,

including scores on programming exercises, discussion topics, and the final examination, for both

groups. The second key finding of the independent-samples t-test showed a significant difference

in learning achievement among freshman students who used VPL and discussion forum

capabilities in the treatment group (M = 82.28 SD = 6.01) conditions, t(57) = 18.31, p < .0001,

which supports research hypothesis 2. Furthermore, these results are consistent with those of

Alhazbi (2016), Chowdhry et al. (2014), and Prieto-Blazquez et al. (2009), who found that LMS

features can help learners improve their learning process. For example, Chowdhry et al. (2014)

found that participants with a high level of LMS accessibility to course content saw a positive
 39
impact on their learning achievement. In addition, Koorsse (2015) found that learners benefit in

their learning process when they use LMS features, as found in the present study and in support of

its second hypothesis. For instance, Koorsse discovered that learners in the treatment group who

used VPL in programming class had higher mean tests results than learners in the control group

when learning programming Looping concepts. Furthermore, he found that the students in the

treatment group who applied VPL in their class activities perceived and understood the

programing concepts more easily than those in the control group.

Study Limitations

The study’s main limitation was that it only examined freshman-level students in

programming courses and the impact of LMS features on LMS usability and learning

achievement. Other demographic variables, such as participants’ genders, were not included in the

study’s scope. Furthermore, according to the statistical descriptive analysis test results in Table 1,

only 3.3 % of participants in the control group were female, while 96.7% of the participants were

male. In addition, the data showed that in the treatment group, 48.3 % of participants in the were

female, and 51.7% of participants were male. Therefore, the difference in gender among the

participates in both groups might influence LMS usability in programming courses and learning

achievement. Furthermore, Table 10 showed the Chi-square test results, which indicated a

difference in the participants’ LMS usability based on their gender.

Table 10: Summary of Chi-Square Tests Results N (59)

Value df Asymp. Sig. (2-sided)

Pearson Chi-Square 6.225a 3 .101
Likelihood Ratio 7.702 3 .053
Linear-by-Linear Association 2.654 1 .103
N of Valid Cases 59
 40
Implications

These findings could help universities around the world develop and design LMS features

that will meet students’ needs, in particular by easing freshman-level learners’ transition into

programming courses. Likewise, ensuring the usability of LMS functions could help boost the

efficiency of the learning environments used in programming courses while improving outcomes

for computer science programs.

Moreover, the findings of this experiment add to the body of knowledge about LMS

research through an evaluation of the usability of users’ experience and LMS functions. The

resulting knowledge can provide a starting point for boosting the efficiency of learning

environments in difficult courses while improving the quality of education programs’ outcomes.

Recommendations for Future Research

The implications of this study’s findings for the suggested research areas form the basis of

certain recommendations. First, because this study examined LMS usability in freshman-level

students, a similar study should be performed on participants in programming courses at different

academic levels. Second, a further statistical test such as a Chi-squared test is needed to examine

any relationship between demographic variables and the level of LMS features’ usability among

learners in programming courses and the impact on their learning outcomes.

Third, because this study was conducted in Saudi Arabia, similar research should be

conducted in other developed countries, allowing comparison of outcomes. Fourth, because this

study used Shackel’s usability model (1991) to examine LMS features’ usability in terms of four

constructs—effectiveness, learnability, flexibility, and attitude—use of different usability

constructs could strengthen its findings. Fifth, because this study focused on two LMS features—

VPL and discussion forum—future studies should study different LMS functions.
 41
Summary

The objective of this study was to evaluate LMS usability for freshman-level learners who

use the VPL and discussion forum features, assessing the effect of these features on those

students’ learning outcomes in programming courses. The literature review revealed that the

effectiveness of LMS features had already been investigated in a wide range of contexts and

variables. Some researchers identified two categories of novice learners facing a difficult course:

stoppers and movers (Perkins et al., 1989). The theoretical foundation for this research, being

based on Shackel’s usability model (1991), measured LMS functions’ usability within that

framework, using four criteria: learnability, flexibility, effectiveness, and user attitude.

The literature identified programming courses as the greatest challenge facing freshman-

level learners in a computer science major (Alakeel, 2015). Furthermore, it revealed that some

previous studies have failed to establish whether the usability of specific LMS features can

support learners in a programming course, most having focused on experienced learners whose

learning style already incorporated LMS use.

This study’s quasi-experimental design encompassed two groups of freshman-level

students enrolled in an introductory programming course. Participants were observed for a half-

semester over the course of 8 weeks during the Spring 2019 semester at the University of Tabuk,

Saudi Arabia. Significant differences in LMS usability were seen between these groups, with

participants in the treatment group exhibiting greater learning than those in the control group.

Building on this research, subsequent studies should investigate the usability of additional LMS

features with a view to broadening understandings of this topic.

 42
Appendices

A. Invitation to Participate
 43
Appendix B

B. Informed Consent
44
 45
Appendix C

C. Study Participants’ Survey Instrument

46
47
48
49
50
 51

Appendix D

D. Participating Institution Study Approval

 52
Appendix E

E. IRB Approval Letter

 53
References

Alakeel, A. M. (2015). Investigating Difficulties of Learning Computer Programming in Saudi

Arabia. Universal Journal of Educational Research, 3(9), 567-577.

Alassaf, N., Harfoushi, O., Obiedat, R., & Hammouri, T. (2014). Learning management systems

and content management system: Definitions and characteristics. Life Science Journal,

11(12), 39–41.

Albalawi, A., & Badawi, M. (2008, November). Teachers’ perception of e-learning at the

University of Tabuk. In Proceedings of World Conference on E-Learning in Corporate,

Government, Healthcare, and Higher Education (pp. 2434–2448). Association for the

Advancement of Computing in Education (AACE).

Alabbadi, M. M. (2010, April). Learner’s acceptance based on Shackel’s usability model for

supplementary mobile learning of an English course. CSEDU, 1, 121–128.

Al-Busaidi, K. A., & Al-Shihi, H. (2010). Instructors’ acceptance of learning management

systems: A theoretical framework. Communications of the IBIMA, 2010.

Alhazbi, S. (2016). Active blended learning to improve students’ motivation in computer

programming courses: A case study. In Advances in Engineering Education in the Middle

East and North Africa (pp. 187-204). Springer International Publishing.

Al-Hamad, A. Q. (2016, February). Students’ perception of implementing a smart learning system

(SLS) based on Moodle at Fujairah College. In Remote Engineering and Virtual

Instrumentation (REV), 2016 13th International Conference on (pp. 315–318).

Alharbi, S., & Drew, S. (2014). Using the technology acceptance model in understanding

academics’ behavioural intention to use learning management systems. International

Journal of Advanced Computer Science and Applications (IJACSA), 5(1).

Arroway, P., Davenport, E., Xu, G., & Updegrove, D. (2010). EDUCAUSE core data service
 54
fiscal year 2009 summary report. Boulder, CO: EDUCAUSE.

Azevedo, R., & Feyzi-Behnagh, R. (2010). Dysregulated learning with advanced learning

technologies. In AAAI Fall Symposium: Cognitive and Metacognitive Educational

Systems.

Chaffar, S., & Frasson, C. (2004). Inducing optimal emotional state for learning in intelligent

tutoring systems. In International Conference on Intelligent Tutoring Systems (pp. 45–54).

Berlin: Springer, Heidelberg.

Chughtai, R., Zhang, S., & Craig, S. D. (2015). Usability evaluation of intelligent tutoring system:

ITS from a usability perspective. In Proceedings of the Human Factors and Ergonomics

Society Annual Meeting (Vol. 59, No. 1, pp. 367–371). Sage Publications: Los Angeles,

CA

Bevan, N., Carter, J., Earthy, J., Geis, T., & Harker, S. (2016, July). New ISO standards for

usability, usability reports and usability measures. In International Conference on

Human-Computer Interaction (pp. 268–278). Springer, Cham

Cavus, N., & Momani, A. a. M. (2009). Computer aided evaluation of learning management

systems. Procedia—Social and Behavioral Sciences, 1, 426–430.

Cavus, N., Uzunboylu, H., & Ibrahim, D. (2006). The effectiveness of using learning

management systems and collaborative tool in web-based teaching of programming

languages. Retrieved from ERIC database. (ED503541)

Cho, M. H., & Tobias, S. (2016). Should instructors require discussion in online courses? Effects

of online discussion on community of inquiry, learner time, satisfaction, and

achievement. The International Review of Research in Open and Distributed

Learning, 17(2).

Chowdhry, S., Sieler, K., & Alwis, L. (2014). A study of the impact of technology-enhanced
 55
learning on student academic performance. Journal of Perspectives in Applied Academic

Practice, 2(3).

Clapp, B., & Swenson, J. (2013). The collaborative classroom: New technology brings new

paradigm. Atlantic Marketing Journal, 2(3), 6.

Davies, J., & Graff, M. (2005). Performance in e-learning: Online participation and student

grades. British Journal of Educational Technology, 36(4), 657–663.

Davuluri, P., & Madadi, P. R. (2016). Moodle Java Autograder: Technical report. Retrieved from

http://opus.govst.edu/cgi/viewcontent.cgi?article= 1200& context =capstones

Esteves, M., Fonseca, B., Morgado, L., & Martins, P. (2009). Using second life for problem

based learning in computer science programming. Journal For Virtual Worlds

Research, 2(1).

Holbl, M., & Welzer, T. (2010). Students’ feedback and communication habits using

Moodle. Elektronika ir Elektrotechnika, 102(6), 63-66.

Horvat, A., Dobrota, M., Krsmanovic, M., & Cudanov, M. (2015). Student perception of Moodle

learning management system: A satisfaction and significance analysis. Interactive

Learning Environments, 23(4), 515–527.

Jeng, J. (2005). Usability assessment of academic digital libraries: Effectiveness, efficiency,

satisfaction, and learnability. Libri, 55(2–3), 96–121.

Joo, S., Lin, S., & Lu, K. (2011). A usability evaluation model for academic library websites:

Efficiency, effectiveness and learnability. Journal of Library and Information

Studies, 9(2), 11–26.

Kaunang, S. T. G., Paturusi, S. D., Usagawa, T., Mangindaan, G., Sambul, A., & Sugiarso, B.

(2016). Student perceptions of virtual programming lab on e-learning class at University

 56
of Sam Ratulangi. In Information & Communication Technology and Systems (ICTS),

2016 International Conference on (pp. 244–248). IEEE.

Koohang, A., & Du Plessis, J. (2004). Architecting usability properties in the e-learning

instructional design process. International Journal on ELearning, 3(3), 38.

Koorsse, M., Cilliers, C., & Calitz, A. (2015). Programming assistance tools to support the

learning of IT programming in South African secondary schools. Computers &

Education, 82, 162–178.

Lasrado, F. (2009). Attitudes towards e-learning: Exploratory evidence from UAE. Paper

presented at the 2nd Annual Forum on e-Learning Excellence in the Middle East, Dubai,

UAE.

Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., ... & Simon, B.

(2004, June). A multi-national study of reading and tracing skills in novice programmers.

In ACM SIGCSE Bulletin (Vol. 36, No. 4, pp. 119-150). ACM.

Luxton-Reilly, A., & Petersen, A. (2017, January). The compound nature of novice programming

assessments. In Proceedings of the Nineteenth Australasian Computing Education

Conference (pp. 26-35). ACM.

Mhashi, M. M., & Alakeel, A. L. I. M. (2013). Difficulties facing students in learning computer

programming skills at Tabuk University. In Proceedings of the 12th International

Conference on Education and Educational Technology (EDU’13), Iwate, Japan (pp. 15-

24).

Mijatovic, I., Cudanov, M., Jednak, S., & Kadijevich, D. M. (2013). How the usage of learning

management systems influences student achievement. Teaching in Higher

Education, 18(5), 506-517.

 57
Paulsen, M. F. (2002). Online education systems: Discussion and definition of terms. NKI

Distance Education.

Pereira, J. A., Pleguezuelos, E., Merí, A., Molina‐Ros, A., Molina‐Tomás, M. C., &

Masdeu, C. (2007). Effectiveness of using blended learning strategies for teaching

and learning human anatomy. Medical education, 41(2), 189-195.

Prieto-Blazquez, J., Herrera-Joancomartí, J., & Guerrero-Roldan, A. E. (2009). A virtual

laboratory structure for developing programming labs. International Journal of

Emerging Technologies in Learning (iJET), 4(2009), 47-52.

Perkins, D. N., Hancock, C., Hobbs, R., Martin, F., & Simmons, R. (1989). Conditions of learning

in novice programmers. Soloway and Spohrer, 1989, 261–279.

Preece, J. (Ed.) (1993). A guide to usability: Human factors in computing. Wokingham: Addison-

Wesley.

Powell, L. M., Wimmer, H., Kilgus, L., & Force, C. M. (2017). Impact of online discussions on

web-based assessments. International Journal of Distance Education Technologies

(IJDET), 15(4), 99–111.

Pozenel, M., Furst, L., & Mahnic, V. (2015, May). Introduction of the automated assessment of

homework assignments in a university-level programming course. In Information and

Communication Technology, Electronics and Microelectronics (MIPRO), 2015 38th

International Convention on (pp. 761–766). IEEE.

Priyanto, A. D. (2010). Facilitating language learning with LMS: A brief review on Blackboard

and Moodle. Retrieved from http://agusdepe.staff.uns.ac.id/2009/04/06/facilitating-

languagelearning-with-lms-a-brief-review-on-blackboard-and-moodle/

Rahman, K. A., Ghazali, S. A. M., & Ismail, M. N. (2010). The effectiveness of learning

management system (LMS): Case study at Open University Malaysia (OUM), Kota Bharu
 58
Campus. Journal of Emerging Trends in Computing and Information Sciences, 2(2), 73–

79.4

Ramos, J., Trenas, M. A., Gutierrez, E., & Romero, S. (2013). E‐assessment of Matlab

assignments in Moodle: Application to an introductory programming course for

engineers. Computer Applications in Engineering Education, 21(4), 728–736.

Sahid, D. S. S., Santosa, P. I., Ferdiana, R., & Lukito, E. N. (2016, August). Evaluation and

measurement of Learning Management System based on user experience. In Engineering

Seminar (InAES), International Annual (pp. 72–77). IEEE.

Shackel, B. (1991). Usability—Context, framework, design, and evaluation. In B. Shackel & S.

Richardson (Eds.), Human factors for informatics usability (pp. 21–38). Cambridge:

Cambridge University Press.

Shaw, R. S. (2012). A study of the relationships among learning styles, participation types, and

performance in programming language learning supported by online

forums. Computers & Education, 58(1), 111-120.