E-Learning Evaluation Using PIECES Framework at Mercubuana University
E-Learning Evaluation Using PIECES Framework at Mercubuana University
E-Learning Evaluation Using PIECES Framework at Mercubuana University
First Indonesian Communication Forum of Teacher Training and Education Faculty Leaders International Conference on Education 2017 (ICE 2017)
Abstract—Mercubuana University (UMB) is a private This E-Learning need to be evaluated, so with the
university that was getting Higher Education Accreditation evaluation can make E-Learning run better. Evaluation in thise
(AIPT) with an A in 2016. UMB has implemented E-learning study will use the framework PIECES (Performance,
since 2009 in all faculties. E-learning users, such as students and Information, Economic, Control. Service) and it is used to
professors, can use an E-Learning Mercubuana University for 24 construct the quistionare (qualitative and quantitative models)
(twenty-four) hours with a range of facilities as well as the use of and use Likert to make measurement of the value of each
Single Sign On for login. Facilities at E-Learning are the question. The questionare distributed to the odd semester in
Dashboard, Course Overview, Forum, Quiz, Assignment, and
2016-2017 to lectures and students of Mercubuana University.
others. In this research, the questionnaire was designed using the
framework of PIECES modification and distributed to 23 A. E-learning
lectures and 23 students related to college the first semester of
Development of computer technology, especially the
2016-2017 with a sample random sampling method. The results of
Implementation of online courses at the University Mercubuana
internet, is fast developing makes very possibility for education
agree in Performance, Information, Economic, Control, and or course to use the Internet. Roosenberg (2001), cited by
Service. As for the assessment of the implementation Efficiency to Khan, said: “Internet technologies have fundamentally altered
E-Learning is Strongly Agree. the technological and economic landscapes so radically that is
now possible to quantum leaps in the use of technology for
Keywords---PIECES, E-Learning, University Education, E- learning” [1]. There are numerous names for online learning
Learning Management, Moodle. activities, including e-learning, Web-based learning (WBL),
Web-based instruction (WBI), Web-based training (WBT),
I. INTRODUCTION Internet-based Training(IBT), distributed learning (DL),
Education using the internet, known as E-Learning become Advanced distributed learning(ADL), distance learning, online
one of the teaching and learning methods that implemented by learning (OL), off-site learning (m-learning), or nomadic
Mercubuana University since 2009. E-Learning applied learning, remote learning, a-learning and others”. Mention of
Learning Management System (LMS) Moodle web based E-Learning at the Mecubuana University is Kuliah Online.
application developed by Information Technology department.
The purpose of the use of the application is to increase the
quality of learning. The online application allows students to
learn without limits time and place.
E-Learning method has been applied to all faculties at the
Mecubuana University, namely the Faculty of Design and
Creative Arts, Faculty of Engineering, Faculty of Computer
Science, Faculty of Economics and Business, Faculty of
Communication, Faculty of Psychology, and Masters. There
are two types of the meetings of E-Learning, namely Face to
Face in The Class number 4 (four) meetings, midtest 1(one)
meeting, final exams number 1 (one) meeting and online
method using internet facilities number 8 (eight) meetings.
Students and teachers can use Mercubuana University’s E-
Learning for 24 hours with a range of facilities and the use of Fig. 1: Educational systems approach to integrated e-learning
Single Sign On of user and password. Facilities in E-Learning
are the Dashboard, Course Overview, Forum, Quiz, Assigment, Jochems refering Banathy 1996, “Integration of the
and others. different viewpoints is best guaranteed by using a systems
approach for educational change. The complete educational
system should be considered, at least from the viewpoints
mentioned, if change made to it. From this point if view the • Economic covers cost (unknown, untraceable, too high)
introduction of e-learning is not considered merely as an and profits (find new markets, improve old markets, and
addition to instruction, but as an innovation, an integral part of increase orders).
the educational cystem. Accordingly, a redesign of the
educational system is needed in orthe to arrice at added • Control involves security, either too much or too little.
'learning value' for instance in terms of higher order learning Inadequate security can lead to fraud, embezzlement,
outcomes. Thus organizational, pedagogical and technological and potential privacy violations.
aspects have to be managed in harmony in order to solve an • Efficiency considers time wasting activities, especially
educational problem adequately” [2]. those caused by redundancy.
There are differences in the role of students and teachers in • Service includes many aspects of accuracy as well as
teaching and learning traditional versus E-Learning according ease of use.
MC. Login and R. Oliver cited by Berenda following table
below [3]: II. METHODS
TABLE I. TEACHER-STUDENT ROLES IN TRADITIONAL VERSUS
This study uses PIECES Framework where the questions
ONLINE LEARNING ENVIRONMENTS were designed with quantitative and qualitative methods that
can be used in management education. Quantitative methods in
Learning
Type of Role Teacher Role Student role
Environment
the questioner using the measurement of Likert scale consisting
Traditional Manager, Listener, Objectivist, directed or of five (5) votes, ie Strongly Disagree (score 1), Disagree
role in face- expert, receiver, teacher centered, much (score 2), Fair (score 3), Agree (score 4), and Strongly Agree
to-face class diciplinenarian, novice, teacher scaffolding, use (score 5). Qualitative methods are expected to get an opinion
controller, passive suplantive strategies, from responden regrading problem and solutions that are
dispenser of learner, highly structured suggestion of respondent can be studied, so can be strategies
information, reliant on Constructivist, leaner
goal setter, time instructor for centered, emphasis on that are interactive and flexible.
keeper assessment collaborative learning, Based on the above, the design of the question in the
learning use of generative
Supportive Resource, Problem learning strategies, less research questioner that uses PIECES with modification, as
role in online coparticipant, solver, or adaptive teacher follows:
learning scaffolder, explorer, scaffolding, peer
colearner, researcher, scaffolding • P(erformance). Do you think that E-Learning can
moderator, collaborator, improve and maintan learning process and ensure
facilitator, goal setter, consistaency, completeness and quality?
coach, monitor, moderator,
advisor facilitator, • I(nformation). Do you think that E-Learning make
scaffolder, excellent information information communication
participant between university and student?
• E(conomic). Do you think that the use of E-Learning
B. PIECES can save costs?
Wetherbe and Vitalari proposed Framework PIECES cited
by [4], “PIECES is an acronym for performance, information, • C(ontrol). Whether using E-learning can improve the
economics, control, efficiency, and service and is used as accuracy, interaction and feedback in the online
checklist for identifing problems and determining priorities fro materials as well as increase the opportunities students
developing new information systems”. For further explanation can ask questions directly to the lecturer and vice versa?
is as follows [5]: • E(ffeciency). Does the existence of E-Learning can
• Performance, Performance is related to timeliness in increase the Efficiency to specific problems such as
terms of the system responding to user requests to time, location constraints, slow response and paperless?
provide current, up-to-date data. Performance also • S(ervice). Is the use of E-Learning outcomes is
extends to a consideration of the total amount of work becoming easier rather than face to face in the class?
that can be put through the system in a given time.
At each question there is a qualitative inquiry command
• Information related to several DQ dimensions. it which is optionally filling, are as follows: "Please give reasons
includes lack of necessary information, irrelevant why and if there are problems please give a solution!".
information, too mauch information, information that is
not formatted propperly, inaccurate information, Mercubuana University
information that is not timely, and information that is In December 2016, Mercubuana University established in
difficult or costly to produce. It also includes data 1985 in the region Kopertis III Indonesia in the region Kopertis
capture, data redundancy, data inconsistency due to III Indonesia, got the Accreditation of Higher Education
redudancy, and data that is not secure, not organized, (Akreditasi Institusi Perguruan Tinggi-AIPT) with score A [1].
not flexible and not accessible
Founder of the University Mercubuana is an entrepreneur
named H. Probosutedjo and has experience as a teacher in
369
Advances in Social Science, Education and Humanities Research, volume 174
Taman Siswa Siantar, North Sumatra. Date October 22, 1985, Any questions from the questioner use the LIKERT scoring
he established the University Mercubuana with faculties and method. Rensis Likert, a sociologist at the University of
departments as follows: Faculty of Engineering, Department of Michigan, was concerned with measuring psychological
Architecture and Civil Engineering Department; Faculty of attitudes in a scientific way. Specially, he sought to create a
Agriculture, Department of Social Economics of Agriculture reasonable measurement using a proper metric scale that would
(Agribusiness) and the Department of Agriculture (Agronomy); measure respondent’s attitudes (Uebersax, 2006). In this study
and the Faculty of Economics, Department of Management and refers to Tullis who suggests that a typical item in a Likert
Accounting. Currently, the development of the University scale is a statement to which respondents rate their level of
Mercubuana very rapidly, and the faculty and the department agreement. The statement may be positive or negative. Usually,
becomes more, namely: the Faculty of Design and Creative a five-point scale of agreement like the following is used [8]:
Arts, Faculty of Engineering, Faculty of Computer Science, 1). Strongly disagree, 2). Disagree, 3). Neither agree nor
Faculty of Economics and Business, Faculty of disagree, 4). Agree, 5). Strongly agree.
Communication, Faculty of Psychology, and Masters.
A. Performance
III. RESULT AND DISCUSSION Assessment of Performance in E-Learning is as follows of
Ref. [3] in his research titled "Performance in e-learning: some 223 people filling the questionnaire there are 12 people
online participation and student grades" has the following who Strongly Agree, Agree 91 people, Fair 91 people, Disagree
results: The findings revealed that greater online interaction did 25 people, and Strongly Disagree 4 people. The results of data
not lead to significantly higher performance for students processing can be seen in Fig. 2, as follows:
achieving passing grades; however, students who failed in their
courses tended to interact less frequently. Other factors that Summary Report for performance
may be salient in online interactions are discussed. Anderson-Darling Normality Test
A-Squared 14.50
P-Value <0.005
Ref. [6] in his research titled "An Evaluation of the Mean
StDev
3.3677
0.8216
Effectiveness of E-Learning, Mobile Learning, and Instructor- Variance
Skewness
0.6750
-0.380225
Kurtosis 0.154494
Led Training in Organizational Training and Development" N 223
Minimum 1.0000
that Learning effectiveness was measured by the change in 1st Quartile
Median
3.0000
3.0000
score between the pre- and post-assessment. An ANOVA was 3rd Quartile
Maximum
4.0000
5.0000
was significant. Learners were also surveyed to capture 95% Confidence Interval for Median
3.0000 4.0000
demographic information. This research showed no significant 95% Confidence Interval for StDev
0.7517 0.9058
Ref. [7] in their research titled "Evaluating Usability of E- While the results of the Likert scale in the Performance
Learning Systems in Universities" study affect e-learning ratings have results can be seen in Table II, as follows:
systems in Kenyan universities. The study had two-fold
objectives; determining status of e-learning platforms and TABLE II. LIKERT RESULT FOR PERFORMANCE
evaluating usability issues affecting e-learning adoption in
# Rate Total Weight Total Likert
Kenyan universities. The research took a case study of one of
Vote
the public universities which has implemented Moodle e- 1. Strongly Agree 12 5 60
learning system. The usability attributes evaluated were user- 2. Agree 91 4 364
friendliness, learnability, technological infrastructure and 3. Fair 91 3 273
policy. 4. Disagree 25 2 50
5. Strongly Disagree 4 1 4
In this study, all components of PIECES framework are Total Likert 751
used to evaluate E-Learning and the questionnaire was Max Likert = 223 * 5 1,115
designed using it and distributed to personal related to college Likert (in %)= Total Liker / Max Likert * 100 67.35
the first semester of 2016-2017 with a sample random sampling Statment (between 60-
method with a total of 223 personal, with details as follows: 79.99) Agree
some 23 professors and students of some 200 people. Gender
comparison of sources, namely: some 182 men and women The results of the assessment for Performance, respondents
some 41 people. The faculties participating filling said Agreed that E-Learning can improve and maintain the
questionnaire, Faculty of Economics and Business 27 people, learning processes ensure consistency, completeness and
the Faculty of Computer Science 136, Faculty of Engineering, quality of the Mercubuana University’s E-Learning.
some 59 people, and a Postgraduate number 1 person. The data While the results of respondents' opinions regarding
were processed using descriptive statistics with Minitab Performance is a learning process can be done in addition to
application.
370
Advances in Social Science, Education and Humanities Research, volume 174
on-campus or erect and to students who are also the status of student work on the forum and the task at late time. Most
workers was greatly aided by the E-Learning. While the respondents also wanted a video conference and chat module to
opinion of the lecturer is to some subjects that are practical and facilitate the learning process.
can not count on-the-Online because it resulted from Student
Performance decline so that better use of teaching and learning C. Economic
with face to face method or class. The things that are Assessment of Economic in E-Learning is as follows of
considered to reduce the value of Performance is the frequent 223 people filling the questionnaire there are 53 people who
access to the application servers are often problematic, Strongly Agree, agree 126 people, Fair 31 people, Disagree 10
especially Saturday and Sunday. people, and Strongly Disagree 3 people. The results of data
processing can be seen in Fig. 4, as follows:
B. Information
Summary Report for Economic
Assessment of Information in E-Learning is as follows of Anderson-Darling Normality Test
223 people filling the questionnaire there are 5 people who A-Squared
P-Value
18.50
<0.005
Median
TABLE IV. LIKERT RESULT FOR INFORMATION
3.00 3.05 3.10 3.15 3.20
# Rate Total Weight Total Likert
Fig. 3: Summary Report for Information Vote
1. Strongly 53 5 265
Agree
While the results of the Likert scale in the Information
2. Agree 126 4 504
ratings have results can be seen in Table III, as follows: 3. Fair 31 3 93
4. Disagree 10 2 20
TABLE III. LIKERT RESULT FOR INFORMATION 5. Strongly 3 1 3
# Rate Total Weight Total Likert Disagree
Vote Total Likert 885
1. Strongly Agree 5 5 25 Max Likert = 223 * 5 1,115
2. Agree 473 292 Likert (in %)= Total Liker / Max Likert * 79.37
3. Fair 390 270 100
4. Disagree 226 92 Statment (between 60-79.99) Agree
5. Strongly Disagree 1 9 9 The results of the assessment for Economic, respondents
Total Likert 688 said Agreed that E-Learning can improve and maintain the
Max Likert = 223 * 5 1,115 learning processes to ensure consistency, completeness, and
Likert (in %)= Total Liker / Max Likert * 100 61.70 quality of the Mercubuana University’s E-Learning.
Statment (between 60-
79.99) Agree It can reduce the cost because there is need to come to the
campus, so that not only the costs of the fare as well in the
The results of the assessment for Information, respondents terms of dining. Since it not necessary to the campus,
said Agreed that E-Learning can improve and maintain the respondents can do the work everywhere and every time.
learning processes an ensure consistency, completeness, and Respondents argued that the work could be hampered if the
quality of the Mercubuana University’s E-Learning. internet quotas are insufficient. Moreover, in places or areas
that have limited internet.
Mercubuana University’s E-Learning application has a
complete, but the learning process becomes ineffective when 1) Control
lecture and students are not active in progress, in particular on Assessment of Control in E-Learning is as follows of 223
the forum. Lectures often only provide information at the people filling the questionnaire there are 4 people who Strongly
beginning of the end of the last closing time. Similarly, the Agree, Agree74 people, Fair 89 people, disagree 48 people, and
371
Advances in Social Science, Education and Humanities Research, volume 174
Strongly Disagree 8 people. The results of data processing can Summary Report for Eficiency
be seen in Fig. 5, as follows: Anderson-Darling Normality Test
A-Squared 19.66
P-Value <0.005
Mean 4.0404
Summary Report for Control StDev 0.7493
Variance 0.5614
Anderson-Darling Normality Test Skewness -0.90875
A-Squared 13.00 Kurtosis 1.97101
P-Value <0.005 N 223
Mean 3.0807 Minimum 1.0000
StDev 0.8713 1st Quartile 4.0000
Variance 0.7592 Median 4.0000
Skewness -0.322456 3rd Quartile 5.0000
Kurtosis -0.451259 Maximum 5.0000
N 223
95% Confidence Interval for Mean
Minimum 1.0000 3.9415 4.1392
1 2 3 4 5
1st Quartile 2.0000
95% Confidence Interval for Median
Median 3.0000
3rd Quartile 4.0000 4.0000 4.0000
Maximum 5.0000 95% Confidence Interval for StDev
95% Confidence Interval for Mean 0.6856 0.8261
1 2 3 4 5 2.9657 3.1957
95% Confidence Interval for Median
3.0000 3.0000
95% Confidence Intervals
95% Confidence Interval for StDev
0.7973 0.9607 Mean
Median
95% Confidence Intervals
3.95 4.00 4.05 4.10 4.15
Mean
Median
Fig. 6: Summary Report for Efficiency
3.00 3.05 3.10 3.15 3.20
Fig. 5: Summary Report for Control While the results of the Likert scale in the Efficiency
ratings have results can be seen in Table VI, as follows:
While the results of the Likert scale in the Control ratings
TABLE VI. LIKERT RESULT FOR EFFICIENCY
have results can be seen in Table V, as follows:
# Rate Total Weight Total Likert
TABLE V. LIKERT RESULT FOR CONTROL Vote
1. Strongly Agree 5 56 280
# Rate Total Weight Total Likert 2. Agree 4 129 516
Vote 3. Fair 3 31 93
1. Strongly Agree 5 4 20 4. Disagree 2 5 10
2. Agree 74
4 296 5. Strongly Disagree 1 2 2
3. Fair 89
3 267 Total Likert 901
4. Disagree 48
2 96 Max Likert = 223 * 5 1,115
5. Strongly Disagree 1 8 8 Likert (in %)= Total Liker / Max Likert * 100 80.81
Total Likert 687 Statment (between 80-100)
Max Likert = 223 * 5 1,115 Strongly Agree
Likert (in %)= Total Liker / Max Likert * 100 61.61
Statment (between 60-
79.99) Agree Based on table 6, the results of Likert that respondents
Strongly Agree with the E-Learning can increase the efficiency
The results of the assessment for Control, respondents said of the organization to a specific problem such as time, location
Agreed that E-Learning can improve and maintain the learning constraints, slow response and the use of paper.
processes an ensure consistency, completeness and quality of 3) Service
the Mercubuana University’s E-Learning. Service assessment in E-Learning is as follows of 223
From the table 5, showed that the respondents agree to use people filling the questionnaire there are 9 people who Strongly
E-Learning can improve, interaction, and feedback in the Agree, Agree 74 people, Fair 79 people, Disagree 52 people,
online materials as well as increase the opportunities to student and Strongly Disagree 9 people. The results of data processing
to ask questions directly to the lecturer and vice versa. The can be seen in Fig. 7, as follows:
Application is considered safe to support the learning process. Summary Report for Services
Anderson-Darling Normality Test
A-Squared 11.17
P-Value <0.005
2) Efficiency Mean 3.0987
StDev 0.9392
Assessment of Efficiency in E-Learning is as follows of Variance
Skewness
0.8821
-0.198450
223 people filling the questionnaire there are 56 people who Kurtosis
N
-0.561950
223
Mean
Median
372
Advances in Social Science, Education and Humanities Research, volume 174
While the results of the Likert scale in the Service ratings IV.CONCLUSION
have results can be seen in Table VII, as follows: Implementation of e-Learning of the University
Mercubuana has value in performance, Information, Economic,
TABLE VII. LIKERT RESULT FOR SERVICE
Control, and Service. With these results, the University must
# Rate Total Weight Total Likert increase the results on these points in order to have a Strongly
Vote Agree value. As for the assessment of the implementation,
1. Strongly Agree 5 9 45 Efficiency Online Learning is Strongly Agree and must be
2. Agree 4 74 296
3. Fair 3 79 237
maintained.
4. Disagree 2 52 104
5. Strongly Disagree 1 9 9
REFERENCES
Total Likert 691 [1] Khan. BH. 2005. Managing E-learning: Design, Delivery,
Max Likert = 223 * 5 1,115 Implementation, and Evaluation. George Washington-USA: Infosci.
Likert (in %)= Total Liker / Max Likert * 100 61.97 [2] Jochems, W. et. All. 2004. Integrated E-learning: Implications for
Statment (between 60- Pedagogy, Technology and Organization. New York: USA.
79.99) Agree RutledgeFlamer.
[3] Davis, J. and Graff, M. 2005. An Evaluation of the Effectiveness of E-
The results of the assessment for Service, respondents said Learning, Mobile Learning, and Instructor-Led Training in
Organizational Training and Development. United Kingdom: British
Agreed that E-Learning can improve and maintain the learning Journal of Educational Technology Vol 36.
processes an ensure consistency, completeness and quality of [4] Fisher, C. et all. 2011. Introduction to Information Quality.
the Mercubuana University’s E-Learning. Bloomington-USA: Author House.
4) PIECES [5] Mercubuana. Sejarah Mercubuana. http://mercubuana.ac.id/id/tentang-
umb/sejarah-umb. [Date acessed, 6th January 2017].
The overall results of the calculations obtained from
[6] V. Paul, T. An Evaluation of the Effectiveness of E-Learning, Mobile
respondents, it can be seen Efficiency has the highest order Learning, and Instructor-Led Training in Organizational Training and
using the calculation method Likert score Strongly Agree, Development. Taipei-Taiwan: British Journal of Educational
while the rest get ratings Agree, namely Economic (79.37%), Technology Vol 36 No 4.
Performance (67%), Information (71.7%), and Control [7] Kipkurui, N. Evaluating Usability of E-Learning Systems in
(61.61%). While the Mean value of each point is Performance Universities. Bradford: United Kingdom. IJACSA.
3.3677, Information 3.0852, Economic 3.9686, 3.0807 Control, [8] Tullis, W. and Albert W. 2008. Measuring The User Experience-
Efficiency 4.0404 and 3.0987 Service. And the overall average Collecting, Analyzing, and Presenting Usability Metrics. Burlingron:
Elsevier.
is 3.4402.
Interval Plot of performance, Information, Economic, Control, ...
95% CI for the Mean
4
Data
373