SPECIAL SECTION ON DATA MINING FOR INTERNET OF THINGS

Received January 14, 2020, accepted February 24, 2020, date of publication March 2, 2020, date of current version March 16, 2020.
Digital Object Identifier 10.1109/ACCESS.2020.2977679

Design and Evaluation of a Deep Learning

Recommendation Based Augmented Reality
System for Teaching Programming and
Computational Thinking
PEI-HSUAN LIN 1 AND SHIH-YEH CHEN 2
1 Department of Engineering Science, National Cheng Kung University, Tainan 701, Taiwan
2 Department of Computer Science and Information Engineering, National Taitung University, Taitung 950, Taiwan

Corresponding author: Shih-Yeh Chen ([email protected])

This work was supported in part by the Ministry of Science and Technology of the Republic of China, Taiwan, under Contract MOST
108-2511-H-143-002.

ABSTRACT Programming is considered a skill to arouse and inspire learner’s potential. Learning to program
is a complex process that requires students to write grammar and instructions. The structure of a programming
language does not cause impose problems to students, the real obstacle is how to apply these learned
grammars and present them in a complete and correct program code for problem solving. In this study,
a deep learning recommendation system was developed, which includes augmented reality (AR) technology,
and learning theory, and is provided for study by students in non-major and also from different learning
backgrounds. Those students divided into two groups, the students participating in the experimental group
were using the AR system with deep learning recommendation and the students participating in the control
group were using the AR system without deep learning recommendation. The results show that students in
experimental group perform better than the control group with regards to learning achievement. On the other
hand, in the part of computational thinking ability, students using a deep learning recommendation based
AR system is significantly better than those using non-deep learning recommendation based AR system.
Among the various dimensions of computational thinking, creativity, logical computing, critical thinking,
and problem-solving skills are significantly different among the two groups of students. The students in
experimental group perform better than the control group with regards to the dimensions of computational
thinking, creativity, logical computing, critical thinking, and problem-solving skills.

INDEX TERMS Deep learning recommendation, computational thinking, AR technology.

I. INTRODUCTION which leads to challenges and obstacles for students to solve

Due to the widespread applications of computers, people are real problems in programming in the process of programming
paying more attention to computer science education, and the compilation [2], [3]. The learning process of programming is
skills and knowledge related to programming have attracted a complex process that requires students to write grammar
attention and become an indispensable part of education. and instructions. The structure of the programming language
Most teachers teach unilaterally, and even when they teach does not cause learning difficulties for students, the real
programming. A programming class is usually taught by problem is how to apply these learned grammars and present
lecturing and demonstrations of fixed and static program them in a complete and correct program code [4], [5]. In order
codes. Most students think that program compilation is a to enable students to write appropriate and correct program
difficult area to understand, they cannot understand the actual code during the compilation process, this study uses aug-
situation of the programming compilation during the learning mented reality (AR) features to dynamically overlap digital
process, and do not know how to apply implementation, material with the real environment, which provides students
with a context-aware learning environment for writing and
The associate editor coordinating the review of this manuscript and compiling program codes in a problem-solving manner. AR
approving it for publication was Mu-Yen Chen . technology is now widely used in learning environments

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 8, 2020 45689
 P.-H. Lin, S.-Y. Chen: Design and Evaluation of a Deep Learning Recommendation Based AR System

and is considered to be an educational tool with positive • Are there differences in learning achievement between
impact [6], [7]. AR can provide students with a context-aware students who learn by AR system with deep learning
environment between reality and virtual, and can enhance recommendation and AR system without deep learning
students’ immersive experience in the learning process [8]. recommendation?
Students can interact with the virtual world through practical • Are there differences in computational thinking between
operations to achieve learning results [9]. However, in an students who learn by AR system with deep learning
AR learning environment, which provides a large amount of recommendation and AR system without deep learning
knowledge content, it is still necessary to effectively help recommendation?
students master the tools of learning focus [10], [11].
How to predict the difficulties students encounter in the II. RELATED RESEARCH
process of learning, as well as after they learn, is a con- A. AR IN EDUCATION
cern of teachers and parents. In the past, students’ academic AR is a technology that superimposes dynamic digital con-
performance or classroom assessments were used to judge tent in a real-world environment, thus, providing users
students’ understanding of learning content. There are also with a realistic and immersive perspective [11], [22]–[24],
many different data mining techniques applied to predict stu- while also practicing the visualization of ‘‘invisible’’ con-
dent performance, such as Naive Bayes, Decision tree, Sup- cepts or objects, meaning that in a 3D virtual environment,
port Vector Machine, K-Nearest Neighbor, etc [12]. The use AR technology can combine real and virtual objects to pro-
of artificial intelligence and deep learning has increased in vide users with immersive and interactive experiences [6],
recent years. Deep learning, which is an emerging technology [25]. In recent years, many educational researchers have
in educational data mining, is a type of neural network, and applied AR technology to training and education, and their
is regarded as one of the best predictors of student learning findings indicate that AR technology can allow students to
behavior, as it that can detect correlations with predictive engage in authentic learning activities and explore real envi-
variables. Even if there is a complex nonlinear relationship ronments [26]–[28]. Through AR technology, students can
between dependent variables and independent variables, deep interact with virtual objects from different perspectives and
learning can still make predictions [13], [14]. Fok et al. [15] enhance their visual perception, which can help students to
used the Google Tensorflow Deep Learning analytic engine to improve their learning and understanding, and also attract
predict the development of students after study according to students to explore and investigate problems in the real world
their academic performance, such as the results of the exam- [25], [29] Dunleavy et al. [8] pointed out that if an AR
inations of various subjects: Chinese, English, history, and learning environment is carefully designed, it can increase
physics, and their non-study performance, such as learning students’ situational awareness and experience in an immer-
behavior, sports, arts, learning services, etc. Therefore, this sive learning environment. There have been many studies
study records students’ operational processes and learning exploring AR learning systems related to learning activity
behaviors for analysis, in order to recommend the grammar design, such as: Di Serio et al. [24] applied AR technol-
and unit where the students need remedial teaching, and ogy to the visual arts curriculum of middle school student.
provide students with a personalized learning way. Moreover, Wang et al. [30] used AR simulation systems for collabora-
ability assessment after learning is an important step in the tive inquiry learning activities, and then, for comparison with
process of promoting computer science education [16], [17] traditional teaching methods; Huang and Lin [28] applied
Chen et al. [17] mentioned that programming courses are fast AR in the primary school astrology curriculum to present
and diverse learning areas, thus, if students are only required the movements of the stars through virtual objects, and then,
to complete a single instruction, it is not a completely success- to explore the learning outcomes and flow experience of
ful study. The focus of truly successful learning is to allow students in the learning activities; Lin et al. [31] use the
students to gain different skills and mindsets to facilitate their digital picture books combined with AR technology to allow
learning and problem solving in the future. Computational students to understand the four stages of insect life cycle;
thinking is considered to be a very important core competence and explore students’ imagination and learning motivation.
to help students learning and thinking in the future [18]–[20]. Therefore, this study applies AR in a programming course,
How to help those students of different learning areas and in order that students can see the objectives of the project
ages inspire computational thinking skills is also a topic that to be completed in the real environment, and complete the
is often discussed [21]. learning content through the application and editing of the
Based on the above, this study proposes an image-based learned program syntax.
programming learning system, which includes AR technol-
ogy, deep learning with recommendations, and learning the- B. DEEP LEARNING
ory, and is provided for study by students in different learning Artificial intelligence and big data are the trend of research
areas, in order that they can overcome the learning difficulties topics in various/different fields, these technologies sup-
of being non-major undergraduates, and then, explores the port predicting, problem-solving and decision-making [32].
differences of computational thinking abilities. The research Deep learning, which is a process where multi-level neural
questions are as follows: networks perform specific tasks, is a branch of machine

45690 VOLUME 8, 2020

P.-H. Lin, S.-Y. Chen: Design and Evaluation of a Deep Learning Recommendation Based AR System

learning and a part of AI [33]. Through a series of logics science. Wing [48] also proposed four main dimensions of
from a large number of examples and data, multiple processes computational thinking ability: (1)Decomposition; (2)Pat-
have been nonlinearly transformed to provide results that tern Recognition; (3)Pattern Generalization and Abstraction;
are sufficient to represent data characteristics and features (4)Algorithm Design; and with training of those four abilities,
without specific rules [34]. Deep learning is known as one of students can effectively develop their thinking skills in a
illustrious technology for analysis, classification and predic- planned manner. In the past, many scholars and experts in
tions [35]–[38]. Deep learning is a kind of mechine learning related fields applied a wide range of computational think-
technology and it could be semi-supervised, unsupervised or ing;therefore, there are different definitions in this import
supervised [35]–[37]. In the other words, deep learning model process [49]–[53]. For example, Cuny et al. [49] proposed
can builded from learning experience and with minimal exter- that computational thinking is a thinking process that involves
nal interference [39]. Peters [40] presented the trends of deep computation and problem solving, and can present effective
learning: (1) deep learning has a long and rich history, and solutions by means of information; Barr and Stephenson [52]
has been discussed more and more in recent years. There are put forward a structured model of the core concepts and capa-
also many different opinions reflected in the definitions of bilities of computational thinking, such as data collection,
nouns; (2) deep learning is considered useful and usable due data analysis, data presentation, problem decomposition, etc.;
to its extensive use; (3) with the advancement of software Aho [50] simplified the definition of computational thinking
and hardware, the scope of deep learning model application into a method to solve problems through computational steps
is expanding; and (4) with the evolution of time, deep learn- and algorithms; Sysło and Kwiatkowska [53] emphasized that
ing can address more and more complex applications. Deep computational thinking is a series of thinking skills, and not
learning can be said to be a collection of many emerging tech- just the result of computer compilation; García-Peñalvo [51]
nologies, and there are many breakthrough developments and suggested that computational thinking is a high-level abstract
applications in different fields, as compared to other existing and computational approach to solving problems.
machine learning algorithms. For example, Baldi et al. [41] In simple terms, computational thinking is the way of
used deep learning in the field of high-energy physics to thinking and practicing computing, as well as a way to
solve the problem of signal-versus-background classification; positively solve problems; however, it is not mandatory to
LeCun et al. [36] proposed deep learning to identify abstract use technology to solve problems, instead, it guides stu-
data processing, and achieved flexibility and high accuracy dents to solve problems with the concept of technology [54].
for speech and image recognition; Gulshan et al. [34] applied The topic of computational thinking is considered in
deep learning to medicine for automatic detection of retinopa- the field of research and in the application of educa-
thy; Day and Lin [42] employed deep learning to emotional tion [55]. Many experts in the education field empha-
analysis to evaluate smartphone user reviews and find user sized that computational thinking is an important skill
opinions by using emotional dictionaries; Goh et al. [43] in the field of education and technology in the 21st
considered deep learning as the most valuable tool in the century [56]. Computational thinking has been applied
application of computational chemistry; Esteva et al. [44] to education, such as Chen et al. [17], who explored the
used deep learning to analyze image data of skin disorders; changes in students’ inspirations regarding challenges that
Wu et al. [45] translated handwritten text content through focused on potential and computational thinking in a
deep learning. Huang et al. [32] apply deep learning to robot coding course; Tsai and Tsai [57] explored externally-
massive open online courses (MOOCS), due to the main facilitated regulated learning in a quasi-experimental manner,
method of MOOCS is watching vedio, they analysis student’s and created a blended learning environment of computa-
learning log to predict student are able to respond specific tional thinking for the improvement of students’ computer
difficultly questions and recognition the degree of question skills; Wu et al. [58] analyzed and discussed computational
correctly. Xing and Du [46] use deep learning mechanism thinking ability according to the co-compiling learning
to predict individual student who discontinue their studies, method.
provide the method to avoid the high risk of student dropout. The abovementioned shows the importance of computa-
Most such researches applied deep learning to the analysis of tional thinking ability. This study aims to develop a program
data results. This study combines the characteristics of deep compiling learning system for different learning fields and
learning with an AR system for real-time application at an non-major undergraduates, where the purpose is to inspire
education site, offers students immediate learning feedback, students’ computational thinking ability and explore the dif-
and provides relevant learning tasks to help clarify the dif- ferences between those non-major undergraduates.
ficulties and doubts of the students in learning the program
language. III. METHOD
In order to evaluate the impact of innovative learning methods
C. COMPUTATIONAL THINKING on students, this study conducted an experiment to explore
Computational thinking was proposed by Wing [47], which the differences between students’ learning outcomes and
aims to solve problems, design systems, and understand computational thinking according to different learning strate-
human behavior through the basic concepts of calculator gies. The course is named the ‘‘Program Logic Thinking

VOLUME 8, 2020 45691

 P.-H. Lin, S.-Y. Chen: Design and Evaluation of a Deep Learning Recommendation Based AR System

FIGURE 1. The structure of the deep learning recommendation based AR system.

Education’’ course in the university general education cur- which can more accurately learn the feature of users and items
riculum, and the purpose of this course is to provide students to achieve supervised learning path recommendations. As for
with basic programming concepts and to develop students’ the back-end development, Firebase is selected, which is an
computational thinking skills in the process of program com- app development platform that supports Android, iOS and
pilation. website, to help app developers for quickly building back-end
services in the cloud (Show as Figure 1).
A. PARTICIPANTS The AR system combined AI with deep learning tech-
The participants of this study are 97 students from a university nology to determine its relevance through the compilation
in eastern Taiwan, who were assigned to an experimental and operation process of students, and recommended differ-
group and a control group. The students participating in this ent learning tasks for students who were confused or found
experiment were not students in the information technology the learning process incomprehensible, in order to improve
related department. There were 48 college students in the students’ logic and application of related programming lan-
experimental group, including 23 students from the Depart- guages. Figure 2 shows the AR system architecture, which
ment of Arts and 25 students from the Department of Music; is mainly composed of an AR learning system module and a
the control group consisted of 49 college students, with personalized learning module. The AR learning system mod-
24 students from the Department of Chinese and 25 students ule includes: (1) Deep Learning recommendation function,
from the Department of Public Administration. This study where the system recommendation strengthens the program
used the convenience sampling method, the average age of syntax and logic related learning tasks, and provides more
students was 20 years old, and one teacher taught the same practice opportunities for students according to their learn-
course content to all students. ing process; (2) AR object control, which includes learning
tasks and learning materials to present material management;
B. DEVELOPMENT OF DEEP LEARNING (3) learning mission and materials, which provide different
RECOMMENDATION BASED AR SYSTEM AR learning materials and tasks for students to query and
In this study, Unity 3d is a cross-platform game engine that is practice according to the compiled learning units of different
used for developping the AR funcation. Regarding the recom- programming languages. The personalized learning module
mendation function, it built on the Unity Machine Learning includes: (1) learning process, which records the student’s
Agents based on tensorflow framework to allow Unity scripts learning and operation process; (2) learning achievement,
to receive data from python scripts. The recurrent neural where the teacher gives the student’s grade according to the
network (RNN) is adpot to model dynamic and sequence data, completion degree of the student’s learning task; and (3) code
45692 VOLUME 8, 2020
P.-H. Lin, S.-Y. Chen: Design and Evaluation of a Deep Learning Recommendation Based AR System

FIGURE 2. The structure of the deep learning recommendation based AR system.

hints, where relevant prompt content is provided to the stu- All participating students used their mobile devices in
dents according to the task. their learning activities for AR presentation of program lan-
The system is divided into two parts, the teacher part guage learning content and learning tasks. Figure 4 shows
and the student part, each part with four main functions. the operation screen of students in the learning task. Students
Figure 3 shows the functions of AR system. The teacher must be individually logged into the learning system, which
part includes (1) Account: Teacher can add, modify and provides the related learning tasks for different learning units,
delete student account authority for participating in the as presented by AR technology. The 3D dynamic objects
programming language course which also access student’s were combined with the real environment, where students
major, gender and other related information; (2) Material could disassemble, compile, and apply the learned knowledge
Bank: Teacher can add, modify and delete learning mate- content, and then, complete the learning tasks and achieve
rials based on student’s learning progress and chapter con- learning goals.
tent; (3) Mission Bank: Teacher can add, modify and delete
learning mission based on student’s learning progress and C. LEARNING ACTIVITY DESIGN
chapter content; (4) Learning Portfolio: Teacher can grasp The experimental activities were carried out in the ‘‘Pro-
and control the learning status of students and make adjust- gram Logic Thinking Education’’ university general educa-
ments in teaching method or progress at any time. The tion curriculum. The learning system was designed with an
student part includes (1) Know-How: Student can choose image program, which combines deep learning recommenda-
to read different chapters depend on their own learning tion with AR technology to explore the impact on the learn-
progress; (2) We’re Pro: Provide AR based programming ing effectiveness and computational thinking of non-major
language learning mission to student; (3) Common term: undergraduates. Figure 5 shows the learning activity process.
Provide student with commonly used programming language First, this experiment divided four non-major undergraduate
grammar; (4) Exam: Provide related questions for each students into the experimental group into the control group.
chapter. In the first stage of the experiment, the same teacher taught

VOLUME 8, 2020 45693

 P.-H. Lin, S.-Y. Chen: Design and Evaluation of a Deep Learning Recommendation Based AR System

FIGURE 3. The functions of the deep learning recommendation based AR system.

the basic concepts of programming language, and their logic

applications, in order to confirm that all students have similar
prior knowledge, the pre-test was applied. In the second stage,
the teacher explained the operating procedures of the AR sys-
tem to all students on their mobile devices, where the learning
content and 3D materials in learning activities were all the
same, and the students were divided into the experimental
FIGURE 4. The snapshot of learning mission.
group and the control group for a total of 5 hours of learning
activities. The experimental group students used the AR sys-
tem with deep learning recommendation to perform learning password to confirm their identity and permissions. After
tasks in a ubiquitous learning environment, where different verify, student can login to the system for the programming
3D objects were presented in the real environment depending language course. Student can choose different chapters of
on the learning tasks, students judged and chose the program- programming language course to read according to their per-
ming syntax that could achieve the goal according to the task sonal learning progress. After reading, student can practice
instructions, and the deep learning recommendation mecha- and apply programming language through the Mission func-
nism suggested the learning task to be strengthened according tion. The learning mission provide different 3D route mission
to the student’s logical judgment and the application of the to allow student coding from the starting point to the end point
procedural grammar. The control group students used the AR through programming language. During the programming
system without deep learning recommendation to perform language learning process, not only examine student’s ability
learning tasks in a ubiquitous learning environment, where of programming language but also train student’s ability of
different 3D objects were presented in the real environment logical thinking and judgment in solving mission. The results
depending on the learning tasks, and students judged and of the compiled programming code will be recorded in the
chose the programming syntax that could achieve the goal database for analysis. The deep learning recommendation
according to the task instructions. In the third stage, after mechanism will decompose and analysis student’s program-
the learning activities, all students conducted post-test and ming language learning process and recommends the related
questionnaires to evaluate the impact of the learning activities learning mission that the student who lack of programming
on the students’ computational thinking. In addition, students language understanding. Finally, the assessment of student’s
in the experimental group were interviewed to understand learning achievement is carried out by exam. If the learning
their willingness to use the system. achievement isn’t as expected, student can restart the learning
In addition, figure 6 shows the student-operated learning scenario of programming language until the apply it well.
scenario of this study, student entered their own account and All the learning process will be recorded in the database,

45694 VOLUME 8, 2020

P.-H. Lin, S.-Y. Chen: Design and Evaluation of a Deep Learning Recommendation Based AR System

FIGURE 5. Learning activity scenario.

FIGURE 6. Student-operated learning scenario.

including login and logout time, learning material read by language before the learning activities of the students of
student, time of reading, time of learning mission, the steps the experimental group and control group, which includes
of student solving mission. 10 multiple-choice questions and 10 fill-in-the-blank ques-
tions. Moreover, the post-test was used to evaluate students’
D. MEASURING TOOL understanding and application of the programming courses,
In this study, the pre-test, post-test, and questionnaire were including five yes-no questions, five multiple-choice ques-
designed to assess the impact of learning activities on tions, and ten fill-in-the-blank questions. The pre-test and
students’ learning achievement and computational thinking post-test questions, with a total score of 100, were designed
skills. The pre-test was used to confirm there was no signif- by experts with years of programming experience. The com-
icant difference in the basic concepts of the programming putational thinking ability questionnaire was adapted from

VOLUME 8, 2020 45695

 P.-H. Lin, S.-Y. Chen: Design and Evaluation of a Deep Learning Recommendation Based AR System

TABLE 1. The Cronbach’s α value of computational thinking dimension.

TABLE 2. T-test result for learning achievement.

TABLE 3. T-test result for computational thinking.

Korkmaz et al. [1], which has a total of 29 questions scored the existence of possible variables in the process of solving
according to a five-point Likert scale, where the range is 1 problems’’, and its Cronbach’s α is 0.965. In addition, in the
(very disagree) to 5 (very agree). Table 1 shows the Cron- qualitative interview part, the experimental group students
bach’s α value of computational thinking dimension. The (with the use of AR system with deep learning recommenda-
questionnaire is divided into 5 dimensions: Creativity, Algo- tion) were taken as the interview subjects, and the interview
rithmic Thinking, Cooperativity, Critical Thinking, and Prob- content is the Technology Acceptance Model (TAM), as pro-
lem Solving. There are eight questions in creativity, such as posed by Davis [59].
‘‘I believe I can solve problems when I face new tasks’’, ‘‘My
dream is an important factor when I perform many tasks’’, and IV. DATA ANALYSIS AND RESULTS
‘‘In the course of performing tasks, I believe in my intuition This study used the SPSS 25 Windows version to analyze the
and feelings about solving problems’’, and its Cronbach’s α study data, where the P-value must be less than 0.05 to be
is 0.953. There are six questions for logical thinking, such as considered significant. This study used independent sample
‘‘I am particularly interested in the mathematical process’’, t-test to determine the difference between the data of the
‘‘I can solve problems with mathematics in my daily life’’, students in the experimental group and the control group.
and ‘‘I think I can quickly discover the relationship between Table 2 shows the t-test used to examine the difference in
numbers’’, and its Cronbach’s α is 0.951. There are four learning outcomes between the students before and after
questions for cooperativity, such as ‘‘I like to experience the learning activity. The results show that, in the pre-test
cooperative learning with my classmates’’, ‘‘More ideas will part, the average of the experimental group is 52.5, and the
be generated in the cooperative learning process’’, and ‘‘I standard deviation is 17.894; the average of the control group
think that through cooperative learning, we can achieve/get is 54.9, and the standard deviation is 16.153. The t-test result
better results’’, and its Cronbach’s α is 0.912. There are a (t = −0.693) shows no significant difference between the
total of 5 questions for critical thinking, such as ‘‘I am very two groups. In the post-test part, the experimental group
good at formulating steps to solve complex problems’’, ‘‘I average is 84.06, and the standard deviation is 13.033; the
think it is very interesting to try to solve complex problems’’, control group average is 68.88, and the standard deviation
and ‘‘I am willing to learn something that is challenging for is 17.920. The t-test result (t = 4.780, p < 0.05) shows
myself’’, and its Cronbach’s is 0.964. There are 6 questions a significant difference between the two groups. It can be
for problem solving, such as ‘‘I can put forward many ideas seen that the two groups of students had the same cognition
when considering ways to solve problems’’, ‘‘I can gradually level of the programming language course before the learning
apply my ideas to solve problems’’, and ‘‘I can consider activities, and after the learning activities, the experimental

45696 VOLUME 8, 2020

P.-H. Lin, S.-Y. Chen: Design and Evaluation of a Deep Learning Recommendation Based AR System

TABLE 4. T-test result for the five dimensions of computational thinking.

group students have better learning outcomes than the control were interviewed after participating in the learning activities.
group students. There are four dimensions in the TAM model, perceived
Table 3 shows the computational thinking of the two usefulness, perceived ease of use, attitude toward using, and
groups of students after the learning activities. The average behavioral intention to use. In terms of perceived usefulness,
of the experimental group is 4.46, and the standard deviation the experimental group students believe that the deep learn-
is 0.504; the average of the control group is 3.51, and the ing recommendation method helped ‘‘improve their learn-
standard deviation is 0.649. The t-test result (t = 8.045), ing scores’’ and ‘‘organize knowledge and content faster’’.
p < 0.05) shows a significant difference between the two Students mentioned that the task which provide from deep
groups, meaning that the experimental group students sig- learning recommended learning allowed them to enhance and
nificantly improved their computational thinking ability after apply those program syntaxes that they are not familiar with
the learning activities. Table 4 shows the influence of each or do not understand, rather than just memorizing the syntax.
group of students in computational thinking. In the part of In terms of perceived ease of use, the experimental group
creativity, the experimental group average is 4.53, and the students believe that the AR system is ‘‘easy to operate’’
standard deviation is 0.407; the control group average is and ‘‘the interface is clear and easy to use’’. Students men-
3.02, and the standard deviation is 0.643. The t-test result tioned that system operation can be directly used intuitively,
(t = 13.848, p < 0.05) shows a significant differ- without much thinking. In terms of attitude toward using,
ence between the two groups. In the logical thinking part, the experimental group students believe that the AR system
the experimental group average is 4.47, and the standard ‘‘made learning interesting and improved their confidence in
deviation is 0.579; the control group average is 3.21, and the learning’’. Students mentioned that the presentation of the AR
standard deviation is 0.853. The t-test result (t = 8.537, learning tasks made the tasks feel interesting and changeable,
p< 0.05) shows a significant difference between the two and they felt satisfied with their ability to actually compile
groups. In the part of cooperativity, the experimental group the programming language to solve the problem. In terms of
average is 4.01, and the standard deviation is 0.839; the con- behavioral intention to use, the experimental group students
trol group average is 3.85, and the standard deviation is 0.985. expressed ‘‘I hope to use this system to learn in the future.’’
The t-test result (t = 0.851) shows no significant difference Students mentioned that courses like programming languages
between the two groups. In the part of critical thinking, the are usually rather boring, but the guidance and practice of
experimental group average is 4.33, and the standard devi- these learning tasks were a great help when constructing their
ation is 0.622; the control group average is 3.86, and the programming language knowledge content.
standard deviation is 1.111; The t-test result (t = 2.566,
p< 0.05) shows a significant difference between the two
groups. Finally, in the problem-solving part, the experimental V. CONCLUSION
group average is 4.69, and the standard deviation is 0.427; This research provides an image-based programming
the control group average is 3.68, and the standard deviation learning system, including deep learning based learning
is 1.044. The t-test result (t = 6.284, p < 0.001) shows a recommendations, AR technology, and learning theory to
significant difference between the two groups. students in different learning areas to overcome the learn-
The qualitative interview content design is based on the ing difficulties of non-major undergraduates. Deep learning
TAM model, and the students of the experimental group recommendation method can enable students facing different

VOLUME 8, 2020 45697

 P.-H. Lin, S.-Y. Chen: Design and Evaluation of a Deep Learning Recommendation Based AR System

learning tasks to apply and practice the programming lan- learning behaviors of different majors. In addition, this
guage and logic. research was designed for a programming language, thus,
In order to evaluate the effect of the AI learning rec- related research in the future can focus on different subjects.
ommendation methods, this study designed an experiment
to compare the effects of deep learning recommendation REFERENCES
based AR system with non-deep learning recommendation [1] Ö. Korkmaz, R. Çakir, and M. Y. Özden, ‘‘A validity and reliability study of
based AR learning system on student learning achievement, the computational thinking scales (CTS),’’ Comput. Hum. Behav., vol. 72,
pp. 558–569, Jul. 2017.
as well as inspire students’ computational thinking abilities. [2] L. Ma, J. Ferguson, M. Roper, and M. Wood, ‘‘Investigating and improv-
The experimental results show that, through deep learning ing the models of programming concepts held by novice programmers,’’
recommendation method, the students’ learning achievement Comput. Sci. Edu., vol. 21, no. 1, pp. 57–80, Mar. 2011.
[3] M. J. Rubin, ‘‘The effectiveness of live-coding to teach introductory pro-
is significantly better than that of non-deep learning recom- gramming,’’ in Proc. 44th ACM Tech. Symp. Comput. Sci. Edu. (SIGCSE),
mendation method. This means the deep learning recommen- 2013, pp. 651–656.
dation method is quite helpful for non-major undergraduates [4] K. M. Cauley, ‘‘Studying knowledge acquisition: Distinctions among
procedural, conceptual and logical knowledge,’’ Virginia Commonwealth
to learn through a programming language. In addition, in the Univ., Richmond, VA, USA, Tech. Rep. ED 278 682, 1986.
part of computational thinking ability, students using deep [5] E. Soloway, ‘‘Learning to program = learning to construct mechanisms and
learning recommendation based AR system are significantly explanations,’’ Commun. ACM, vol. 29, no. 9, pp. 850–858, Sep. 1986.
[6] K.-Y. Chin, C.-S. Wang, and Y.-L. Chen, ‘‘Effects of an augmented reality-
better than using non-deep learning recommendation based
based mobile system on students’ learning achievements and motiva-
AR system. Among the various dimensions of computational tion for a liberal arts course,’’ Interact. Learn. Environ., vol. 27, no. 7,
thinking, creativity, logical computing, critical thinking, and pp. 927–941, Jul. 2018.
problem-solving skills are significantly different among the [7] T.-C. Huang, C.-C. Chen, and Y.-W. Chou, ‘‘Animating eco-education:
To see, feel, and discover in an augmented reality-based experiential
two groups of students; there was no significant difference in learning environment,’’ Comput. Edu., vol. 96, pp. 72–82, May 2016.
cooperativity. Learning through deep learning recommenda- [8] M. Dunleavy, C. Dede, and R. Mitchell, ‘‘Affordances and limitations of
tion system not only improves students’ learning outcomes, immersive participatory augmented reality simulations for teaching and
learning,’’ J. Sci. Edu. Technol., vol. 18, no. 1, pp. 7–22, 2009.
it also helps students to develop their computational thinking [9] K. R. Bujak, I. Radu, R. Catrambone, B. MacIntyre, R. Zheng, and
skills. As the cooperation between students was only limited G. Golubski, ‘‘A psychological perspective on augmented reality in the
to discussions of the learning activities, there was no signifi- mathematics classroom,’’ Comput. Edu., vol. 68, pp. 536–544, Oct. 2013.
[10] F. Ke and Y.-C. Hsu, ‘‘Mobile augmented-reality artifact creation as a
cant difference in cooperativity. component of mobile computer-supported collaborative learning,’’ Internet
Qualitative interviews show that most students accepted Higher Edu., vol. 26, pp. 33–41, Jul. 2015.
and were willing to learn the programming language with [11] P.-H. Wu, G.-J. Hwang, M.-L. Yang, and C.-H. Chen, ‘‘Impacts of inte-
grating the repertory grid into an augmented reality-based learning design
the deep learning recommendation system. Students said that on students’ learning achievements, cognitive load and degree of satisfac-
this way of learning improved their learning effectiveness and tion,’’ Interact. Learn. Environ., vol. 26, no. 2, pp. 221–234, 2018.
made them more confident in the subject of programming [12] A. M. Shahiri, W. Husain, and N. A. Rashid, ‘‘A review on predicting
student’s performance using data mining techniques,’’ Procedia Comput.
language, which is very helpful for non-major undergraduate Sci., vol. 72, pp. 414–422, 2015.
students to overcome learning gaps. [13] D. Liu and C. Yang, ‘‘A deep reinforcement learning approach to proactive
content pushing and recommendation for mobile users,’’ IEEE Access,
vol. 7, pp. 83120–83136, 2019.
VI. DISCUSSION AND FUTURE STUDIES [14] Z. Huang, C. Yu, J. Ni, H. Liu, C. Zeng, and Y. Tang, ‘‘An efficient hybrid
In this study, students learned a programming language recommendation model with deep neural networks,’’ IEEE Access, vol. 7,
through a deep learning recommendation based AR system, pp. 137900–137912, 2019.
[15] W. W. T. Fok, Y. S. He, H. H. A. Yeung, K. Y. Law, K. Cheung, Y. Ai,
and the students’ learning performance and computational and P. Ho, ‘‘Prediction model for students’ future development by deep
thinking ability after the learning activities were explored. learning and tensorflow artificial intelligence engine,’’ in Proc. 4th Int.
This way of learning can effectively solve the programming Conf. Inf. Manage. (ICIM), May 2018, pp. 103–106.
[16] D. Weintrop, E. Beheshti, M. S. Horn, K. Orton, L. Trouille, K. Jona,
language difficulties of students, and abstract syntax and and U. Wilensky, ‘‘Interactive assessment tools for computational thinking
logic application can be improved in the learning activi- in high school STEM classrooms,’’ in Proc. Int. Conf. Intell. Technol.
ties. Through the recommendation of deep learning, which Interact. Entertainment, 2014, pp. 22–25.
[17] G. Chen, J. Shen, L. Barth-Cohen, S. Jiang, X. Huang, and M. Eltoukhy,
can reflect on the myths and misunderstandings in learn-
‘‘Assessing elementary students’ computational thinking in everyday rea-
ing, and allow students to change from passive absorption soning and robotics programming,’’ Comput. Edu., vol. 109, pp. 162–175,
of knowledge to actively understanding problems, thinking Jun. 2017.
about problems, and solving problems. [18] A. A. de Souza, T. S. Barcelos, R. Munoz, R. Villarroel, and L. A. Silva,
‘‘Data mining framework to analyze the evolution of computational
This research was mainly designed for college students thinking skills in game building workshops,’’ IEEE Access, vol. 7,
as the research subjects. In the learning activities, all stu- pp. 82848–82866, 2019.
dents were non-major undergraduates, thus, their learning [19] R. Munoz, R. Villarroel, T. S. Barcelos, F. Riquelme, A. Quezada, and
P. Bustos-Valenzuela, ‘‘Developing computational thinking skills in ado-
behaviors were slightly different from information tech- lescents with autism spectrum disorder through digital game program-
nology based students. For example, possibly due to their ming,’’ IEEE Access, vol. 6, pp. 63880–63889, 2018.
sufficient knowledge of visual arts, students from the Depart- [20] N. Da Cruz Alves, C. Gresse Von Wangenheim, and J. C. R. Hauck,
‘‘Approaches to assess computational thinking competences based on code
ment of Arts were quicker in the progress of the AR learn- analysis in K-12 education: A systematic mapping study,’’ Inform. Edu.,
ing activities, thus, future research can discuss the different vol. 18, no. 1, pp. 17–39, Apr. 2019.

45698 VOLUME 8, 2020

P.-H. Lin, S.-Y. Chen: Design and Evaluation of a Deep Learning Recommendation Based AR System

[21] T.-C. Hsu, S.-C. Chang, and Y.-T. Hung, ‘‘How to learn and how to teach [45] Y. Wu et al., ‘‘Google’s neural machine translation system: Bridging the
computational thinking: Suggestions based on a review of the literature,’’ gap between human and machine translation,’’ 2016, arXiv:1609.08144.
Comput. Edu., vol. 126, pp. 296–310, Nov. 2018. [Online]. Available: http://arxiv.org/abs/1609.08144
[22] A. Mejías Borrero and J. M. A. Márquez, ‘‘A pilot study of the effective- [46] W. Xing and D. Du, ‘‘Dropout prediction in MOOCs: Using deep learning
ness of augmented reality to enhance the use of remote labs in electrical for personalized intervention,’’ J. Educ. Comput. Res., vol. 57, no. 3,
engineering education,’’ J. Sci. Edu. Technol., vol. 21, no. 5, pp. 540–557, pp. 547–570, 2019.
2012. [47] J. M. Wing, ‘‘Computational thinking,’’ Commun. ACM, vol. 49, no. 3,
[23] R. T. Azuma, ‘‘A survey of augmented reality,’’ Presence, Teleoperators pp. 33–35, 2006.
Virtual Environ., vol. 6, no. 4, pp. 355–385, 1997. [48] J. M. Wing, ‘‘Computational thinking and thinking about computing,’’
[24] Á. Di Serio, M. B. Ibáñez, and C. D. Kloos, ‘‘Impact of an augmented Philos. Trans. Roy. Soc. A, Math., Phys. Eng. Sci., vol. 366, no. 1881,
reality system on students’ motivation for a visual art course,’’ Comput. pp. 3717–3725, 2008.
Edu., vol. 68, pp. 586–596, Oct. 2013. [49] J. Cuny, L. Snyder, and J. M. Wing, ‘‘Demystifying computational think-
[25] H.-K. Wu, S. W.-Y. Lee, H.-Y. Chang, and J.-C. Liang, ‘‘Current status, ing for non-computer scientists,’’ unpublished, 2010. [Online]. Available:
opportunities and challenges of augmented reality in education,’’ Comput. http://www.cs.cmu.edu/~CompThink/resources/TheLinkWing.pdf
Edu., vol. 62, pp. 41–49, Mar. 2013. [50] A. V. Aho, ‘‘Computation and computational thinking,’’ Comput. J.,
vol. 55, no. 7, pp. 832–835, 2012.
[26] T. H. C. Chiang, S. J. H. Yang, and G.-J. Hwang, ‘‘Students’ online interac-
[51] F. J. Garcia-Penalvo, ‘‘Editorial computational thinking,’’ IEEE Revista
tive patterns in augmented reality-based inquiry activities,’’ Comput. Edu.,
Iberoamericana de Tecnologias del Aprendizaje, vol. 13, no. 1, pp. 17–19,
vol. 78, pp. 97–108, Sep. 2014.
Feb. 2018.
[27] G.-J. Hwang, P.-H. Wu, C.-C. Chen, and N.-T. Tu, ‘‘Effects of an aug- [52] V. Barr and C. Stephenson, ‘‘Bringing computational thinking to K-12:
mented reality-based educational game on students’ learning achieve- What is involved and what is the role of the computer science education
ments and attitudes in real-world observations,’’ Interact. Learn. Environ., community?’’ ACM Inroads, vol. 2, no. 1, p. 48, Feb. 2011.
vol. 24, no. 8, pp. 1895–1906, 2016. [53] M. M. Sysło and A. B. Kwiatkowska, ‘‘Informatics for all high school
[28] Y.-M. Huang and P. H. Lin, ‘‘Evaluating students’ learning achievement students,’’ in Proc. Int. Conf. Inform. Schools, Situation, Evol., Perspect.,
and flow experience with tablet PCs based on AR and tangible technology 2013, pp. 43–56.
in u-learning,’’ Library Hi Tech, vol. 35, no. 4, pp. 602–614, Nov. 2017. [54] F. J. García-Peñalvo and A. J. Mendes, ‘‘Exploring the computational
[29] Y.-C. Chen, H.-L. Chi, W.-H. Hung, and S.-C. Kang, ‘‘Use of tangible and thinking effects in pre-university education,’’ Comput. Hum. Behav.,
augmented reality models in engineering graphics courses,’’ J. Prof. Issues vol. 80, pp. 407–411, Mar. 2018.
Eng. Edu. Pract., vol. 137, no. 4, pp. 267–276, Oct. 2011. [55] L. Mannila, V. Dagiene, B. Demo, N. Grgurina, C. Mirolo, L. Rolandsson,
[30] H.-Y. Wang, H. B.-L. Duh, N. Li, T.-J. Lin, and C.-C. Tsai, ‘‘An investi- and A. Settle, ‘‘Computational thinking in K-9 education,’’ in Proc. Innov.
gation of university students’ collaborative inquiry learning behaviors in Technol. Comput. Sci. Educ. Conf., 2014, pp. 1–29.
an augmented reality simulation and a traditional simulation,’’ J. Sci. Edu. [56] J. Voogt, P. Fisser, J. Good, P. Mishra, and A. Yadav, ‘‘Computational
Technol., vol. 23, no. 5, pp. 682–691, May 2014. thinking in compulsory education: Towards an agenda for research and
[31] P.-H. Lin, Y.-M. Huang, and C.-C. Chen, ‘‘Exploring imaginative capa- practice,’’ Edu. Inf. Technol., vol. 20, no. 4, pp. 715–728, Jun. 2015.
bility and learning motivation difference through picture E-Book,’’ IEEE [57] M.-C. Tsai and C.-W. Tsai, ‘‘Applying online externally-facilitated regu-
Access, vol. 6, pp. 63416–63425, 2018. lated learning and computational thinking to improve students’ learning,’’
[32] N.-F. Huang, C.-C. Chen, J.-W. Tzeng, T.-T. Fang, and C.-A. Lee, ‘‘Con- Universal Access Inf. Soc., vol. 17, no. 4, pp. 811–820, 2018.
cept assessment system integrated with a knowledge map using deep [58] B. Wu, Y. Hu, A. R. Ruis, and M. Wang, ‘‘Analysing computational think-
learning,’’ in Proc. Learn. MOOCS (LWMOOCS), Sep. 2018, pp. 113–116. ing in collaborative programming: A quantitative ethnography approach,’’
[33] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning: Mit Press J. Comput. Assist. Learn., vol. 35, no. 3, pp. 421–434, Feb. 2019.
Essential Knowledge Series. Cambridge, MA, USA: MIT Press, 2019. [59] F. D. Davis, ‘‘Perceived usefulness, perceived ease of use, and user accep-
[34] V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, tance of information technology,’’ MIS Quart., vol. 13, no. 3, pp. 319–340,
S. Venugopalan, K. Widner, T. Madams, J. Cuadros, R. Kim, R. Raman, Sep. 1989.
P. C. Nelson, J. L. Mega, and D. R. Webster, ‘‘Development and validation
of a deep learning algorithm for detection of diabetic retinopathy in retinal
fundus photographs,’’ J. Amer. Med. Assoc., vol. 316, no. 22, p. 2402,
Dec. 2016.
[35] K.-L. Du, and M. N. Swamy, Neural Networks and Statistical Learning.
New York, NY, USA: Springer, 2013. PEI-HSUAN LIN is currently pursuing the Ph.D.
[36] Y. LeCun, Y. Bengio, and G. Hinton, ‘‘Deep learning,’’ Nature, vol. 521,
degree with the Department of Engineering Sci-
no. 7553, p. 436, 2015. ence, National Cheng Kung University, Taiwan.
[37] O. Chapelle, B. Scholkopf, and Eds. A. Zien, ‘‘Semi-supervised learn- Her research interests include education technol-
ing (Chapelle, O. et al., Eds.; 2006) [book reviews],’’ IEEE Trans. Neural ogy, ubiquitous learning, and augmented reality.
Netw., vol. 20, no. 3, p. 542, Mar. 2009.
[38] X. Zhu and A. B. Goldberg, ‘‘Introduction to semi-supervised learning,’’
Synth. Lectures Artif. Intell. Mach. Learn., vol. 3, no. 1, pp. 1–130, 2009.
[39] M. A. Ahad, G. Tripathi, and P. Agarwal, ‘‘Learning analytics for IoE
based educational model using deep learning techniques: Architecture,
challenges and applications,’’ Smart Learn. Environ., vol. 5, no. 1, p. 7,
Aug. 2018.
[40] M. A. Peters, Deep Learning, Education and the Final Stage of Automa-
tion. New York, NY, USA: Taylor & Francis, 2018.
SHIH-YEH CHEN received the Ph.D. degree in
[41] P. Baldi, P. Sadowski, and D. Whiteson, ‘‘Searching for exotic particles in
high-energy physics with deep learning,’’ Nature Commun., vol. 5, no. 1,
mobile agent from the Department of Engineering
p. 4308, Jul. 2014. Science, National Cheng Kung University, Tainan,
[42] M.-Y. Day and Y.-D. Lin, ‘‘Deep learning for sentiment analysis on Google Taiwan, in 2016.
play consumer review,’’ in Proc. IEEE Int. Conf. Inf. Reuse Integr. (IRI), Since 2017, he has been an Assistant Profes-
Aug. 2017, pp. 382–388. sor with the Department of Computer Science
[43] G. B. Goh, N. O. Hodas, and A. Vishnu, ‘‘Deep learning for computational and Information Engineering, National Taitung
chemistry,’’ J. Comput. Chem., vol. 38, no. 16, pp. 1291–1307, Mar. 2017. University, Taitung, Taiwan. His main research
[44] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, interests include e-learning education, social inter-
and S. Thrun, ‘‘Dermatologist-level classification of skin cancer with deep actions, and mobile networking.
neural networks,’’ Nature, vol. 542, no. 7639, pp. 115–118, Jan. 2017.

VOLUME 8, 2020 45699