An Exploratory Study on the Availability of Open

Educational Resources to Support the Teaching and

Learning of Programming
William Simão de Deus and Ellen Francine Barbosa
Institute of Mathematics and Computer Science (ICMC)
University of São Paulo (USP)
São Carlos, Brazil
[email protected], [email protected]

Abstract—Context: This Research Full Paper presents an ex- is involved in activities such as preparing presentations, cor-
ploratory study on the availability of open educational resources recting exercises and creating projects, which requires a lot
(OERs) to support the teaching and learning of programming. of concentration and effort, due to its complexity [6]. At the
Currently, programming is considered a fundamental course in
many undergraduate programs. Moreover, many countries are same time, students also face many challenges. Medeiros et al.
teaching children and young people how to program, considering [2], for instance, reported difficulties regarding abstract/logical
this as an essential skill in the future. However, students report limitations, programming language syntax, structure language
many difficulties when they are learning to program due to (control, data and code), among others. Together, these factors
limitations of abstraction/logic, syntax, structure, among others. evidence that although the teaching and learning of program-
Similarly, teachers also argue that teaching programming is an
arduous task due to the preparation of presentations, correction ming is necessary, it is still a complex task for both teachers
of exercises and creation of projects. Objective: Given this sce- and students [7].
nario, our objective was analyze the availability of OERs - open In this perspective, OERs emerge as a potential solution
materials that present different contents and formats distributed [8]. In short, OERs are free and open teaching materials made
by internet - and their support for programming students and available on the internet [9]. These resources are licensed
teachers. Method: Aiming to reach the aforementioned objective,
we conducted an exploratory study among 3,990 OERs, using under regulations that encourage their use, reuse and sharing
computational processing and descriptive statistic. Results: We among people for educational purposes. Additionally, OERs
analyzed the OER descriptions and compared them with the are known to have different types and forms, such as text-
current programming challenges. Also, we investigated aspects books, courses, exercises, software, etc [10]. Because of these
of the availability of these materials. Conclusion: There are characteristics, OERs have a huge potential to support teachers
several OER available that can support teaching and learning
of programming. However, to enhancing their dissemination, we during programming classes, offering, for instance, presenta-
must facilitate its discovery, reuse and sharing. tions that can be personalized, exercises, code snippets, texts
Index Terms—Teaching and Learning of Programming, Open and videos, among others [11]. Motivated by this scenario,
Educational Resources, Exploratory Study many initiatives have been developed based on OER with
programming content. As an example, we highlight the OER
I. I NTRODUCTION repository with programming exercises presented by Staubitz
et al. [12]. Another case was presented by Dol and Indi [13],
In the last years, programming has been considered a in which OERs have been adopted to teach C programming.
fundamental course for many undergraduate programs as a In both cases, the authors reported good results with the use
way to support student development [1], especially in STEM of OERs among students.
degrees (Science, Technology, Engineering, and Mathematics) However, despite these positive indicators, the use of OERs
[2]. In addition, many countries have suggested and proposed to support programming education still has many challenges.
initiatives to disseminate programming education to children First, many teachers are not familiar with OERs reality yet.
and young people, realizing that knowing how to program will As commented by Marcolino and Barbosa [14], commonly
be an essential skill in the near future. For instance, United the instructors adopt outdated materials, such as lists of
Kingdom and Australia included introductory programming to programming exercises and text-based programming environ-
primary and secondary curricula [3]. Also, more recently, the ments, disregarding the use of OERs. Second, it is difficult to
Brazilian Computer Society released a set of guidelines for gather relevant programming resources. According to Piedra
teaching computing in basic education in Brazil [4]. et al. [15], this is due to the growth of unorganized OER
Traditionally, teaching a student how to program involves collections. Thus, there is no pattern between title, abstract
different stages of preparation and use of different supporting and keywords. As a result, a search of “teaching programming”
materials [5]. Throughout the teaching process, the teacher could return irrelevant results, such as linear programming or

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 19,2021 at 20:27:32 UTC from IEEE Xplore. Restrictions apply.
genetic programming. Finally, it is difficult to exactly estimate discuss how OERs can reduce the incidence of these problems
how many resources exist and which ones are available for use, supporting teachers and students.
reuse and sharing. This occurs because there is little empirical
evidence on the types of adoption and use of OERs in the B. Open Educational Resources
repositories [16]. To understand how OER can support teaching and learning
Considering this context, in this exploratory study we programming, it is necessary to verify its origins. The defini-
investigated the availability of OERs and their support the tion of OER was first introduced by UNESCO [9] as “the open
teaching and learning of programming. Initially, we searched provision of educational resources, enabled by information
in different digital sources for resources on programming. So and communication technologies, for consultation, use, and
next, we extracted information about the resources using web adaptation by a community of users for non-commercial pur-
crawlers1 . Thus, through the use of computational processing poses”. Thus, OER has two main characteristics: first, it refers
and descriptive statistics we identified the main programming to information and communication technologies. Second, it is
languages, types of resources, licenses and restrictions of use, about the use, reuse, and adaptation.
adapt and change. Due to information and communication technologies, OERs
The remainder of this paper is organized as follow: Section can support different types of resources [10]. Thus, it is very
II provides a summary of programming and OERs. Section III common to find resources such as videos, audios, courses,
presents a comparison of related works. The extraction process software, exercises, etc. This feature can already support
is described in Section IV. Section V is analyzed the current teachers and students, considering that OERs can offer mate-
challenges for the teaching and learning of programming rials to support the preparation of classes and also maximize
and their support by OERs. Next, Section VI shown the student learning. Furthermore, when referring to use, reuse,
availability, licences and languages of OERs for programming. and adaptation, OERs are subject to their own licenses that
Discussions are presented in Section VII. Finally, the conclu- make it possible to share teaching and learning materials with
sion and future works are presented in Section VIII. freedom of remix [22]. Thus, to prevent plagiarism and/or
piracy problems, the resources are generally completely free
II. BACKGROUND or licensed under specific permissions [23]. Therefore, the
materials can be adopted as an attempt to suppress the basic
A. Teaching and Learning of Programming
difficulties of the students at the same time that they can be
According to Lokar and Pretnar [17], there are different found in different forms, also avoiding demotivation of the
models and techniques for the teaching and learning of pro- classic forms of teaching.
gramming. However, recent researches show that most teachers
only adopt classic forms, such as problems exposure and lists III. R ELATED W ORK
of exercises [14]. Although this is the most common way The intersection between OERs and the teaching and learn-
of teaching, teachers, and students of programming end up ing of programming is being investigated by different authors.
overwhelmed. Initially, we highlight the investigation conducted by Staubitz
Initially, from the teacher’s point of view, teaching pro- et al. [12]. The authors presented a collection of OERs for
gramming is a complex task. Borges et al. [18], for example, open auto-gradable programming exercises. In parallel, Maksi-
noted that the work is intensified due to the need of reviewing menkova and Podbelskiy [24] provided OERs to develop tasks
the code developed by students, and as each one has its and assessments for programming students. However, both
own programming logic, it is essential to reflect and analyze approaches have the same limitations: the OER collections
several possibilities. Oliveira et al. [19] also noted that the high had few resources and have type restrictions. Also, resources
number of students by classroom and their distinct degrees of did not cover different programming domains.
code complexity also end up increasing the workload. More specifically, Dol and Indi [13] presented how OERs
Regarding the student’s view, we highlight that program- can assist the teaching of C language programming. Nonethe-
ming learning is also complicated. According to Yildiz [20], less, the proposed solution was planned considering specific
many students have difficulties to learn the coding process, platforms, requiring integration among different tools to work.
producing low-quality code. Moreover, for many students, Due to the complexity of this operation, the solution was not
it is difficult to assimilate abstract aspects that the learning practical.
of programming involves [21]. Thus, such factors end up Considering the use of OERs in the general aspects, To-
contributing to demotivation, making the programming courses var et al. [25] presented how the Merlot repository2 started
achieve the highest failure rates in degree programs [18]. communities dedicated to the use, reuse and sharing of re-
Currently, it is argued that these problems occur due to the sources in Computer Science (CS) and Information Systems
overload of teachers and to the lack of basic foundations, such (IT). Nevertheless, the approach has two main limitations: (i)
as mathematics, abstraction thinking, and problem-solving the authors used only a single source of resources (Merlot
in the student formation [19]. With this in mind, next, we repository) discarding the others; and (ii) only data about the
1 Tool that visits a web page and extracts preformatted information. 2 https://www.merlot.org/merlot/

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 19,2021 at 20:27:32 UTC from IEEE Xplore. Restrictions apply.
top 100 CS and IT materials introduced was analyzed. In Table considering the languages that the authors could understand
I we summarize the limitations of related work. the structure of the sources and consult supporting materials,
such as documentation.
TABLE I After applying these criteria, we elected 11 different digital
S YNTHESIS OF L IMITATIONS IN R ELATED W ORK
sources. The list of selected sources are shown in Table II.
Similar Work Limitations
Staubitz et al. [12], Maksimenkova Few resources, few programming
and Podbelskiy [24] domains and resources type restric- TABLE II
tions L IST OF S ELECTED S OURCES
Dol and Indi [13] External dependency and does not
Number Name Link
support different resources type
01 Curriki https://www.curriki.org/
Tovar et al. [25] Unique source and small sample
02 EduCAPES https://educapes.capes.gov.br/
(top 100)
03 Merlot https://www.merlot.org/merlot/
04 OER Commons https://www.oercommons.org/
05 OpenStax https://openstax.org/
06 Open UCT https://open.uct.ac.za/
IV. P LANNING OF E XPLORATORY S TUDY 07 Skills Commons https://www.skillscommons.org/
A. Research Questions 08 Temoa http://temoa.tec.mx/node
09 TIB https://www.tib.eu/en/
As mentioned before, our goal is to analyze the availability 10 Xpert https://www.nottingham.ac.uk/xpert/
of OERs and their support and availability for programming 11 Zenodo https://zenodo.org/
students and teachers. For this, we elaborated the following
research questions (RQs):
• RQ1 : What challenges of teaching and learning program-
C. Data Extraction
ming are being supported by OERs?
– RQ1.1 : What are the programming languages? Finally, after the preparation was started the extraction pro-
– RQ1.2 : What types of resources? cess. For this, we adopted computational processing. Thereby,
• RQ2 : How are OERs being made available?
for each source a web crawler was created. Initially, the
crawlers searched for the term “programming” in selected
– RQ2.1 : What are the licenses?
sources. Soon after, the crawler checked the total of retrieved
– RQ2.2 : What are the languages?
results and started the extraction process. Basically, this pro-
We elaborated two main RQs: in RQ1 we investigated the cess involved reading the resource HTML, downloading its
current programming challenges. To deepen this analysis, we properties and converting the content to textual format (.csv).
checked the programming languages (RQ1.1 ) and the type of At the end of the process, the crawler adds all the content to
resources (RQ1.2 ) adopted. Meanwhile, in RQ2 we examined a unique spreadsheet.
how the resources are built and made available. For this, we To discard irrelevant resources, we performed a new search
expanded this question with an analysis of permissions (RQ2.1 ) based on the spreadsheet previously generated. In this new
and the languages (idiom) (RQ2.2 ). search, we only selected the resources that had some pro-
B. Data collection gramming language name. To validate this, we adopted a list
Next, we defined how to collect data to solve the RQs. For that contains 752 names of currently known programming
this we searched for digital sources in the OER World Map3 languages4 . As a final result, we retrieved 3,990 OERs. The
according following criteria: detailed summary of each sources is showed in Table III.
• Automatic search: support automatic search to allow the
web crawler to browse the collection; TABLE III
D ETAILS OF E XTRACTION
• Programming resources: have collection of resources
related to programming; and, Source Resources %
• Languages: must be in a language in which the re- Curriki 143 3.58%
EduCAPES 88 2.21%
searchers have domain and understanding. Merlot 223 5.59%
The first criterion has been set to remove sources that OER Commons 126 3.16%
OpenStax 602 15.09%
support only manual search in their collections. That is, the Open UCT 21 0.53%
user can only browse certain topics and cannot search for SkillsCommons 102 2.56%
terms. The second criterion was defined to prevent sources Temoa 2,147 53.81%
TIB 51 1.28%
without programming resources collections from being an- Xpert 32 0.80%
alyzed. Meanwhile in the third criterion, we defined that Zenodo 455 11.40%
only sources in English or Portuguese should be selected, TOTAL 3,990 100%

3 OER World Map is a project that lists OERs initiatives around the world,
including repositories, events, publications, etc. For more information access:
https://oerworldmap.org/resource/. 4 http://bit.ly/programming-languages-list

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 19,2021 at 20:27:32 UTC from IEEE Xplore. Restrictions apply.
 V. P ROGRAMMING C HALLENGES A DDRESSED BY OER S • Mathematics background: Gaps in mathematical
knowledge. 200 resources cited this feature.
To address RQ1 , we summarized each challenge and its A synthesis among the problems and resource descriptions
solution by resources, starting from the students’ view. Ac- is presented in the Table IV. In parallel, we show some the
cording to Medeiros et al. [2], from this perspective there are OER examples in Table V. As can be seen, many of the chal-
four groups of challenges: lenges reported in the literature have the terms or synonyms
1) Problem formulation: This group brings together the in OER description. Nevertheless, the solutions are different.
problems solving (difficulties to solve programming Most are related to the programming language, while the other
problems); abstraction (problems related to the program- challenges refer to the use of more effective techniques or
ming concepts); and algorithm (construct and understand tools. Because of this, we analyze in the following sections
algorithms). We found 535 resources that directly cite we separately analyze how resources can support students and
such problems or have a synonym in their description. teachers.
The algorithm problem was the most noticed, reaching
385 resources. TABLE IV
OER S AND THE C HALLENGES OF P ROGRAMMING
2) Solution expression: Medeiros et al. [2] organized this
group into five different problems with generic and Challenge Resources %
very similar challenges. So, we considered only two 1) Problem formulation 535 13.41%
1.1) Problem solving 73 1.83%
challenges: syntax (problems related to writing a code) 1.2) Abstraction 77 1.93%
and structure (data use, control structure and code). 138 1.3) Algorithm 385 9.65%
resources were found quoting these term, 134 mentioned 2) Solution expression 138 3.46%
2.1) Syntax 4 0.10%
structural problems.

Student
2.2) Structure (data, control, code) 134 3.36%
3) Execution/Evaluation: This group deals with the chal- 3) Execution/Evaluation 71 1.78%
lenges of debugging (debugging code) and tracing (error 3.1) Debugging 71 1.78%
3.2) Tracing 0 0.00%
tracking). We did not find any resource that supported 4) Behaviour 232 5.81%
the tracking problem, mainly because it is a very specific 4.1) Motivation/Engagement 165 4.14%
term that is not widely adopted in resources. However, 4.2) Time Management 17 0.43%
4.3) Study Skills 15 0.38%
71 resources were quoted about debugging. 4.4) Confidence 35 0.88%
4) Behavior: This group is concerned with problems of 5) Tools for teaching 538 13.48%
Teacher

motivation/engagement (when students are not motivated 6) Scale problems 0 0%

7) Feedback 240 6.02%
in the course), time management (schedule organiza- 8) Curriculum 88 2.21%
tion), study skills (self-teaching) and confidence (asking 9) Mathematics background 200 5.01%
for help when needed). The original classification had Note: We analyzed the description of the resources via computational pro-
another category called “perception of programming as cessing, checking for terms or synonyms in the metadata. We do not verify
a complex discipline”, but we discarded this category the content of OERs.
due to its specific name. This group was mentioned in
232 resources, with motivation/engagement having 165 A. The Programming Languages (RQ1.1 )
citations.
Many students find it difficult to learn how to program, es-
Considering the teachers’ perspective, Medeiros et al. [2] pecially when learning a specific language. So, it is important
originally mapped seven challenges. However, two were very to check which programming languages are being used in the
similar to the student’s ones, so we omitted them. The final elaboration of OER. Thus, we also solve the RQ1.1 . For this,
list of challenges for teachers was: we analyzed the three initial groups of challenges related to the
• Tools for teaching: Problems related to the lack of tools students’ view, presented in Table IV, crossing the occurrences
and methods to teach programming. In this category, with the 10 most common programming languages found in
we found 538 resources that mentioned the term or the data set.
synonyms. In general, the Java language was the most used in re-
• Scale problems: Teachers reported problems when es- sources, totaling 86 occurrences. The resources identified as
calating the exercises, such as many students and few “programming language” obtained the second most cited
resources for teaching. Due to the particularities of the occurrence. Although it is not, in fact a programming language
term, we did not find relevant resources. the term was used to present generic resources. We noticed
• Feedback: Common challenges regarding the lack of that 56 resources adopted this designation. The Python and
feedback between teachers and students. In this category, C languages were, respectively, the third and fourth most
we found 240 resources that cited the term. common occurrences, obtaining 51 and 40 occurrences each.
• Curriculum: Difficulties in organizing the students cur- The Scratch language appeared in the fifth position, with 27
riculum. We found 88 resources with this characteristic occurrences. Matlab had 15 occurrences and reached the sixth
in description. position. Meanwhile, Actionscript reached 13 occurrences and

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 19,2021 at 20:27:32 UTC from IEEE Xplore. Restrictions apply.
 TABLE V
E XAMPLES OF OER DESCRIPTIONS

Challenge Resource description

1.1) Problem Solving “introduction to computers and engineering problem solving”
1.2) Abstraction “introduction to computer science: programming abstractions”
1.3) Algorithm “algorithms”
2.1) Syntax “the lesson targets those students who know the syntax of programming in any language”
2.2) Structure “pseudocode examples for control structures”
3.1) Debugging “[...] you’ll learn more about debugging including what it is, why we use it, and what it looks like in action [...]”
4.1) Motivation/Engagement “this is a presentation held during the third r-ladies belgrade meetup about motivation for learning r programming [...]”
4.2) Time Management “self-regulated learning in higher education : strategies adopted by computer programming students”
4.3) Study Skills “promoting different kinds of learners towards active learning in the web-based environment”
4.4) Confidence “student perception of their online learning experience”
5) Tools for teaching “[...] a textook that teaches java programming in a novel and fun way using visuals puzzles and mysteries”
7) Feedback “interactive tutorial on basic programming concepts [...] there is a lot of feedback on how well they do”
8) Curriculum “the purpose of this curriculum is to provide an additional technology option”
9) Mathematics background “examine two sample programs and a sample game-programming math library [...]”
Note: challenges that did not have associated resources were omitted. We used sentences found in resource metadata such as title and descriptions, to provide
an overview of the content.

was ranked in seventh. Closing the 10 most used languages, with 6 occurrences, Syllabus with 5 occurrences, and Support
Javascript with 11 occurrences was in eighth, Perl with 4 Material (such as references) with only 2 occurrences. The
occurrences was in ninth and PHP with only 1 occurrence was other types were grouped in the Other, with 86 occurrences.
tenth. In addition to these languages, others were also used; Figure 2 shows the synthesis of challenges and resources types.
however, due to their lower occurrence, they were suppressed
and grouped in the “other” group. In the end, there were 201
occurrences. Finally, it was noted that 239 resources had no
language specified. Observe in Figure 1 a heat map that shows
the occurrence of each programming language in relation to
the mapped challenges.

Fig. 2. Heat Map of Students Challenges versus Resources Type

Fig. 1. Heat Map of Students Challenges versus Programming Languages

In general, we noted that the three most common types
of resources (Text, Image and Application) support virtually
B. The Resource Types (RQ1.2 ) all challenges, except the Scale problem. Although the other
We find many types of resources to support the challenges types of resources do not have such broad coverage, they are
of teaching and learning programming, solving the RQ1.2 . still able to provide at least one type of resource for each
The most common occurrence was the use of Images, with challenge.
total of 535 resources. Then, resources with type of Text
VI. AVAILABILITY OF O PEN E DUCATIONAL R ESOURCES
presented 475 occurrences. In this case, the resources were
TO P ROGRAMMING
in plain text format or more technical, such as HTML or
CSS files. Soon after, came resources of Application with 144 In this section we analyze RQ2 about the availability of
occurrences, containing software and executable programs. resources. Initially, to resolve this question, we extracted
The Course resources obtained 22 occurrences. Both Textbook information about localization of OER publishers, verifying
and Collection presented 8 occurrences, being a large part open whether the OER production is a concentrated effort in some
textbooks and online courses. From this, the individual other regions or if it is dispersed in initiatives around the world.
types of resources had a frequency equal to or less than 7. However, only 273 resources (6.84% of data set) showed this
This was the case of resources of the type Presentations that information. Nevertheless, it was possible to map that the
had 7 occurrences, Academic (such as thesis, research papers) production of OERs for programming follows a global trend.

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 19,2021 at 20:27:32 UTC from IEEE Xplore. Restrictions apply.
 Fig. 3. Initiatives of OERs on the Teaching and Learning of Programming around the World

We found that 21 different countries are producing OER did not have any type of license. That is, the authors did
with programming content, are they: Brazil, United States, not present what types of permissions users have rights over
South Africa, Portugal, Spain, Ireland, Switzerland, Germany, the resource (reuse, adaptation, etc.). In addition, 143 (3.58%)
Czech Republic, Poland, Romania, Ukraine, Russia, Turkey, resources had the value “not specified” in the license field
Kyrgyzstan, Pakistan, United Arab Emirates, India, Myanmar, and 99 (2.48%) presented an invalid value, such as numbers,
Malaysia, and Indonesia. Among these countries, United States invalid links or business names. In the other resources, 1,451
stood out the most in terms of resources production. Practi- (36.36%) had some permission derived from Creative Com-
cally all states near the East Cost have resources with focus mons (CC) and 202 (5.06%) had other types of permissions,
on teaching programming. Another country that also had a which we investigate next. In Figure 4 we show the dispersion
significant presence of OERs for programming was India, with between resources and licenses.
a wide dispersion of initiatives across the country.
We also noticed a significant difference between the re-
sources produced by both countries (United States and India)
related to the source. While American resources largely come
from universities and colleges, Indian resources come from
scientific sources, such as open access journals.
Other highlights are the countries of Brazil, Portugal and
South Africa. Brazilian initiatives are disseminated in five
different states via public universities. We found three ini-
tiatives in Portugal that have produced OERs with content
related to the teaching and learning of programming. Finally,
South Africa has two initiatives, one of which is located in
Cape Town, the birthplace of the OER declaration [26]. All
initiatives from different countries found in the data set are
shown in Figure 3.
Fig. 4. The Occurrence of Licenses
A. Licenses (RQ2.1 )
We also analyzed the availability of OERs through their Additionally, we analyze the permission level for each
licenses. Initially, our investigation would be based on the license. For this, we investigate three perspectives of an OER
identification of restrictive licenses, which could reduce the license:
use and reuse of OERs. However, another problem was • Reuse: Allows a particular resource to be used and reused
revealed. We noticed that 2,095 (52.50%) of the resources by other users. It is the most basic concept of an OER.

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 19,2021 at 20:27:32 UTC from IEEE Xplore. Restrictions apply.
 • Adapt: Ensures that a resource can be adapted or the permissions granted by each type of license is showed in
remixed. It is a more elaborate permission, allowing users Table VI.
to add, remove or unify portions of the resources.
• Change: Grant a user the ability to change the permission B. Languages (RQ2.2 )
when the resource is adapted. Thus, the user can adapt Finally, we also analyze the occurrence of languages among
the OER and define a license with a lower or higher level the resources in the data set. In total, we found 14 different
of restriction. languages. However, predominantly English was the most
used language among the analyzed resources, being present
In general, we noticed that the resources licensed by CC
in 97.95% of the resources. Soon after came the resources in
have a great support for use and adaptation. The coverage rate
Portuguese (1.38%), German (0.18%), Spanish (0.15%) and
is 87.78% by use and 85.24% by adapt. This means that more
Chinese (0.06%) languages. All other languages concentrated
than 85% of the data set resources that are under a CC license
on 0.03% of occurrence each, as shown in Figure 5.
can be used and adapted freely by users. While this, 73.26%
of CC licensed resources can be changed in your permissions
in case of adaptation. We also observed that the license CC
BY was the most common occurrence, while the CC BY-ND,
was the least recurring.
Relating others licenses, we found 9 different types. Ini-
tially, we highlight Custom licenses. In most cases, the re-
source has the value of “Custom License”, and it is not
possible to verify whether users can adapt and change these
resources. Next, we realized the licenses derived from Apache
and GNU. Both have characteristics inspired by the open
software model, and for that reason they have permission
to use, adapt and change. Similarly, two institutions that
inspired and contributed to the dissemination of the open
education movement have licensed resources. They are MIT
and Open University, even for their openness characteristics,
all resources licensed under these regulations have the option Fig. 5. Occurrence of Languages
to reuse, adapt and change. Licenses named Web Perl and
BSD also reported occurrences. However, in both cases it was VII. D ISCUSSION
not possible to verify whether they allow a license change
It is possible that many current challenges in the teaching
in derived resources. The Flicker web site also supports
and learning of programming can be reduced or overcome with
the dissemination of OERs and therefore has the option for
the support of OERs. As previously discussed, many problems
the user to license their images with permissive regulations.
for teachers and students already have resources that vary in
Finally, resources licensed as Public Domain were also found.
format and programming language.
In these cases, all permissions are enabled. The summary of
In this sense, OERs can serve in different perspectives.
Initially, we noticed the diversity of programming languages
in the resources. These languages support different paradigms,
TABLE VI
OER P ROGRAMMING L ICENSES such as structured and object-oriented programming; and di-
verse domains, for instance web and console. Thus, OERs
license Use Adapt Change provide materials for various contexts of students and teachers.
Creative Commons 87.78% 85.24% 73.26%
CC BY 62.01% 62.01% 62.01% Similarly, we found many types of resources, such as texts,
CC BY-NC 8.71% 8.71% 8.71% images, applications, courses, etc. This diversity of resources
CC BY-NC-ND 1.94% 0.00% 1.94% types can also facilitate the teaching and learning of program-
CC BY-NC-SA 13.25% 13.25% 0.00%
CC BY-ND 0.60% 0.00% 0.60% ming, adapted according to different needs.
CC BY-SA 1.27% 1.27% 0.00% Furthermore, it was possible to verify a dispersion among
Others 12.22% 10.71% 10.47% the main producers of OER in the global scenario. In short,
Custom 1.51% 0.00% 0.00%
Apache 0.06% 0.06% 0.06% we revealed 21 different countries that produce OERs with
BSD license 0.18% 0.18% 0.00% content of teaching and learning of programming. This set
Flicker 0.06% 0.06% 0.06% includes countries from different continents, cultures and lev-
GNU 0.48% 0.48% 0.48%
MIT license 0.48% 0.48% 0.48% els of application in programming. Thus, OERs production
Open University 0.67% 0.67% 0.67% also supports the argument that different countries are turning
Public Domain 8.71% 8.71% 8.71% their attention to developing the ability to program ( [4], [3]).
Web Perl 0.06% 0.06% 0.00%
TOTAL 100% 95.95% 83.73% Moreover, there is also the distribution of resources to assist
this.

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 19,2021 at 20:27:32 UTC from IEEE Xplore. Restrictions apply.
 A central aspect perceived was the gap among the OERs the name of a programming language in their description;
analyzed and the behavioral challenges. Although there are after, we gather random resources in the data set to ensure
resources to support such challenges, the occurrence in the that the materials could support the teaching and learning of
data set was small. Maybe, for being generic aspects faced by programming.
many students who are learning some subject, the resources
capable of supporting these challenges may be related in other VIII. C ONCLUSION AND F UTURE W ORK
areas/domains.
In parallel, two aspects called attention: first, the concen- We conducted an exploratory study on the availability
tration of resources in the English language; and second, the of OERs and their support for teaching and learning of
lack of allocation of licenses on resources. The restriction of programming. For this, we collected 3,990 resources with
the English language is an expected trend, and many of the different formats and characteristics from 11 distinct sources.
available materials adopt this language as a standard. However, More specifically, we observed the main challenges faced
the concentration of resources in a single language allied to by teachers and students, revealed the main programming
the lack of licenses can reduce the potential of OERs, creating languages and the most common material types presented
barriers where challenges should be overcome. It should also in OERs. Also, we also checked the availability of OERs,
be commented the extra layers that OERs have. One of these analyzing their dissemination worldwide and their level of
layers was presented when we analyzed the challenges and permission presented by the licenses.
their most common programming languages in OER (Figure The observed results showed us that there are many re-
1). Most of the resources did not specify the programming lan- sources available. Such materials have different formats and
guage or used generic description (“programming language”). have permissions that can facilitate their reuse and adaptation
Another layer is the lack of attribution of licenses by users by other users. On the one hand, OERs have great potential
(Figure 4). These combined factors end up reducing and to reduce the effort employed by students and teachers of
making the discovery, reuse and adaptation of resources more programming. On the other hand, despite this scenario, the
complex. real potential of OERs for programming has not been fully
This all serves as an important factor for the wider adoption explored yet. The problems are inherited, mainly, due to the
of OERs in the teaching and learning of programming. Un- difficulty of finding relevant resources for each user due to the
derstanding the difficulties and proposing solutions can unify wide range of options.
an intersection between teaching/learning programming and Considering this scenario, we believe that in the future
OERs. As revealed by this investigation, many of the current new mechanisms must be developed capable of supporting the
problems already have support from OERs. That remains is to teaching and learning of programming. In this sense, we ini-
make the discovery of resources a more practical process. tially suggest the construction of a guideline to programming
OER, highlighting aspects such as categories, metadata, and
A. Threats to Validity objectives. In addition, we suggest develop models to classify
Initially, a potential threat in this investigation is related OER according to their characteristics.
to the difficulty of generalizing the results. To avoid this, we
analyzed 11 different sources of OERs and analyzing resources ACKNOWLEDGMENT
with a list that has 752 names of currently known program- This study was financed in part by the Coordenação
ming languages. Another threat is derived from observations de Aperfeiçoamento de Pessoal de Nı́vel Superior - Brasil
of the challenges of teaching programming. We restricted our (CAPES) - Finance Code 001 and the Fundação de Amparo
analysis considering the problems cited by Medeiros et al. à Pesquisa do Estado de São Paulo (FAPESP 2018/26636-2).
[2]. However, probably there are problems that the authors did
not related. Thus, our investigation may have been affected. R EFERENCES
To avoid this, we searched for resources with close terms or
synonyms present in their metadata. In this sense, it should [1] A. Luxton-Reilly, Simon, I. Albluwi, B. A. Becker, M. Giannakos,
be noted that the contents of the OER were not analyzed. The A. N. Kumar, L. Ott, J. Paterson, M. J. Scott, J. Sheard, and
et al., “Introductory programming: A systematic literature review,”
analysis was based only on the description of the metadata in Proceedings Companion of the 23rd Annual ACM Conference
using computational processing. on Innovation and Technology in Computer Science Education,
Finally, the main limitation of our investigation refers to the ser. ITiCSE 2018 Companion. New York, NY, USA: Association
for Computing Machinery, 2018, p. 55–106. [Online]. Available:
difficulty of detecting resources. As there is still no effective https://doi.org/10.1145/3293881.3295779
way to identify OERs, during the collection and extraction data [2] R. P. Medeiros, G. L. Ramalho, and T. P. Falcão, “A systematic literature
and their analysis, it may be that relevant resources have been review on teaching and learning introductory programming in higher
education,” IEEE Transactions on Education, vol. 62, no. 2, pp. 77–90,
discarded (or yet, that irrelevant resources have been added). May 2019.
This is a real challenge for any OER investigation. Currently, [3] C. Duncan, T. Bell, and S. Tanimoto, “Should your 8-year-old
there are no effective search and classification criteria for learn coding?” in Proceedings of the 9th Workshop in Primary
and Secondary Computing Education, ser. WiPSCE ’14. New
resources. To mitigate this problem, we take two initiatives: York, NY, USA: ACM, 2014, pp. 60–69. [Online]. Available:
initially, we created the data set only with resources that had http://doi.acm.org/10.1145/2670757.2670774

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 19,2021 at 20:27:32 UTC from IEEE Xplore. Restrictions apply.
 [4] S. B. de Compuração, “Introductory programming: A systematic Technologies (LACLO), Oct 2018, pp. 292–298. [Online]. Available:
literature review,” in Diretrizes para ensino de Computação https://doi.org/10.1109/LACLO.2018.00059
na Educação Básica. Porto Alegre, Rio Grande do [20] H. Yildiz Durak, “The effects of using different tools in programming
Sul, Brasil: SBC, 2019, pp. 1–25. [Online]. Available: teaching of secondary school students on engagement, computational
https://www.sbc.org.br/documentos-da-sbc/send/203-educacao-basica/ thinking and reflective thinking skills for problem solving,” Technology,
1220-bncc-em-itinerario-informativo-computacao-2 Knowledge and Learning, Aug 2018. [Online]. Available: https:
[5] A. Ferchichi, “Teaching programming via specification, execution //doi.org/10.1007/s10758-018-9391-y
and modification of reusable components: An integrated approach,” [21] J. Hidalgo-Céspedes, G. Marı́n-Raventós, and V. Lara-Villagrán,
in Shifting Paradigms in Software Engineering, R. Mittermeir, Ed. “Playing with metaphors: a methodology to design video games for
Vienna: Springer Vienna, 1992, pp. 238–249. [Online]. Available: learning abstract programming concepts,” 06 2014. [Online]. Available:
https://doi.org/10.1007/978-3-7091-9258-0 27 http://dx.doi.org/10.1145/2591708.2602661
[6] A. Wakatani and T. Maeda, “Automatic generation of programming [22] S. Imberman and A. Fiddler, “Share and share alike:: Using creative
exercises for learning programming language,” in 2015 IEEE/ACIS commons licenses to create OER,” ACM Inroads, vol. 10, no. 2, pp. 16–
14th International Conference on Computer and Information Science 21, Apr. 2019. [Online]. Available: http://doi.acm.org/10.1145/3324887
(ICIS), June 2015, pp. 461–465. [Online]. Available: https://doi.org/10. [23] D. Hanna, A. Abhari, and A. Ferworn, “Comparing quantitative and
1109/ICIS.2015.7166637 comment-based ratings for recommending open educational resources,”
[7] R. Minelli, A. M. and, and M. Lanza, “I know what you did last summer: in Proceedings of the 20th Communications & Networking Symposium,
An investigation of how developers spend their time,” in Proceedings of ser. CNS ’17. San Diego, CA, USA: Society for Computer
the 2015 IEEE 23rd International Conference on Program Comprehen- Simulation International, 2017, pp. 4:1–4:10. [Online]. Available:
sion, ser. ICPC ’15. IEEE Press, 2015, p. 25–35. http://dl.acm.org/citation.cfm?id=3107979.3107983
[8] M. M. Arimoto, L. Barroca, and E. F. Barbosa, “Am-oer: An agile [24] O. Maksimenkova and V. Podbelskiy, “On practice of using open
method for the development of open educational resources,” Informatics data in construction of training and assessment tasks for programming
in Education, vol. 15, no. 2, pp. 205–233, 2016. courses,” in 2015 10th International Conference on Computer Science
[9] Unesco, “Forum on the impact of open courseware for higher Education (ICCSE), July 2015, pp. 233–236. [Online]. Available:
education in developing countries: Final report,” Unesco, Tech. Rep., https://doi.org/10.1109/ICCSE.2015.7250248
[25] E. Tovar, H. Chan, and S. Reisman, “Promoting merlot communities
jul 2002, acesso em 08 de outubro de 2019. [Online]. Available:
https://unesdoc.unesco.org/ark:/48223/pf0000128515 based on OERs in computer science and information systems,” in 2017
IEEE 41st Annual Computer Software and Applications Conference
[10] P. Boytchev and S. Boytcheva, “Innovative elearning technologies in
(COMPSAC), vol. 2, July 2017, pp. 700–706. [Online]. Available:
the open education era,” in Proceedings of the 20th International
https://doi.org/10.1109/COMPSAC.2017.290
Conference on Computer Systems and Technologies, ser. CompSysTech
[26] S. of declaration, “Cape town open education declaration: Unlocking
’19. New York, NY, USA: ACM, 2019, pp. 324–331. [Online].
the promise of open educational resources,” in Cape Town Open
Available: http://doi.acm.org/10.1145/3345252.3345300
Education Declaration, Cape Town, South Africa, 2007. [Online].
[11] C. Dichev and D. Dicheva, “Open educational resources in computer Available: https://www.capetowndeclaration.org/
science teaching,” in Proceedings of the 43rd ACM Technical
Symposium on Computer Science Education, ser. SIGCSE ’12. New
York, NY, USA: ACM, 2012, pp. 619–624. [Online]. Available:
http://doi.acm.org/10.1145/2157136.2157314
[12] T. Staubitz, R. Teusner, and C. Meinel, “Towards a repository
for open auto-gradable programming exercises,” in 2017 IEEE 6th
International Conference on Teaching, Assessment, and Learning
for Engineering (TALE), Dec 2017, pp. 66–73. [Online]. Available:
https://doi.org/10.1109/TALE.2017.8252306
[13] S. M. Dol and T. S. Indi, “Open educational resource (OER): C-
programming handout based lab session using moodle for engineering
education,” Journal of Engineering Education Transformations, vol. 0,
no. 0, 2017. [Online]. Available: http://www.journaleet.org/index.php/
jeet/article/view/111816
[14] A. S. Marcolino and E. Barbosa, “A survey on problems related to the
teaching of programming in brazilian educational institutions,” in 2017
IEEE Frontiers in Education Conference (FIE), Oct 2017, pp. 1–9.
[15] N. Piedra, J. Chicaiza, J. López, O. Martı́nez, and E. T. Caro,
“An approach for description of open educational resources based
on semantic technologies,” in IEEE EDUCON 2010 Conference,
April 2010, pp. 1111–1119. [Online]. Available: https://doi.org/10.
1109/EDUCON.2010.5492453
[16] V. Rodés-Paragarino, A. Gewerc-Barujel, and M. Llamas-Nistal,
“Use of repositories of digital educational resources: State-of-
the-art review,” IEEE Revista Iberoamericana de Tecnologias del
Aprendizaje, vol. 11, no. 2, pp. 73–78, May 2016. [Online]. Available:
https://doi.org/10.1109/RITA.2016.2554000
[17] M. Lokar and M. Pretnar, “A low overhead automated service for
teaching programming,” in Proceedings of the 15th Koli Calling
Conference on Computing Education Research, ser. Koli Calling ’15.
New York, NY, USA: ACM, 2015, pp. 132–136. [Online]. Available:
http://doi.org/10.1145/2828959.2828964
[18] R. P. Borges, P. R. F. Oliveira, R. G. da R. Lima, and R. W. de Lima,
“A systematic review of literature on methodologies, practices, and
tools for programming teaching,” IEEE Latin America Transactions,
vol. 16, no. 5, pp. 1468–1475, May 2018. [Online]. Available:
https://doi.org/10.1109/TLA.2018.8408443
[19] T. de Oliveira, D. Stringhini, and D. G. M. Corrêa, “Strategies focused
on the teaching of programming logic: A systematic review of brazilian
literature,” in 2018 XIII Latin American Conference on Learning

Authorized licensed use limited to: UNIVERSITY OF CONNECTICUT. Downloaded on May 19,2021 at 20:27:32 UTC from IEEE Xplore. Restrictions apply.