2020 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE) | 978-1-7281-6942-2/20/$31.

00 ©2020 IEEE | DOI: 10.1109/TALE48869.2020.9368459

Bernard Yett Caitlin Snyder

Institute for Software Integrated Systems Institute for Software Integrated Systems
Vanderbilt University Vanderbilt University
Nashville, USA Nashville, USA
[email protected] [email protected]

Nicole Hutchins Gautam Biswas

Institute for Software Integrated Systems Institute for Software Integrated Systems
Vanderbilt University Vanderbilt University
Nashville, USA Nashville, USA
[email protected] [email protected]

Abstract Computational thinking (CT) skills are necessary of conceptual definitions, which is unlikely to produce a deep
for solving the real-world problems of today and are therefore understanding of these concepts that addresses the applicabil-
being incorporated into K-12 curricula. Cybersecurity is of sim- ity of cybersecurity concepts. Examples of such problems in-
ilar importance; however, it can be difficult for young learners clude students failing to recognize points of vulnerability be-
to grasp the required concepts and use them to construct mean- tween two communicating systems, and the need for reliable
ingful algorithms. We discuss our approach that combines a encryption methods intended to secure communication of in-
hands-on robotics platform with a block-based programming formation [5]. Similar problems arise in computational think-
environment to facilitate the learning and application of cyber- ing (CT), which requires gaining conceptual knowledge of
security and CT concepts. Throughout a week-long interven-
logical thinking, algorithmic thinking, pattern recognition, ab-
tion, high school students were introduced to cybersecurity and
CT and given the opportunity to apply this knowledge in a col-
straction, generalization, evaluation, and automation along
laborative setting to solve security problems on the robotics plat- with procedural knowledge for problem decomposition, arte-
form with instructor and peer support. A series of competitions fact creation, testing, debugging, and iteration [6].
between groups of students further motivated students to trans- Learning algorithmic thinking and programming is argua-
late their learned concepts to practice, often leading to break- bly a core requirement for K-12 students. AP CSP courses are
throughs as students incorporated new algorithms into their ex- a mechanism for accomplishing this [2], but successful curric-
isting projects to counteract previous security flaws. We present
ulum design requires forethought by educators and research-
evidence of the learning behaviors of several such groups
ers. It can be challenging to transfer complex ideas - such as
through mixed-method case studies integrating data collected
from learning performance, collaborative discourse, and analy- cyber-attacks and cyber-defenses - into usable granular
sis of program development. We discuss the impact of this ap- knowledge segments and their implementation in programs.
proach on cybersecurity and CT learning and then present fu- In addition, programs generated should be based on sound
ture directions for this work. knowledge of underlying cybersecurity principles and pro-
gramming practices, such as developing an algorithm, trans-
Keywords K-12 STEM education, educational robotics, cy- lating key ideas into programming constructs, and debugging
bersecurity, computational thinking, collaboration any issues that arise [7]. The cognitive load associated with
learning and combining these concepts for application tasks
I. INTRODUCTION can be too much for K-12 students without assistance.
Given the importance of computational platforms in our Our solution to these issues was to develop a block-based
everyday lives, students need to be introduced to cybersecu- programming environment (BBPE), NetsBlox [8], that is
rity principles while still in high school. There are also in- combined with a robotics platform, Roboscape [9], to situate
creasing demands for jobs in this field to combat the prolifer- learning and to link concepts and practices. The benefits of
ation of attacks, and to save companies billions of dollars per -documented in K-12 environments, often
year in cyber theft [1]. This need has been reflected in the de- leading to greater learning gains and increased interest in fu-
sign of AP Computer Science Principles (AP CSP) courses - ture computing courses [10, 11]. Similarly, educational robots
the latest suggested curriculum includes key security concepts have been shown to positively influence student learning of
[2]. Mishra et al. [3] specifically targeted this need by integrat- programming and other complex topics, particularly when in-
ing cybersecurity first principles into existing AP CSP corporated into a framework containing competitions and col-
courses. Another example is the series of GenCyber summer laboration [12]. The value of collaboration has been estab-
camps [4], which successfully brought cybersecurity to the lished in terms of assessment scores [13] and development of
forefront for both teachers and students. problem-solving skills [14]. By combining these elements, we
However, cybersecurity concepts are often abstract and have developed an intervention to encourage the learning of
difficult to understand from an operational viewpoint. The in- cybersecurity and CT concepts and practices in an engaging
ability to make these concepts explicit and related to everyday and collaborative manner. To evaluate the effects of this plat-
computing leads to this subject being taught by memorization form, we propose the following research questions:

978-1-7281-6942-2/20/$31.00 ©2020 IEEE December 8–11, 2020, Online

IEEE TALE2020 – An International Conference on Engineering, Technology and Education
Authorized licensed use limited to: La Trobe University. DownloadedPage 213 14,2024 at 00:23:08 UTC from IEEE Xplore. Restrictions apply.
on December
 1. Does our intervention produce learning gains in cyber- students had obtained the requisite CT knowledge to explain
security and CT concepts? and discuss key principles within the field.
2. One aspect that benefits student learning of CT and cyber-
learning gains to? security while using a robotics platform is a collaborative en-
vironment where students can work together to discuss and
3. How does collaborative discourse during program- overcome their difficulties. Collaboration has a variety of
ming combined with learning gains impact the transla-
proven benefits, including providing a support structure when
tion of cybersecurity concepts into code? new challenges arise [13] and a demonstrated correlation with
RQ1 provides an overall measure of the effect of the inter- assessment scores [23]. Within the robotics domain, Keith,
learning performance in cybersecurity Sullivan, and Pham [24] found that middle school girls were
and CT. RQ2 and RQ3 provide a deeper dive into understand- more likely to take on problems with increased difficulty when
ing the processes that support that learning performance. A they were collaborating well. Nag, Katz, and Saenz-Otero [25]
primary data source is the cybersecurity and CT pre- and post- presented a robotics framework that promoted collaboration
test scores. As part of our case study analyses, we answered in several ways - within matches to accomplish objectives, al-
RQ2 by examining the collaborative roles that students took liances with other teams to increase scores for all participants,
on by examining the activity logs of individual students and and communication via online forum. A final example comes
linking them back to the learning gains. To answer RQ3, we from Karaman et al. [12] as they reported on the pedagogy of
adopted a collaborative discourse analysis framework com- a STEM robotics program that combined collaboration with
bining the ICAP (Interactive- Constructive- Active- Passive) focuses on robotics programming and project-based learning.
[15] and social modes frameworks [16] to evaluate differences
Building upon these solutions, we hope to contribute a bet-
ter understanding of how students work together in these col-
haviors as they implemented key cybersecurity concepts.
laborative robotics environments. This comes from noticing
The paper is organized as follows. In Section II, we pro- that studies in the domain often present only evidence of stu-
vide background on cybersecurity, robotics, and collaboration dent learning (through summative assessments) or qualitative
in education. In Section III, we outline our platform and cur- results (from student self-assessments). These are tools of sig-
riculum, our analysis framework, and the study that was con- nificant value, but we believe that incorporating programming
ducted. Our results answer the research questions as described actions and the assessment of student discourse during collab-
above and are presented in Section IV. Section V presents our orative activities with learning results can provide additional
discussion, conclusion, and directions for future work. insights. Through the rest of the paper, we describe our plat-
form, additional analyses, and the results from these analyses.
II. BACKGROUND
Cybersecurity concepts can be difficult for students to III. METHODS
grasp, especially when they are asked to apply those concepts A. Intervention and Participants
in areas such as secure programming, cyber-attacks and cyber- Thirty-eight high school students completed our week-
defenses, and cryptography. To accommodate this, research- long, 6-hours a day, in-person cybersecurity camp that was
ers and teachers often turn to hands-on learning activities [17]. based on a robotics platform. The students were 50% female,
Jin et al. [18] created a set of games as part of GenCyber that 50% rising seniors (the remainder were a mix of other high
targeted the teaching of cybersecurity principles and safe school grade levels), and 92% had previous programming ex-
practices to high school students. Mirkovic et al. [19] present perience. The intervention was designed to target a variety of
and compare three example implementations - International CT (variables, conditionals, loops, functions, and lists) and cy-
Capture the Flag from University of California at Santa Bar- bersecurity (basic attacks, attack detection, eavesdropping,
bara as a cybersecurity team-based competition, SEED labs as brute force attacks, encryption, and decryption) concepts at
hands-on exercises, and Control-Alt-Hack as a tabletop card various points throughout [26].
game about ethical hacking. The first two interventions fo-
cused on practical implementations, while the final one tar- We implemented an evidence-centered design [27] ap-
geted an increase in theoretical understanding. proach to curriculum and assessment development to map tar-
get concepts to questions administered in the pre-post assess-
Not enough is known about the ability of students to trans- ments. For instance, the CT section of the pre-post-tests ad-
late conceptual knowledge of complex topics like cybersecu- dressed loops and conditionals, both of which featured
rity into an algorithmic structure before implementing it to throughout the intervention. The concepts were selected to
solve problems in a hand on format. Only occasionally have match with desired learning objectives from sources such as
robots been a primary platform for teaching cybersecurity. Ex- AP CSP guidelines [2] and the K-12 Computer Science
amples include Señor Robot and its successor Frogbotics [20], Framework [28] and some of the multiple-choice pre-post as-
which used unplugged activities with students as robots to in- sessment questions were pulled from these sources to evaluate
troduce cybersecurity and programming concepts to 3rd grade the effectiveness of our intervention. Some questions had po-
students. However, the use of robotics to support STEM and tential half-credit or had multiple parts, but all were reduced
CS learning is quite common. Nugent et al. [21] studied results to a scale from 0 for completely incorrect and 1 for completely
from a series of robotics camps featuring over 5000 students correct. Additional details can be found in previous work [26].
ages 9-14 and found that the hands-on nature of robotics ac-
tivities led to STEM knowledge growth, particularly in terms Students worked individually on the first day of the inter-
of computer programming. Grover [22] detailed the effects of vention. They were introduced to the BBPE and CT concepts
a robotics framework on increasing CT knowledge of students via interactive games and applications, alternating between
following along with an instructor and having additional time
to implement new ideas on their own. Day 2 was dedicated to

978-1-7281-6942-2/20/$31.00 ©2020 IEEE December 8–11, 2020, Online

IEEE TALE2020 – An International Conference on Engineering, Technology and Education
Authorized licensed use limited to: La Trobe University. DownloadedPage 214 14,2024 at 00:23:08 UTC from IEEE Xplore. Restrictions apply.
on December
 increasing familiarity with the robotics platform. Students work are presented in Table I. We postulate that groups en-
learned the syntax for commands and applied them to create a gaging in primarily Interactive discourse while discussing key
basic manual driving program. Students separated themselves features of the intervention are the most likely to demonstrate
into dyads or triads on their own, with instructors only inter- improvements in pre-post-test results related to those features.
vening to facilitate group formation. The remaining days of Meanwhile, groups who primarily communicated using Con-
the intervention involved developing a sequence of cyber-at- structive or Active discourse were not achieving true collabo-
tacks and cyber-defenses of increasing complexity, with stu- ration and were less likely to show pre-post-test gains. Dis-
dents eventually using advanced encryption and automated tinctions within the Interactive and Constructive codes pro-
key-generating programs to defend against the attacks of their vided further explanation for the group behaviors and results.
peers. Collaboration amongst group members was encouraged
for each of these projects. Groups worked together on a project In choosing groups for further analysis through case stud-
using a collaborative editor that allowed each student to de- ies, the first step was to narrow down from the original dyads
velop and modify the code for the assigned challenge or com- and triads. We decided to focus only on dyads or only on triads
petition. Each student participated in building parts of the pro- in order to provide consistency across groups during analysis.
gram by turn and took responsibility for monitoring a set of The result was the discovery that four of the six triads had
program functions during testing and competitive play. enough data for us to review, leading to the decision to analyze
triads for this paper. An additional benefit of studying triads
Student dialog was recorded using OBS. These recordings instead of dyads was the richer variety of possible roles stu-
included screen capture, audio, and video sources. Though au- dents could play within a group.
dio was the primary data source, others were used to cross-
reference with programming action logs and discourse seg- TABLE I. DISCOURSE FRAMEWORK CODES AND DESCRIPTIONS
ments for improved accuracy. For this analysis we focus on
two half-days during the intervention: the morning of day Codes Description
three and the morning of day four. The selected half-days tar-
Interactive Conflict- Multiple group members are programming
geted applications of a majority of the CT and cybersecurity Oriented Consensus and/or discussing the model, but they are
concepts evaluated in the pre- and post-test, while also requir- Building disagreeing over what should be done next
ing advanced problem solving as groups translated new cyber- Multiple group members are programming
Interactive Integration-
security ideas to extend their basic robot driving programs. Oriented Consensus
and/or discussing the model, but they are
This was important when compared to other time periods that not sure what to do next and no student is
Building
taking a strong position
involved competing with other groups of students, as during Multiple group members are programming
those time periods fewer original programs were created. and/or discussing the model, then one group
Interactive Quick
member makes a suggestion and other
The morning of day three featured three distinct tasks - a Consensus Building
group members accept their idea with no
simple cyber-attack, an attack detection program, and a denial further discussion
of service attack. The simple cyber-attack only required Multiple group members are programming
knowledge of variables and basic attacks, so it was not directly Interactive Elicitation and/or discussing the model, and one
related to any assessment questions. However, the attack de- student questions another student
tection program required an understanding of variables, con- Only one student is programming and/or
Constructive
discussing the model, and that student is
ditionals, detection, and eavesdropping, mapping directly to Externalization
narrating their actions
the conditionals question from the CT section. Likewise, the Only one student is programming and/or
denial of service attack mapped to variables, loops, denial of Constructive discussing the model, and that student asks
service attacks, and eavesdropping; as a result, this task is re- Elicitation a question of the other group members that
lated to the loops question from the CT section and the DoS they do not sufficiently answer
One or more students are reading from the
question from the cybersecurity question. Active Externalization
materials without contributing new ideas
Encryption and encryption breaking were the two main No students from the group are
Passive
programming and/or discussing the model
tasks during the morning of day four of the intervention. The
One or more students are having a
encryption program required students to understand variables, Off Task
discussion not related to the task at hand
loops, functions, and of course encryption. Therefore, this task
related to the loops question from the CT section and the en-
cryption and decryption question from the cybersecurity sec- The next step involved pre-processing of the available vid-
tion. The encryption breaking task was the most involved of eos in order to identify segments of discourse related to cyber-
any of the projects we examine here, requiring knowledge of security discussion and programming. This was accomplished
variables, conditionals, loops, lists, decryption, eavesdrop- by taking advantage of the speech recognition capabilities of
ping, and brute force attacks. The task maps to both CT ques- free speech-to-text software. For two of the four groups, this
tions along with the encryption and decryption question and sufficiently identified at least one relevant discourse segment
the brute force attacks question from the cybersecurity section. across both days. For the other two groups, the audio quality
was too poor for appropriate segments to be identified. It was
B. Analysis Approaches left to the authors to manually listen to the videos to identify
We make use of a discourse analysis framework that com- appropriate segments. Complete segments were chosen such
bines the Interactive-Constructive-Active-Passive (ICAP) that they were ten minutes long, generally starting with a key
framework [15] and the Weinberger & Fischer [16] frame- word related to the current project. On day three, words such
work related to social modes during knowledge construction
to create a comprehensive method for collaborative discourse groups to indicate their creation of an attack detection pro-
analysis that provides insight into the engagement of students gram. On and
[29]. The nine different components of the resulting frame-

978-1-7281-6942-2/20/$31.00 ©2020 IEEE December 8–11, 2020, Online

IEEE TALE2020 – An International Conference on Engineering, Technology and Education
Authorized licensed use limited to: La Trobe University. DownloadedPage 215 14,2024 at 00:23:08 UTC from IEEE Xplore. Restrictions apply.
on December
 were used to indicate that the groups were devel- along with the average results as comparison. Group 1 was the
oping either encryption programs or brute force attacks. only group to consistently rank as high performers, gaining
CT knowledge even as they slightly regressed in cybersecu-
The final step was to observe and transcribe the sections rity. Group 2 was consistently strong in CT to remain as high
of the video recordings that were identified through the prior performers from pre- to post-test and showed improvement in
processing. The transcription process was split between au- cybersecurity. The prior knowledge of Group 3 was signifi-
thors, with each author being solely responsible for a group. cantly lower than their peers, though they showed improve-
This consideration was made to maintain consistency within a ment in CT and to a lesser extent cybersecurity. Meanwhile,
group in terms of properly identifying which group member Group 4 deteriorated in performance, starting as high perform-
was speaking at a given time. One author proceeded to code ers when they began the intervention but finishing as low per-
the discourse segments using the framework established formers.
above. Some conversations were initiated or heavily influ-
enced by instructors or students from a separate group. During When examining each question related to the two specific
the coding process, these conversations are coded and counted half-days, more patterns emerge (Table III). For the questions
separately from conversations between only group members. related to CT skills, Group 1 seemed to gain an understanding
of both loops and conditionals through their time spent collab-
Netsblox (Fig. 1) [8] has an extensive logging structure orating. Group 2 already had a strong understanding of each
that provides information such as the type of action, user iden- skill and was not able to demonstrate any improvement. Group
tification, project identification, and a Unix timestamp. The 3 showed an overall improvement in each area, seeming to
most common actions involved adding, removing, modifying, close the initial gap between themselves and their peers.
or moving blocks within a program, inserting variables or val- Group 4 demonstrated no additional improvements of their
ues within blocks, or executing scripts or the entire program. baseline knowledge when entering the intervention.
For this analysis, collaborative programming contribution was
evaluated by generating a raw count of actions taken by each TABLE II. OVERALL PRE-TEST AND POST-TEST RESULTS
student in our chosen groups throughout all collaborative days
of the intervention, plus a focused look at the mornings of days Overall
Group Group Group Group
Section Average
three and four. In addition, we isolate model-building actions (SD)
1 2 3 4
(MBAs) actions that contributed to creating or modifying
CT - pre 0.70 (0.32) 0.75 0.92 0.42 0.75
the program to further examine each group member s con-
tribution to the final group code for each task. We hypothesize CT - post 0.84 (0.20) 1 0.92 0.75 0.64
that greater programming contribution by each group member Cybersecurity
may support more interactive collaborative dialogue. 0.64 (0.22) 0.92 0.33 0.5 0.75
- pre
Cybersecurity
0.79 (0.18) 0.83 0.67 0.58 0.75
- post

TABLE III. SELECTED QUESTION RESULTS

Total Group 1 Group 2 Group 3 Group 4
Question
(Pre/ (Pre/ (Pre/ (Pre/ (Pre/
Topic
Post) Post) Post) Post) Post)
0.75/
Loops 0.83/1 0.83/0.83 0.5/0.83 0.83/0.83
0.86
0.66/
Conditionals 0.67/1 1/1 0.33/0.67 0.67/0.44
0.83
Denial of
0.34/
Service 0.67/1 0/1 0/0.33 0.67/1
0.87
Attacks
Brute Force 0.87/
1/0.67 0.67/1 1/1 1/1
Attacks 0.89
Encryption
0.5/
and 1/1 0/0 0/0 0.67/0.67
Fig. 1. Example simple cyber-attack within Netsblox 0.68
Decryption
We also used the log data as verification of what occurred
during the on-screen programming of students. This data From a cybersecurity perspective, the largest growth over-
aligned with the to provide a deeper un- all occurred for the question on basic attacks (Table III). The
derstanding of how students went about their tasks. We hy- correct answer for that question required a conceptual under-
pothesize that groups of students who had productive discus- standing of denial of service attacks, which Groups 1, 2, and
sions and asked insightful questions of their peers or instruc- 4 all demonstrated. Group 3 showed the only real lack of im-
tors also demonstrated strong group programming behaviors. provement on that question, though one student answered it
Taken together, we put together a more complete picture of correctly on the post-test after all answered incorrectly on the
how groups developed their attack and defense algorithms and pre-test. Group 2 showed moderate improvement on brute
worked together to overcome difficulties. force attacks, but Group 1 indicated increased confusion. No
IV. RESULTS AND ANALYSIS changes were demonstrated at a group level for the question
related to encryption and decryption. The lack of improve-
A. Summative Results ment on these two questions was common amongst all stu-
We begin our analysis by answering RQ1 (Does our inter- dents participating in the intervention.
vention produce learning gains in cybersecurity and CT con-
cepts?). Table II presents the overall test scores for our triads,

978-1-7281-6942-2/20/$31.00 ©2020 IEEE December 8–11, 2020, Online

IEEE TALE2020 – An International Conference on Engineering, Technology and Education
Authorized licensed use limited to: La Trobe University. DownloadedPage 216 14,2024 at 00:23:08 UTC from IEEE Xplore. Restrictions apply.
on December
 B. Case Studies TABLE IV. GROUP 1 SEGMENT INTERACTIVE ELICITATION

We continue our analysis by examining results related to

Speaker Discourse
RQ2 (
learning gains to?) and RQ3 (How does collaborative dis- Do you think maybe our encryption changes everytime
S1
course during programming combined with learning gains
impact the translation of cybersecurity concepts into code?) S3 Ask them about why our key says false
in the form of case studies of our four highlighted groups.
S1 Yeah
a) Group 1: Student 1 (S1) started with low CT
S2 S1, why isn't it working with or without the encryption?
knowledge (half-credit for the loops question, incorrect
Because they said once you start using encryption, the
answer to the conditionals question), but improved by giving S1
robot expects the encryption
correct responses to both questions on the post-test. S1 also
S2 Ohhhhh
answered all cybersecurity questions correctly on both pre-
and post-test. S2 answered all questions observed here S1 So now it can't listen to normal commands
correctly on the pre-test; however, S2 answered incorrectly
on the brute force attacks question on the post-test, regressing
slightly. S3 answered all CT questions correctly on the pre- b) Group 2: S4 answered both CT questions correctly
test, and only answered incorrectly on the question related to on the pre-test, but only achieved half-credit on the loops
DoS attacks on the cybersecurity section. This misconception question on the post-test. For the cybersecurity questions, S4
appears to have been cleared up during the intervention, as S3 answered incorrectly on both the denial of service attacks and
had a perfect score amongst these questions on the post-test. encryption/decryption questions. Though S4 answered
correctly for the former on the post-test, they were unable to
Throughout the intervention, S1 took the majority of total do the same for the latter. S5 followed the same pattern as S4
actions and total model-building actions (MBAs; 2355 and on the cybersecurity section, improved from half-credit to full
1312, respectively). S1 was supported by S2 (1617 actions credit on the loops question, and answered correctly on the
with 985 MBAs) and S3 (1129 actions with 772 MBAs), in- conditionals question each time. S6 answered perfectly on the
dicating that all groups members were contributing to the pro- CT questions on both pre- and post-test. However, S6 also
gram code throughout the intervention. Similar ratios were answered all three cybersecurity questions that we are
found during the morning of day three (383 actions for S1, analyzing here incorrectly on the pre-test. On the post-test, S6
232 for S2, and 222 for S3) and the morning of day four (311, answered the DoS and brute force attacks questions correctly,
245, and 240 respectively). This action data combined with but like the other group members still could not respond
the pre-post-test results indicates that S1 was the group leader properly to the encryption and decryption question.
but was well-supported by S2 and S3, as each student con-
sistently contributed to the programming work while demon- S4 took the fewest group actions of any student
strating their understanding of concepts. This combination of throughout the intervention, with only 53 overall (30 MBAs)
learning performance and supportive group contributions including 2 on the morning of day three and 12 on the
serves as an affirmation of the high performing nature of this morning of day four. S5 (with 1139 group actions and 645
group and their success in collaboration. MBAs overall along with 189 and 186 actions on the
mornings of days three and four respectively) and S6 (with
This is further confirmed through an analysis of their col- 2002 groups actions and 1447 MBAs overall along with 285
laborative dialogue. While the group is considered Interac- and 364 actions on the mornings of days three and four
tive throughout most of their conversations, S1 led the discus- respectively) were much more involved. Despite these action
sion and guided the group. As seen in the segment below (Ta- disparities, each group member demonstrated a shared
ble IV), the group participated in Interactive Elicitation dis- understanding of DoS attacks as a result of the intervention;
course. As they tried to determine why the robot is not react- each member answered incorrectly on the pre-test but
ing to the commands sent, S1 elicited reactions from the other correctly on the post-test. This ties back into RQ2, as this
group members by suggesting a possible cause while S2 and knowledge acquisition was likely caused by the collaborative
S3 asked for help from other sources. S1 explained the idea process combined with learning opportunities. S6 was clearly
to the other members, which they accepted as the cause, but the action leader with some support from S5, which stands in
they still sought confirmation and assistance from the camp contrast to the pre-existing CT knowledge of all three
facilitators. Approximately a quarter of the conversation dur- students, as they each scored above the average of 0.7 (Table
ing this time was influenced by outsiders from the group. II) on the pre-test.
During the ten-minute window surrounding this conver- In terms of collaborative dialogue, S5 and S6 led the
sation, S1 (with thirty-two actions), S2 (with twenty-nine ac- discussion about the project while S4 followed along. In the
tions), and S3 (with twenty-two actions) all contributed well. segment below (Table V), S6 began by reading an
S3 taking the fewest number of actions was somewhat mis- explanation from the handouts provided to students which
leading, as each of the other two students engaged in the un- detailed the intended usage of newly introduced blocks. S5
productive behavior of repeatedly executing a script with no and S6 proceeded to an interactive integration-oriented
changes in between. Overall, all three students were success- consensus discussion and then explained what was going on
fully modifying the program at various times within this win- to S4. This group also barely interacted with outsiders to the
dow. Their collaboration and group-wide participation appear group during the observed periods, as only about 10% of their
to answer RQ2 and RQ3, as they were successful at discuss- discourse was influence by those outsiders.
ing and then applying the concepts presented here.

978-1-7281-6942-2/20/$31.00 ©2020 IEEE December 8–11, 2020, Online

IEEE TALE2020 – An International Conference on Engineering, Technology and Education
Authorized licensed use limited to: La Trobe University. DownloadedPage 217 14,2024 at 00:23:08 UTC from IEEE Xplore. Restrictions apply.
on December
 S4 expressed a desire to follow along and contribute to 50 during the morning of day three, and 86 during the
the group dynamic; however; they took no actions within the morning of day four. These results correspond with the
ten-minute window containing this conversation. S5 and S6 quantitative results of the pre-post-tests, as S8 showed strong
were both significantly contributing to the model during that CT skills entering the intervention plus improvement in
window, with forty-two actions taken by S5 and thirty-four cybersecurity.
taken by S6. None of these actions were simply long
Similar to the other groups, the action leader was also the
sequences of executing scripts with no changes in the middle.
group member primarily leading the discussions. In the
Instead, each student was modifying the program by adding
segment below (Table VI), which falls under the Interactive
and removing blocks, creating new variables, changing the
Integration-oriented consensus building mode, we can see
values contained within existing blocks, and rearranging
the group expressing confusion about what to do next. S8
existing blocks.
conceptualized the problem for the group, explaining what
TABLE V. GROUP 2 SEGMENT INTERACTIVE INTEGRATION- the end goal was and suggesting a possible plan for
ORIENTED CONSENSUS BUILDING implementation. S9 contributed minimally, asking what to
do or to agreeing with what S8 said. S7 also contributed
Speaker Discourse minimally to this discussion but then attempted to start
programming, actively implementing the idea conceptualized
There are 2 types of messages - the robot command is
S6 issued by others, and the robot message is issued by the by S8. Approximately a third of the conversation surrounding
robot this presented segment was influenced by outsiders to the
S6
For example, the get range command is sent to robot, the group; however, other segments from the same group were
command value of robot command is get range much less influenced.
S5 So we have to set up one of these for each robot
TABLE VI. GROUP 3 SEGMENT INTERACTIVE INTEGRATION-
S6 Robot *space* command ORIENTED CONSENSUS BUILDING
S5 Oh ok
Speaker Discourse
S5 So here we're listening to a command

S4 Uh huh S7

S5 And here is when you receive the command S8

S4 Uh huh S7 Ok you tell me what to do

S5 S9 Yeah tell me too
S6 Yeah figure out a way for us to
S8
know that the hacker is hacking us
S4 So whenever the robot receives more commands than we
S8
S4 I got this
S8

S9 Ok so how do we do that?
c) Group 3: S7 earned half-credit on the loops question
and no credit on the conditionals question on the pre-test. S8 That's hard
They performed better on the post-test, answering correctly
S9 That's hard right?
on the loops question and earning one-third of the points on
the conditionals question. S7 only answered correctly on the S8 That's the hard part
brute force question amongst the cybersecurity questions on
both pre- and post-test. S8 improved from half-credit on the
loops question to full credit by the post-test, and answered The programming actions data indicates that the group
correctly on the conditionals question each time. S8 answered took sixty-four actions during the ten-minute window
correctly on the DoS question on the post-test after failing to including this conversation. As indicated by the conversation,
do so on the pre-test, was consistently correct on the brute S8 was originally the only student significantly
force attacks question, and was consistently incorrect on the programming. However, S7 quickly began to contribute, such
encryption and decryption question. S9 consistently earned that by the end of the segment each of the two students had
half-credit on the loops question, and improved from no taken an approximately equivalent number of productive
credit to two-thirds credit on the conditionals question. They model-building actions. On the other hand, S9 only had four
had the same results as S7 on the cybersecurity questions. programming actions of those original sixty-four. The
sporadic collaboration and single leader situation of this
In terms of actions, S8 was the clear leader of the group, group provides some insight into RQ3 when combined with
performing 1774 actions (1157 MBAs) during group projects their poor pre- and post-test results.
throughout the week along with 305 during the morning of
day three and 276 during the morning of day two. Some d) Group 4: S10 started by answering all five questions
programming support was provided by S7 with 856 actions under review here correctly on the pre-test, but regressed to
overall (382 MBAs), 132 during the morning of day three, one-third of the credit on the conditionals question on the
and 140 during the morning of day four. However, S9 was post-test. S11 earned half-credit on the loops question and no
only minimally active, with 348 actions overall (171 MBAs), credit on the conditionals question on both pre- and post-tests.
They were able to improve on the DoS attack question from

978-1-7281-6942-2/20/$31.00 ©2020 IEEE December 8–11, 2020, Online

IEEE TALE2020 – An International Conference on Engineering, Technology and Education
Authorized licensed use limited to: La Trobe University. DownloadedPage 218 14,2024 at 00:23:08 UTC from IEEE Xplore. Restrictions apply.
on December
 pre- to post-test, while answering correctly on the brute force by S12. They added and removed blocks from the program
attacks question and incorrectly on the encryption and and modified existing blocks with new values. Meanwhile,
decryption question each time. S12 answered all questions S10 only completed a handful of actions, while S11 mislead-
correctly in both sections on both pre- and post-tests. ingly had the largest quantity of actions by a large margin.
However, these actions involved executing either sections of
S10 took 1377 actions (628 MBAs) throughout the group the program or the entire program repeatedly with no changes
portion of the intervention while S11 contributed 806 (359 to the program in the middle - an unproductive behavior.
MBAs) and S12 contributed 563 (334 MBAs). However,
unlike every other group, the relative number of actions from V. DISCUSSION AND CONCLUSIONS
each student changed for our analysis period. S11 contributed
Group 1 was consistently the best performing group of
184 actions on the morning of day three and 167 actions on these four throughout the intervention. They collaborated in
the morning of day four to lead this group, while S10 was an interactive manner, all worked together on their program-
down to 137 and 105 respectively. S12 only contributed 47 ming projects, were high performers in all areas of the pre-
actions on the morning of day four, but was a positive test, and only answered a single question incorrectly on the
contributor with 124 group actions on the morning of day post-test out of the questions we focused on here. Even S3 was
three. Based upon observations from the video, we realized able to increase their understanding of DoS attacks through
that an additional 233 actions taken by S12 on what appeared collaboration. The consistency of programming actions during
to be a solo project were actually positively contributing to a the mornings of days three and four for all three students sup-
separate project with the overall group goal still in mind. If port the claim that Interactive discourse along with pre- to
post-test gains are strong indicators of programming success.
Group 2 appeared to gain a shared understanding of denial
knowledge of CT and cybersecurity demonstrated by both of service attacks and did well on the pre- and post-tests out-
S10 and S12, the fewer actions overall compared to many side of the encryption and decryption question. However, de-
other groups was not expected. These results could have spite all three students performing well on the CT section of
simply indicated that this group did not need to commit as the pre-test, S4 seemed to largely disengage from the other
many actions to processes like trial-and-error, leading to group members. It was surprising that S5 and S6 could not
more efficient program creation. However, when combined obtain an understanding of encryption and decryption, as they
with the few MBAs and other results, this appears unlikely. performed a similar number of actions during each morning
of the intervention studied here and were noteworthy for their
As seen in their constructive elicitation discourse
Interactive discourse. Future work could include coherence
segment (Table VII), S11 was primarily conversing with an
analysis [30] of the modeling actions to evaluate the effective-
instructor. They asked clarifying questions and tried to ness or ineffectiveness of each programming action during
determine the appropriate path forward for defending the
tual knowledge of the cybersecurity concepts into code.
collaborative discourse from this group did require the
intervention of an instructor they were the only group with
a majority of their conversations influenced by outsiders on the pre-test. S8 took charge of the group in terms of dis-
during the transcribed segments. course and programming actions, resulting in strong improve-
ment on the post-test as they answered four out of the five
TABLE VII. GROUP 4 SEGMENT CONSTRUCTIVE ELICITATION questions correctly. The other two group members minimally
contributed in programming and seemed to defer to S8, ulti-
Speaker Discourse mately not demonstrating much improvement. This highlights
the importance of hands-on participation S7 and S9 from this
S11 Wait the key to another key? group along with S4 from Group 2 struggled to demonstrate
So say if you want to hit a
improvement and took fewer actions than many of their peers.
hacked and you want to set another key, you have to the Though the Interactive discourse within this group could have
Instructor had an impact as S8 attempted to teach S7 and S9, the lack of
next time remember that you were encrypted. So set up
another keyboard key to actually change the key programming effort interrupted such a possibility.
S11 Finally, Group 4 demonstrated an impressive amount of
number or would we still do the pick random?
Instructor You can still do the pick random for the next key knowledge entering the intervention, scoring above average
on all five questions examined here. S10 and S12 each scored
S11 perfectly on the pre-test, while S11 was far behind them. Un-
Instructor So for the second key you can do random like the other groups, no student really seemed to take charge
during programming - though each contributed some actions,
S11 Okay no single student took the lead with a significantly large num-
Instructor But you need the first key to change to the second key ber of actions. This perhaps led to the Constructive discourse
So you need to keep track of what your current key is to
characterizing this group. Considering they were collaborat-
Instructor ing with two strong group members and at least partially con-
change it to
tributing to the programs, the lack of improvement for S11 is
S10 and S12 did not actively converse during this conver- concerning. noted com-
sation, but they were still attentive and asked questions of in- puter struggles and requisite need to undertake a programming
structors at other times. Within the same ten-minute window
oration and negatively impacted learning.
as this conversation, most model-building actions were taken

978-1-7281-6942-2/20/$31.00 ©2020 IEEE December 8–11, 2020, Online

IEEE TALE2020 – An International Conference on Engineering, Technology and Education
Authorized licensed use limited to: La Trobe University. DownloadedPage 219 14,2024 at 00:23:08 UTC from IEEE Xplore. Restrictions apply.
on December
 A few limitations should be addressed here that would be [10] -Based and Text-
implemented in future interventions. One is that there are
ACM Transactions on Computing Education, vol. 18, no. 1, pp. 1 25,
some concerns that the pre- and post-test questions did not ad- Dec. 2017.
equately cover the broad range of topics introduced to stu- [11] I. Ouahbi, F. Kaddari, H. Darhmaoui, A. Elachqar, and S. Lahmine,
dents. Questions specifically related to CT concepts such as
variables and lists would have been particularly welcome. Ad- - Social and Behavioral
ditionally, the lack of formative assessments added some Sciences, vol. 191, pp. 1479 1482, Jun. 2015.
[12] -based, collaborative, algorithmic robotics
tervention. Including them would instead provide an insight for high school students: Programming self-
in 2017 IEEE Integrated STEM Education Conference (ISEC), 2017.
into what areas students are succeeding in or struggling with.
[13] L. Werner, J. Denner, S. Campe, and D. C. Kawamot
Future work could involve a study designed to analyze the
behaviors of group leaders. Even though they were not given symposium on Computer Science Education -
such a role by instructors, Group 1 (S1), Group 2 (S6), and [14]
international handbook of collaborative learning, 437, Mar. 2013.
Group 3 (S8) all had clear leaders who contributed a larger
[15]
than average quantity of programming actions and discourse.
Discovering what led to these students becoming leaders and vol. 49, no. 4, pp. 219 243, Oct. 2014.
any associated positive and negative effects would further in- [16]
crease understanding of collaborative learning. Other plans we knowledge construction in computer-supported collaborative
have considered involve applying this platform to other oft- 95, Jan. 2006.
neglected domains such as computer networking, with the [17]
continued goal of increasing interest in and knowledge of l
these abstract, complex, and important areas. These additions Symposium on Computer Science Education, 2020.
could lead greater understanding of the collaborative behav- [18] G. Jin, M. Tu, T.-
Game-Based Learning in Cybersecurity Education for High School
iors of students and the related impact on learning gains.
1, p. 150, Feb. 2018.
ACKNOWLEDGMENT [19]
This material is based in part upon work supported by Na- Security & Privacy, vol. 13, no. 3, pp. 63 69, May 2015.
tional Security Agency Science of Security Lablet H98230- [20] K. Rand, S. Sengupta, and D. Feil- Robotics as a
18-D-0010 and National Science Foundation grants CNS-
1644848, CNS-1521617, and DRL-1640199. Any opinions, in Journal of The Colloquium for Information System Security
findings, and conclusions expressed in this material are those Education, vol. 6, no. 2, pp. 18-18, Feb. 2019.
of the author(s) and do not necessarily reflect the views of the [21] ps,
US Government. and Autonomous Systems, vol. 75, pp. 686 691, Jan. 2016.
[22]
REFERENCES meeting of the
[1] J. Jang- American educational research association, 2011.
[23] A. Saenz-Otero, J. Katz, S. Mohan, D. W. Miller, and G. E. Chamitoff,
5, pp. 973 993, Aug. 2014. -Robotics: A student competition aboard the International
[2] AP Computer Science Principles Course and Exam Description. 2020.
[Online]. Available: https://apcentral.collegeboard.org/pdf/ap- [24] P.
computer-science-principles-course-and-exam- Development of Computational Thinking in a Robotics Learning
description.pdf?course=ap-computer-science-principles. -245,
[3] 2019.
Integrating cybersecurity into the computer [25] S. Nag, J. G. Katz, and A. Saenz- aming and
in 2017 IEEE Frontiers in Education Conference (FIE), 2017. competition for CS-
[4] Acta Astronautica, vol. 83, pp. 145 174, Feb. 2013.
[26] -On Cybersecurity Curriculum Using a
pp. 84 86, Sep. 2016.
[5] A. T. Sherman Technical Symposium on Computer Science Education, Feb. 2020.
377, Sep. 2017. [27] -Centered
[6]
hing Practice, vol. 25, no. 4, pp. 6 20, Apr. 2007.
and learning in school, 19, 2018. [28] C.S.F.S Committee. K-12 Computer Science Framework. 2016.
[7] [Online]. Available: https://k12cs.org/, 2016.
[29]
Information Technologies, vol. 21, no. 3, pp. 559 588, Jul. 2014.
[8] ributed Programming with NetsBlox is a the 13th International Conference on Computer Supported
Collaborative Learning 2019, vol. 1, pp. 360-367, Jun. 2019.
Technical Symposium on Computer Science Education, Mar. 2017 [30]
[9] Analysis to Characterize Self-Regulated Learning Behaviours in Open-
e 50th ACM Technical Symposium on
Computer Science Education, Feb. 2019. 2015.

978-1-7281-6942-2/20/$31.00 ©2020 IEEE December 8–11, 2020, Online

IEEE TALE2020 – An International Conference on Engineering, Technology and Education
Authorized licensed use limited to: La Trobe University. DownloadedPage 220 14,2024 at 00:23:08 UTC from IEEE Xplore. Restrictions apply.
on December