Journal of Research on Technology in Education

ISSN: 1539-1523 (Print) 1945-0818 (Online) Journal homepage: www.tandfonline.com/journals/ujrt20

Development, Implementation, and Outcomes

of an Equitable Computer Science After-School
Program: Findings From Middle-School Students

Chrystalla Mouza, Alison Marzocchi, Yi-Cheng Pan & Lori Pollock

To cite this article: Chrystalla Mouza, Alison Marzocchi, Yi-Cheng Pan & Lori Pollock (2016)
Development, Implementation, and Outcomes of an Equitable Computer Science After-
School Program: Findings From Middle-School Students, Journal of Research on Technology in
Education, 48:2, 84-104, DOI: 10.1080/15391523.2016.1146561

To link to this article: https://doi.org/10.1080/15391523.2016.1146561

Published online: 29 Feb 2016.

Submit your article to this journal

Article views: 1612

View related articles

View Crossmark data

Citing articles: 23 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=ujrt20
 JRTE | Vol. 48, No. 2, pp. 84–104 | Ó 2016 ISTE | iste.org/jrte
DOI: 10.1080/15391523.2016.1146561

Development, Implementation, and Outcomes of an Equitable

Computer Science After-School Program: Findings From
Middle-School Students

Chrystalla Mouza
University of Delaware

Alison Marzocchi
California State University, Fullerton

Yi-Cheng Pan & Lori Pollock

University of Delaware

Abstract

Current policy efforts that seek to improve learning in science, technology, engineering, and
mathematics (STEM) emphasize the importance of helping all students acquire concepts and
tools from computer science that help them analyze and develop solutions to everyday problems.
These goals have been generally described in the literature under the term computational
thinking. In this article, we report on the design, implementation, and outcomes of an after-
school program on computational thinking. The program was founded through a partnership
between university faculty, undergraduates, teachers, and students. Specifically, we examine how
equitable pedagogical practices can be applied in the design of computing programs and the
ways in which participation in such programs influence middle school students’ learning of
computer science concepts, computational practices, and attitudes toward computing.
Participants included 52 middle school students who voluntarily attended the 9-week after-
school program, as well as four undergraduates and one teacher who designed and implemented
the program. Data were collected from after-school program observations, undergraduate
reflections, computer science content assessments, programming products, and attitude surveys.
The results indicate that the program positively influenced student learning of computer science
concepts and attitudes toward computing. Findings have implications for the design of effective
learning experiences that broaden participation in computing. (Keywords: computational
thinking, programming, middle school, mixed methods)

R
ecent policy reports emphasize the importance of helping all students acquire a deeper
understanding of how to recognize aspects of computation in the world around us, solve
real-world problems, design systems, and understand human behavior by drawing on
computer science (CS) concepts (PCAST, 2010; Royal Academy of Engineering, 2012; Wing,
2006). These goals have been generally described in the literature under the term computational
thinking (CT) (Wing, 2006). Wing (2006) suggested that CT is a fundamental skill for everyone and
that “to reading, writing, and arithmetic, we should add computational thinking to every child’s ana-
lytical ability” (p. 33).
Although many children are exposed to new technologies in their daily lives, they often acquire
skills as consumers and are given little opportunity to become creators of computing innovations

Color versions of one or more of the figures in this article can be found online at www.tandfonline.com/ujrt.

84 l Journal of Research on Technology in Education l Volume 48 Number 2

 Equitable Computer Science After-School Program

(Repenning et al., 2015). In fact, only students who make up a small and fairly homogeneous group
acquire skills required to create technological products. Certain populations such as females and
non-Asian minorities remain severely underrepresented in computing (Cuny, 2012). To address this
challenge, research recommends investing in local partnerships, implementing service-learning pro-
grams where undergraduates assist local teachers, using age-appropriate programming tools, and
implementing equitable CS pedagogical practices (Cuny, 2012; Maloney, Peppler, Kafai, Resnick,
& Rusk, 2008; Shah et al., 2013).
In this article, we report on our work to foster the development of CT at the middle school level
through an after-school program. The program was founded through a strong partnership between
university faculty, CS undergraduates, teachers, and students. The overall intention was to broaden
participation in computing by providing equal access to CS knowledge and skills through opportu-
nities to create computational artifacts using Scratch (http://scratch.mit.edu), a block-based pro-
gramming language for novice programmers (Maloney et al., 2008). Specifically, we explore two
research questions:

 How can equitable CS pedagogical practices be applied in the design and implementation of
after-school programs for middle school students?
 Does participation in such programs impact middle school students’ learning of CS concepts,
computational practices, and attitudes toward computing?

Literature Review
Computational Thinking
The importance of learning to think computationally has long been recognized in K–12 education
(Howland & Good, 2015). Often credited with being a pioneer in studying CT among children,
Papert (1980) claimed that students’ procedural thinking skills would improve through program-
ming activities. Recently, the need to help all students acquire CT skills has gained increased atten-
tion as a result of policy efforts seeking to improve science, technology, engineering, and
mathematics (STEM) learning, as well as industry initiatives (e.g., Code.org, Khan Academy, cs-
first.com) aimed at promoting a more technology-savvy workforce. In response to these calls, a
number of countries (e.g., the United Kingdom, Estonia, Russia, New Zealand) have recently
announced initiatives that introduce CT as a core component of the school curriculum.
Despite the attention, the term CT has been surrounded by ambiguity. This ambiguity prompted
the National Academy of Sciences to organize a series of two workshops that brought together edu-
cation and industry experts to explore the characteristics of CT and its pedagogical aspects (National
Research Council [NRC], 2010, 2011). Workshop participants agreed that although programming is
key to supporting cognitive tasks involved in CT, efforts to promote CT should move beyond pro-
gramming to include a focus on creativity, innovation, and understandings around the impacts of
computing for society. Further, workshop participants delineated pedagogies for teaching CT across
education contexts. Low-technology approaches, for instance, such as those available through CS
unplugged (http://csunplugged.org), were identified as promising for introducing novices to CT
skills and practices (Bell, Alexander, Freeman, & Grimley, 2008).
In an effort to provide an operational definition of CT specific to K–12 contexts, the Computer
Science Teachers Association (CSTA) and the International Society for Technology in Education
(ISTE) also convened a group of leaders who discussed definitions, implementation, standards, and
artifacts associated with CT in K–12. Definitions of CT emerging from this group focused on CT as
an approach to “solving problems in a way that can be implemented with a computer” (Barr &
Stephenson, 2011, p. 51). This definition emphasizes the role of students as tool builders who use a
set of concepts to solve problems and recognize that “problems can be solved in multiple ways” (p.
51). Nevertheless, Barr and Stephenson acknowledged that promoting CT in K–12 settings is chal-
lenging because it requires a practical approach and answers to a set of questions related to what CT
looks like in the classroom, what skills students would need to demonstrate, what teachers would
Volume 48 Number 2 l Journal of Research on Technology in Education l 85
Mouza et al.

need to put CT in practice, and what teachers are already doing that could be modified and extended.
In this work, we describe a practical approach to introducing CT to middle school students through
a service-learning, school–university partnership. This aligns with Cuny’s (2012) recommendation
that university CS departments create service-learning partnership programs with K–12 schools.

Tools for Computational Thinking

Over the last several decades, the field has seen many developments in kid-friendly programming
environments (Brennan, 2013; Papert, 1980). One such environment is Scratch (https://scratch.mit.
edu), which builds upon ideas of Logo but replaces typing code with a drag-and-drop approach
(Maloney et al., 2008). Scratch is a “low floor, high ceiling” visual programming environment, such
that it is easy for beginners to create working programs (low floor), but also powerful enough to
support more advanced programmers (high ceiling) (Grover & Pea, 2013). In Scratch, users snap
together CS commands known as “blocks” to create scripts that are executed by characters, known
as “sprites.” Scratch blocks are aligned with CT concepts including loops, conditionals, variables,
and event triggers (Brennan, 2013; Maloney et al., 2008). The block-based interface eliminates syn-
tax errors and gives immediate visual feedback through the behavior of the sprites (Meerbaum-Sal-
ant, Armoni, & Ben-Ari, 2010). Further, it supports programming activities aligned with the
interests of middle school students, such as creating animated stories and games (Maloney et al.,
2008).
A number of studies demonstrated positive learning outcomes as a result of programming in Scratch.
Nevertheless, there is variability in the context in which programming was introduced in these studies, as
well as the measures used to represent student learning. Maloney et al. (2008), for instance, examined the
experiences of urban youth by analyzing Scratch projects collected in an after-school setting independent
of teacher support or direct instruction. Meerbaum-Salant et al. (2010), in contrast, examined student out-
comes as a result of following prescribed curriculum exercises developed by the authors. Others examined
collaborative practices and outcomes as a result of student participation in the online learning community
in Scratch (Kafai, Fields, & Burke, 2010). In this work, we examine student outcomes as a result of their
participation in an after-school program, whereby instruction was delivered in the context of creating
computational artifacts in Scratch.

Broadening Participation in Computing

A key challenge to CS education is broadening participation of females and minorities in computing
(Cuny, 2012). Visual programming languages, such as Scratch, aim to introduce a broader, younger,
and more diverse group of students to CS concepts (Kelleher & Pausch, 2005). Yet interest in CS
declines as students progress in their studies (National Center for Education Statistics [NCES],
2011). Reasons for this decline include the lack of rigorous CS standards, inadequate teacher prepa-
ration, and a heavy focus on programming that fails to provide motivating applications (Cuny,
2012). To respond to this challenge, new curricular approaches are currently underway (i.e., CS
Principles) that break down the meaning of CT based on big ideas in computing and emphasize cre-
ativity and relevance of computing to society’s needs (Cuny, 2012).
Although curricular revisions and programming environments are necessary for broadening
participation in computing, they are not sufficient. Rather, a renewed vision of CS pedagogy may
be a more critical part of the solution to the problem (Repenning et al., 2015; Shah et al., 2013;
Webb, Repenning, & Koh, 2012). Toward this end, Shah et al. (2013) proposed a multidimen-
sional framework for equitable CS instruction that aims to “disrupt inequities that occur at the
level of classroom interaction” (p. 263). This framework consists of four dimensions, including
access to rich course content, quality instruction, productive peer relationships, and identities as
computer scientists. In the framework, rich content and quality instruction are viewed as access to
material resources. They include both a focus on curricular materials and teaching practices that
make the curriculum more accessible to students through content-specific and general pedagogical
strategies. Peer relationships and identities as computer scientists are viewed as access to
86 l Journal of Research on Technology in Education l Volume 48 Number 2
 Equitable Computer Science After-School Program

nonmaterial resources. These dimensions focus on building students’ self-efficacy, as well as

awareness of the role of their peers as learning resources.
While the framework holds promise for promoting equitable CS teaching, Shah et al. (2013)
have not provided empirical evidence on student outcomes. In this work, we use the equitable
framework as a lens to examine the CS pedagogical practices designed and implemented in the
after-school program discussed in this study. In addition, we take the work of Shah et al. a step fur-
ther by presenting empirical evidence on student outcomes related to CS concepts, computational
practices, and attitudes toward computing.

Methods
Context of This Work
This work is situated in a larger effort to improve the teaching of CS through a three-pronged
approach: teacher professional development, a college field experience course, and sustainable part-
nerships with local schools. In this article, we focus on the latter two strategies.
The field experience course is a semester-long 1- to 3-credit college course open to undergradu-
ates in CS or other STEM fields. The course combines college classroom meetings with field experi-
ence in middle or high school classrooms where undergraduates teach computing lessons with a
practicing teacher. The once-per-week 75-minute on-campus class time is devoted to identifying
and implementing CS teaching resources, modeling CS classroom lessons, discussing CS pedagogy,
preparing and analyzing lesson plans, and reflecting on the field experiences. The course fulfills ser-
vice-learning requirements at the university level. Although participants do not intend to pursue a
teaching career, they enroll in the course because of their desire to share their CS expertise with
others and to strengthen their ability to communicate technical information to a wide audience.
Field experiences take place in middle or high schools that partner with us in order to integrate
CS principles into their school curricula or after-school programs. Undergraduates in the course
meet with teachers weekly to plan lessons, lead classroom activities, and facilitate after-school pro-
grams. In this work, we used a mixed-methods within-subjects design to examine one such partner-
ship between a group of CS undergraduates enrolled in the field experience course and a middle
school teacher in the context of an after-school CS program.

Participants
Participants in this study included four CS undergraduates in their junior (Jayson and Maggie) or
senior year (Mike and Brenda) enrolled in the field experience course during a one-semester period.
The four undergraduates were partnered with a middle school teacher, Sophia, at a suburban K–8
school serving 1,350 students. Sophia was an experienced technology teacher who taught all tech-
nology classes at the school. She previously attended a week-long summer professional develop-
ment institute offered by our team that focused on helping middle and high school teachers
integrate CS principles into existing curricula or after-school programs. As part of the partnership,
Sophia established a 9-week after-school CS program for students in Grades 4–6. Fifty-two students
(N D 52; 30 boys and 22 girls) attended the program voluntarily. The students were divided between
two groups. Jayson and Maggie were responsible for Group A while Mike and Brenda were respon-
sible for Group B. Group A included primarily students who had no prior experience with Scratch.
Group B included a number of students who had some prior experience with Scratch through intro-
ductory lessons delivered by their technology teacher or participation in a similar after-school pro-
gram offered in prior semesters. The teacher floated between the two groups, facilitating logistics,
focusing on classroom management, and providing assistance to students as time permitted. The
undergraduates were given the primary instructional responsibility.

Data Sources
Data were collected from multiple sources: (a) observations of CS after-school program meetings;
(b) CS undergraduates’ reflective entries on their field experience; (c) pre/post assessment of middle
Volume 48 Number 2 l Journal of Research on Technology in Education l 87
Mouza et al.

school students’ learning of CS concepts (N D 41); (d) Scratch programs developed by students dur-
ing their participation in the after-school program (N D 38); and (e) pre/post surveys on student atti-
tudes and motivation toward computing (N D 41).
Observations of CS program meetings. All weekly CS after-school program meetings (N D 9)
were observed and documented using detailed field notes to provide an accurate description of the
activities. Each after-school program meeting was 90 minutes long. Each observation included
information on the role of the partner teacher and the undergraduates as well as the lesson itself,
including materials and CS concepts addressed. A short debriefing session with the teacher and the
undergraduates accompanied each observation to ensure that the observers understood the purpose
of the activities. Further, all teaching materials created by undergraduates throughout the duration
of the after-school program were collected to gain a better understanding of the CS activities.
Undergraduate reflective journals. An important component of service learning is directed
reflection, which connects what undergraduates learn in the classroom with their experience of real-
world conditions (Bringle & Hatcher, 1999). On a weekly basis throughout the semester, undergrad-
uates reflected on their experience using a Web-based journal (blog). Each reflective entry included
three components: a description of the field activities enacted by each participant during the week,
identified successes and failures, and recommended steps for future action. The length of each entry
varied. Among the four participants, in total 30 journal entries were generated with an average
length of 400 words per entry.
Middle school student data. To document changes in students’ knowledge of CS concepts, we
administered a pre and post multiple-choice content assessment developed and used in similar pro-
grams by Ericson and McKlin (2012). The assessment includes 10 questions examining students’
knowledge of the Scratch programming environment. The first question asks students to select the
category for a block. Questions 2–9 ask students to match a Scratch script to a computing concept
such as loops, conditionals, variables, or parallelism and determine the result of executing a script.
Question 10 asks students to read a code segment and predict the outcome. In total, 41 (N D 41) pre
and post, matched responses were collected.
In addition to the content assessment, we also collected and evaluated students’ Scratch pro-
grams (N D 38). For each student, we selected the one project that was completed closest to the end
of the 9-week after-school program. Only original projects were selected, so if the latest project was
remixed (i.e., a modified version of an existing shared project), it was discarded and the next avail-
able project was selected instead. These projects provided insight into the CS concepts and practices
that the students learned during their participation in the after-school program.
To document potential changes in students’ attitudes toward computing, we administered a pre
and post survey developed by Ericson and McKlin (2012). Although the survey builds upon two
prior instruments for assessing attitudes toward computing, which were found to be valid and reli-
able (Knezek & Christensen, 1996; Williams, Wiebe, Yang, Ferzli, & Miller, 2002), we also
assessed the reliability for our sample. The Cronbach alpha of a scale should be greater than .70 for
items to be used together as a scale (Nunnally, 1978). Results of our analyses revealed that all sub-
scales and the instrument as a whole achieved acceptable alpha levels exceeding .70, indicating that
the survey was reliable (.932 for the survey as a whole).
The survey asks students to rate their agreement or disagreement using a Likert scale format that
ranges from 1 (strongly disagree) to 5 (strongly agree). It includes 36 items organized around seven
constructs found to enhance the number of underrepresented students who persist in the fields of
computing: (a) computing confidence (e.g., I have a lot of self-confidence when it comes to comput-
ing); (b) computer enjoyment (e.g., I enjoy doing things on a computer); (c) computer importance
and perceived usefulness of computing (e.g., I will be able to get a good job if I learn how to use a
computer); (d) motivation to succeed in computing (e.g., I like solving computing problems); (e)
computing identity and belongingness (e.g., I feel like I belong in computer science); (f) gender
equity (e.g., girls can do just as well as boys in computing); and (g) intention to persist (e.g., I can
see myself working in a computing field). In total, 41 (N D 41) pre/post surveys were collected.

88 l Journal of Research on Technology in Education l Volume 48 Number 2

Data Analysis
Undergraduates’ reflections and program observations. Undergraduates’ reflections were
analyzed qualitatively. We first read all reflective entries and summarized key CS activities designed
and implemented by the undergraduates during the CS after-school program. Subsequently, all
entries were entered into the NVivo qualitative software (QSR International, 2010). The first coding
sweep involved creating nodes that captured instances of CS content and pedagogical moves. This
resulted in a comprehensive list of codes for all the content and pedagogy discussed by the under-
graduates. These codes were then categorized and organized using the four equity dimensions iden-
tified by Shah et al. (2013): rich content, quality instruction, peer relationships, and identities as
computer scientists. This resulted in four categories of codes (see Table 1). All reflective entries
were then recoded using the coding scheme presented in Table 1. Observation data and teaching
artifacts collected from undergraduates were used to triangulate findings and confirm or dismiss
emergent themes (Merriam, 2009).
Middle school data. Content assessments of Scratch concepts were scored for correctness.
Results were entered into a spreadsheet and subsequently exported into the Statistical Package for
the Social Sciences (SPSS) software for analysis. Each correct question received 1 point while each
incorrect answer received no points. The percentage of students who scored correctly on each item
was also calculated for both the pre and post administration of the assessment. Further, a total score
was calculated for the instrument as a whole, and a paired-sample t-test was used to assess statisti-
cally significant changes from pre to post administration.
To analyze the computer programs (N D 38) created by students, we utilized a coding scheme
that identified the extent to which students used features of Scratch associated with CS concepts
and computational practices. The coding scheme builds upon prior efforts to investigate student
learning of CS concepts and computational ways of thinking through computer programs (e.g.,
Brennan & Resnick, 2012; Denner, Werner, & Ortiz, 2011). In particular, Denner et al. (2011)
hypothesized that when students develop programming projects, they engage with three key compe-
tencies: (a) CS programming concepts: for example, variables, loops, conditionals, parallelism, and
event programming; (b) code organization: a professional activity that indicates evidence of learn-
ing to think in a computational way; and (c) designing for usability: the extent to which a tool can
be used intuitively and effectively for the stated goal. Based on this hypothesis, Denner et al.
designed an evaluation scheme that they used to examine computer games designed by middle
school girls. The coding scheme used in this study builds upon the framework provided by Denner
et al. (2011), as well as on recommendations provided by Brennan and Resnick (2012) on CT con-
cepts associated with the Scratch programming environment specifically (see Figure A1 in the
Appendix). The coding scheme does not produce an overall score; rather, it helps coders provide a
description of CT concepts represented in students’ programs. Three coders individually coded the
same five projects (130 total codes) and achieved a reliability score of 93%. The remaining projects
were coded in pairs to ensure consistency.
Finally, survey data on students’ attitudes toward computing were scored, entered into a spread-
sheet, and subsequently exported into SPSS, where means and standard deviations were calculated

Table 1 Coding Scheme for Equitable CS After-School Program

Dimension Pedagogical Practices

Rich content Big ideas of CS

CS unplugged
Diverse problem-solving approaches
Quality instruction Formative assessment
Differentiated instruction
Peer relationships Undergraduate paired-teaching
Peer assistance and collaboration
Identities as computer scientists CS undergraduates as role models
Showcasing student work

Volume 48 Number 2 l Journal of Research on Technology in Education l 89

Mouza et al.

for each item and for the survey as a whole. To test for the significance of the score gain, a paired-
sample t-test was conducted on each of the constructs. A total score was also calculated for the sur-
vey as a whole, and a paired-sample t-test was used to assess statistically significant changes from
pre to post administration.

Results
Research Question 1: How Can Equitable CS Pedagogical Practices Be Applied in
the Design and Implementation of After-School Programs for Middle Schools
Students?
In this section, we present the after-school CS program that served as the foundation of this work
through the equity lens described by Shah et al. (2013).
Access to rich content. The responsibility for supporting CS in K–8 U.S. schools has primarily
rested with individual researchers and professional organizations (Gal-Ezer & Stephenson, 2014).
As a result, there are no standardized curricula for CS teaching at the middle school level. Rather,
there is a set of standards, resources, and big ideas in CS (Seehorn et al. 2011.). A big CS idea is
programming, which focuses on all aspects of computation, including human expression, problem
solving, and concepts and techniques used in writing computer programs (College Board, 2013).
The after-school CS program focused on fundamental, but often challenging programming, con-
cepts, similar to those addressed in other successful initiatives (e.g., Georgia Computes!), including
(a) loops—repeated execution of specific code; (b) conditionals: if–then statements; (c) variable
modification: manipulation and modification of data blocks; (d) sending/receiving messages: acti-
vating a block of code; and (e) tracing: interpreting a block of code and identifying the outcome
(Guzdial, Ericson, McKlin, & Engelman, 2014). The specific content and sequence of the after-
school program are shown in Table 2.
Although the specific content and sequence slightly differed in the two groups (see Table 2), the
concepts were consistently introduced through two types of practices: CS unplugged activities that
made CS concepts more concrete to students, and project-based programming using Scratch.
Describing the use of CS unplugged during Week 2, for example, Maggie noted:

We began the lesson with the Marching Orders unplugged activity where one student would
describe a drawing to the class in attempt to get the students to replicate the picture. Two stu-
dents gave directions (somewhat successfully) and then Jayson stepped in and gave clear
instructions for the third example. I then explained that in order to get Scratch to do what you
want, you have to be completely clear and explicit with what you tell it. I think this was a
great exercise, not only did the kids had fun but they also understood the practical relevance
to CS as well.

Following the brief introduction of programming concepts through CS unplugged, students were
given a set of tasks to work on usually in the context of their own projects. This approach allowed
the undergraduates to shift away from teacher-directed activities to guided discovery (Meerbaum-
Salant et al., 2010; Webb et al., 2012). Specifically, it helped them focus on general CS goals through
small chunks of formal instruction while enabling students to follow their own paths, explore differ-
ent design options, and control the overall direction of their projects. During project-based time, the
undergraduates answered questions and provided individual assistance while avoiding direct answers
to encourage diverse problem-solving approaches and multiple solutions. Brenda explained:

I spent a lot of time with one boy who was frustrated that his code wasn’t doing what was
expected. I was patient with him, giving him small problems at a time and letting him figure
them out on his own. We were able to fix his program and he was so proud of it.
90 l Journal of Research on Technology in Education l Volume 48 Number 2
 Equitable Computer Science After-School Program

Table 2. Overview of CS Content at the After-School Program

Group A (No Prior Experience) Group B (Some Prior Experience)

Week 1 Sprites Introduction to Programming

1. Scratch programming: Customizing sprites 1. Scratch programming: introduction/draft and
and backgrounds. begin programming a Scratch project of choice.
Week 2 Algorithms Conditionals
1. CS Unplugged Marching Orders: Introduction 1. CS Unplugged on Conditionals (e.g., IF you
to understanding that computers are pro- are a girl jump and down, ELSE sit down).
grammed to obey instructions. 2. Reviewing Scratch projects including condi-
2. Scratch programming: Drawing a square. tionals.
3. Scratch programming: Continue program-
ming Scratch project of choice.
Week 3 Conditionals Loops
1. CS Unplugged on Conditionals (e.g., IF you 1. CS Unplugged on Loops: Activities that mir-
are a girl jump, ELSE sit down). rored programming blocks such as
2. Scratch programming: motion, moving sprites “repeat#times,” “forever,” and “repeat until” (e.g.,
using keyboard input within project of choice. jump as many times as your age).
2. Scratch programming: Continue program-
ming Scratch project of choice.
Week 4 Variables Boolean Logic
1. CS Unplugged on Variables: Guessing game. 1. CS Unplugged: Students engage in a game of
2. Scratch programming: Using variables in a 20 yes/no questions until the answer is guessed.
guessing game of choice. 2. Scratch programming: Continue working on
Scratch project of choice.
Week 5 Broadcasting Broadcasting and Variables
1. CS Unplugged: Following a skit to understand 1. Using broadcasting in combination with varia-
how one action prompts another. bles to trigger interaction among multiple sprites.
2. Scratch programming: Broadcasting and 2. Scratch programming: Continue working on
sprite interaction within project of choice. Scratch project of choice.
Week 6 Broadcasting (continued) Global Impacts of CS
1. Scratch programming: Students recreate a 1. Global impacts related to computing.
scene from a literacy book using programming. 2. Scratch programming: Continue working on
Scratch project.
Week 7 Loops Reading Other Programmers’ Code1. In pairs
1. CS Unplugged on Loops: Activities that mir- students review three Scratch projects, varying
rored programming blocks such as in complexity in order to predict the execution of
“repeat#times,” “forever,” and “repeat until.” the script.
2. Scratch programming: Students draft and 2. Students discuss/share new insights as a
program a project of their choice. result of their review.
Week 8 Scratch programming: Project development/peer Scratch programming: Continue working on
review. Scratch project of choice/peer review.
Week 9 Demo Day Demo Day

The preceding examples illustrate how the undergraduates provided rich and challenging content
to students while also emphasizing diverse learning paths.
Quality instruction. Although the undergraduates lacked strong pedagogical preparation in
teaching CS, collaboration with university faculty and the in-service teacher allowed them to access
and implement quality instructional practices. Specifically, undergraduates focused on two primary
pedagogical practices, including formative assessment and differentiated instruction. Formative
assessment was implemented through exit tickets, a practice introduced to them by Sophia, their
partner teacher. An exit ticket is a common form of formative assessment that typically includes a
question posed to students at the end of each class (Wylie, Lyon, & Goe, 2009). Students enter their
answer and deliver it to the teacher as they exit. Exit tickets are used to gauge new learning, provide
immediate feedback on the materials and teaching, and provide information for teachers to reflect
on and make adjustments about for the next session.
Initially, Sophia collected exit tickets that elicited students’ views on aspects of the program they
enjoyed and their suggestions for improvements. After a discussion with university faculty, the
undergraduates modified this strategy to include questions focusing on both perceptions and content
learning. Exit tickets included questions on conceptual understanding of loops, conditionals, broad-
casting, and variables (e.g., IF your first name starts with a vowel draw a square ELSE draw a
Volume 48 Number 2 l Journal of Research on Technology in Education l 91
Mouza et al.

circle). Most exit tickets asked students to interpret the output of Scratch code to indicate under-
standing. At the end of each weekly session, the undergraduates reviewed students’ responses in
preparation for the following week’s session. As a result of students’ responses, they either provided
more challenging content or reinforcement until most students expressed satisfaction. Reflecting on
his work, Mike noted:

In one of the first lessons, I was getting exit tickets from several of the students saying that
they were familiar with the materials. I knew that in some cases students might not acknowl-
edge they were learning things if the activity appeared simple. However, I also knew that I
needed more challenging activities in order to keep the attention of the class.

In addition to formative assessment, the undergraduates used differentiated instruction to address

students’ needs. This was done in two ways. At the beginning of the program, Sophia helped under-
graduates divide students into two groups based on their prior experience with CS. Some students,
for instance, had previously participated in a similar CS after-school program offered by our team.
Those students were grouped together in Group B. This allowed the undergraduates to follow more
accelerated or challenging instruction. Students with little or no prior experience were group
together in Group A and the undergraduates began with more basic activities to familiarize them
with programming (see Table 2). Although efforts were made to group students with similar experi-
ences together, it is important to note that differences remained in students’ prior skills and CS
knowledge within each group.
To further address students’ needs, the undergraduates spent time interacting with individual stu-
dents, offering one-to-one scaffolding as they worked on individual exercises, and providing
options in the choice of tasks. Jayson wrote:

Some of the students were faster to pick up on topics than others. While I expected this, at first
I didn’t account for it when giving out assignments to students. The students who were quick
to pick up on stuff would finish early and then be left with nothing to do. After noticing this,
Maggie and I made it a point to provide different tasks. Those students who finished early
were given “additional” tasks to complete, which were generally continuations of the original
tasks. This strategy worked out great because everyone was always working on something,
and those who would work on the additional tasks really dove deep into the concepts, and not
only learned a lot from it, but got excited working on and solving the problems.

Peer relationships. Peer relationships are important for promoting respect and facilitating new
learning. To foster social interactions and help students see their peers as resources for learning, two
pedagogical practices were implemented. First, the undergraduates taught in pairs, modeling collab-
orative interactions and building on each other’s activities. Jayson explained how they modeled the
roles of the “driver” and “navigator” to students:

Maggie and I have this good system going where I stand in front of the students and engage
them in the activity, whereas Maggie is sitting on the computer doing what the students are
saying they think should be done next.

Peer interactions were also supported through intentional activities that asked students to share
and debate ideas. Brenda explained:

We had a series of three programs that got progressively more complicated. Students were
asked to work in groups to discuss the programs and write down their best guess on the out-
put. Then, as a class, we discussed their thoughts.
92 l Journal of Research on Technology in Education l Volume 48 Number 2
 Equitable Computer Science After-School Program

Finally, with input from Sophia, the undergraduates established strategies for soliciting and
receiving assistance through peers. All students, for example, had to solicit assistance from peers
prior to asking the undergraduates and often had to share their new learning with their peers. Brenda
explained:

If I helped one student and the next one had the same issue, I asked the first student to explain
it to the second. I think this was very beneficial—the student who explained the concept rein-
forced his/her knowledge of the topic while the other student received directions from a peer,
and I could go around helping more students.

Identities as computer scientists. Helping students develop positive attitudes toward CS and
identify as being capable of doing CS is important for broadening participation in computing (Guz-
dial et al., 2014; Shah et al., 2013). To facilitate positive engagement with CS, two pedagogical
practices were implemented. First, university faculty focused on recruiting a diverse group of CS
undergraduates. Emphasis was placed on recruiting both males and females from diverse ethnic
backgrounds. Despite the underrepresentation of women and minorities in CS, each pair of CS
undergraduates included one male and one female who worked together to serve as role models to
students and encourage female participation in CS. In this manner, the undergraduates helped break
down stereotypical images of who could be a computer scientist.
In addition to providing role models, the undergraduates fostered a sense of confidence in stu-
dents by encouraging public sharing of work. Specifically, students frequently were encouraged to
share pieces of their code with their peers following a specific presentation format that allowed
them to discuss both new learning and aspects of their work that made them most proud. Further,
during the last week of the program, the undergraduates and their in-service teacher partner hosted
a showcase where students shared their work with a greater audience including parents, siblings,
and school administrators. Discussing preparations for the showcase during Week 8, Brenda
explained:

We gave students a list of four questions: what does your program do, which part are you
most proud of, what new skill did you learn, and explain a piece of code. We then asked them
to practice presenting to their partners. We also asked volunteers to present to the group. I
was impressed with their projects, their presentation style, and their enthusiasm! As they left
for the day, they were all so excited—they love when they get to show off a bit! I’m excited
for that as well because I want to see them succeed and be proud of what they have created!

Research Question 2: Does Participation in the Program Impact Middle School

Students’ Learning of CS Concepts, Computational Practices, and Attitudes Toward
Computing?
CS concepts: Results from content assessment. To examine student knowledge gains on the
pre and post Scratch assessment, we first tested the equivalency of the two groups (Group A and
Group B) at the beginning of their participation in the after-school program. Results indicated that
there were no significant differences in knowledge of Scratch CS concepts among Group A
(M D 4.81, SD D 1.96) and Group B students (M D 5.47, SD D 2.06); t(39) D 1.040, p D .305.
Results also indicated no significant differences in the Scratch knowledge assessment between boys
(n D 22) and girls (n D 19) either at the beginning (Boys M D 5.64, SD D 1.73 and Girls M D 4.53,
SD D 2.20) or end (Boys M D 6.82, SD D 1.62 and Girls M D 6.05, SD D 2.36) of the after-school
program; for pre-assessment t(39) D 1.808, p D .078, and for post-assessment t(39) D 1.225,
p D .228.
Given the preceding findings, we proceeded to analyze the data for both groups as a whole. As
illustrated in Table 3, results from the Scratch assessment indicated statistically significant improve-
ments from pre to post administration on 6 out of the 10 questions (questions 2, 4, 6, 7, 8, and 9).
Volume 48 Number 2 l Journal of Research on Technology in Education l 93
Mouza et al.

Table 3. Scratch Content Assessment

Percent Correct

Question Pre Post Paired t-Test (p) Effect Size (d)

1. Scratch block 90% 98% 0.183 0.307

2. Handling an event 24% 44% 0.031* 0.414
3. Loop (repeated execution) 76% 66% 0.210 ¡0.213
4. Loop (forever) 68% 85% 0.033* 0.404
5. Loop (repeat a set number) 73% 73% 1.000 0.000
6. Conditionals (if) 20% 46% 0.003** 0.059
7. Conditional test 59% 83% 0.006** 0.546
8. Data (modifying a variable) 39% 61% 0.048* 0.045
9. Parallelism (broadcast) 54% 76% 0.027* 0.047
10. Script execution 10% 15% 0.457 0.157

*p < .05.**p < .01.

Further, results indicated there was statistically significant improvements from the pre administra-
tion of the assessment (M D 5.122, SD D 2.01) to the post administration (M D 6.46, SD D 2.00) of
the instrument as a whole: t(40) D 4.40, p < .01, with large effect size of 0.667. These results sug-
gest significant improvements in students’ knowledge of CS concepts associated with the Scratch
programming environment.
The results revealed no significant improvements on questions 1, 3, 5, and 10. Examining those
items closely, it was determined that question 1 was easy with the majority of students getting a cor-
rect answer both at the pre and post administration. Question 3 used language that was not familiar
to students, which might have influenced the results. Questions 5 and 10 asked students to interpret
the output of a computer script, which included drawing a square and drawing a triangle. While the
majority of students were able to answer Question 5 correctly, only a very small number of students
answered Question 10 correctly. In order for students to answer Question 10 correctly, they must
understand Cartesian coordinates, a mathematical concept not addressed in their curriculum.
CS concepts: Results from students’ Scratch programs. To gain a better understanding of the
programming concepts learned by students, we analyzed the use of Scratch commands, code organi-
zation, and usability across 38 Scratch projects using the coding scheme presented in Figure A1.
Findings indicated that all projects used scripts and on average, the coded projects contained 6.21
scripts, 2.71 sprites, and 1.53 different backgrounds. The participants largely preferred to use sprites
that were built into the Scratch platform, with 68.42% of the coded projects using exclusively built-
in sprites, only 7.89% using exclusively student-created sprites, and the remaining 23.68% using a
combination of built-in and student-created sprites. Just over 71% of the coded projects were inter-
active, while 29% were noninteractive.
In an interactive project, the user is required to communicate with the project through the keyboard
and/or the mouse. Most often, students’ interactive projects could be described as games. Several of
the games were of question/answer form, such as the game titled Dawn of the Clans Trivia. In this
game, the user is asked several questions based on a popular book series. The game keeps track of
the number of correct answers and after completing the questions the user is informed of his or her
performance (see Figure 1). Other interactive games were sprite controlled, in which the user moves a
sprite around the screen to complete a task. An example of a sprite-controlled game is one titled
Maze Cat. In this game, the user uses the keyboard arrows to navigate a small version of the cat sprite
through a maze that was hand drawn by the participant (see Figure 2). In contrast, a noninteractive
project can be described as a story whereby the project runs from start to finish without requiring any
information from the user. An example of a noninteractive project is one titled Awesome Rocket
Adventure, in which the cat sprite travels to different lands and meets other characters (see Figure 3).
The projects were also coded for the presence of blocks associated with the CS concepts of loops,
conditionals, parallelism within a sprite (a single sprite doing more than one action at the same
time), parallelism across sprites (different sprites performing actions simultaneously), data, and
operators (see Table 4). Similar to Maloney et al. (2008), we interpret the use of certain Scratch
94 l Journal of Research on Technology in Education l Volume 48 Number 2
 Equitable Computer Science After-School Program

Figure 1. Interactive question/answer project.

programming blocks to indicate that a CS concept was being used in a given project. When present,
we also coded for whether the implemented concept worked as intended. Table 4 summarizes the
coded CS concepts, the percentage of projects containing these concepts, and the percentage of
those projects in which the concept worked as intended.

Figure 2. Interactive maze scratch project.

Volume 48 Number 2 l Journal of Research on Technology in Education l 95

Mouza et al.

Figure 3. Noninteractive Scratch project.

In addition to coding for the use of CS concepts in the participants’ projects, we also coded the
projects for appropriate code organization and documentation. On the whole, the majority of the stu-
dents showed evidence of appropriate code organization and documentation. Specifically, 94.74%
of the participants who used variables used appropriate names, and 65.79% of the participants
named their sprites appropriately (e.g., naming an apple sprite “apple”). Further, nearly 66% of the
participants’ projects contained only programming statements that were necessary without any
extraneous statements. Lastly, we coded for the usability of the projects. Again, the majority of the
participants showed evidence of designing user-friendly programs, with 57.89% of the participants
creating projects with smooth functionality (no glitches) and 65.79% creating projects with a clear
objective.
Students’ attitudes toward computing. As summarized in Table 5, the results indicated posi-
tive changes in student attitudes toward computing from the pre to the post administration of the
survey. With the exception of the gender equity construct, however, results were not statistically sig-
nificant. To gain a better understanding of the findings we disaggregated the data by gender. Results
indicated no significant differences in student attitudes between boys (n D 22) and girls (n D 19) at
either at the beginning (Boys M D 4.04, Boys SD D 0.49; Girls M D 4.05, Girls SD D 0.61) or the
end of the program (Boys M D 4.10, Boys SD D 0.41; Girls M D 4.06, Girls SD D 0.54); for pre-
assessment t(39) D –0.007, p D .994, and for post-assessment t(39) D 0.318, p D .752.
The results, however, indicated significant differences between boys and girls around two con-
structs: computer confidence and gender equity (see Table 6). Regarding computer confidence,

Table 4. CS Concepts in Scratch Programs

CS Concept Percent Containing Block Percent Worked as Intended

Loops 73.68% 85.71%

Conditionals 57.89% 86.36%
Data (modifying a variable) 50.00% 89.47%
Parallelism within object 18.42% 100%
Parallelism across objects 23.68% 88.89%
Operators 5.26% 100%

96 l Journal of Research on Technology in Education l Volume 48 Number 2

 Equitable Computer Science After-School Program

Table 5. Student Attitudes Toward Computing

Number of Mean for Mean for Mean Standard p Significance Effect

Constructs Items Pre Survey Post Survey Difference Deviation t df (Two-Tailed) Size (d)

Computing 6 4.045 4.167 0.122 0.569 1.373 40 .178 0.190

confidence
Computer enjoyment 7 4.450 4.460 0.010 0.328 0.204 40 .839 0.022
Computer importance 6 4.293 4.281 ¡0.012 0.510 ¡0.153 40 .879 ¡0.024
& perceived usefulness
of computing
Motivation to succeed in 6 4.024 3.907 ¡0.118 0.776 ¡0.973 40 .336 ¡0.155
computing
Computing identity and 3 3.781 3.837 0.057 0.741 0.492 40 .626 0.063
belongingness
Gender equity 4 4.445 4.585 0.140 0.422 2.127 40 .040* 0.178
Intention to persist 4 3.281 3.335 0.055 0.616 0.570 40 .572 0.052
Total 36 4.045 4.082 0.036 0.333 0.699 40 .488 0.07

*p < .05.

boys appeared more confident in their computing skills compared to girls in the post-administration
of the survey. Regarding gender equity, girls exhibited significantly more positive attitudes in both
the pre-administration and the post-administration of the survey.

Discussion
Key Findings
CS and the technologies it enables are at the heart of our economy, our daily lives, and scientific
enterprise (Wilson, Sudol, Stephenson, & Stehlik, 2010). Yet there are few opportunities for K–12
students to innovate, design, and produce computer products (Repenning et al., 2015). The comput-
ing community including university faculty and undergraduates has a unique opportunity to affect
K–12 education with its enthusiasm and experience, as well as its ideas on curriculum and pedagogy
(Cuny, 2012). In this work, we described the potential of a school–university partnership to infuse
CS content in a middle school setting using equitable pedagogical practices supported in the litera-
ture (Repenning et al., 2015; Shah et al., 2012). Findings from our work indicate that students
gained significant CS knowledge and computational practices as a result of participating in the
after-school program. Specifically, both the content knowledge assessment and the Scratch pro-
grams provided evidence that students understood and implemented foundational CS concepts such
as loops, conditionals, data, and parallelism. Further, the majority of students were able to exhibit
good computational practices associated with code organization and documentation and to develop
user-friendly programs with smooth functionality. Importantly, our findings indicated that there
were no statistically significant differences in learning and using CS concepts among boys and girls.

Table 6. Results on Gender Differences in Computer Confidence and Gender Equity

Standard Mean Effect

Attitude Survey Mean Deviation Difference t-Test (p) Size (d)

Pre survey computing confidence Boys (n D 22) 4.09 0.71 0.09 .655 0.14
Girls (n D 19) 3.99 0.70
Post survey computing confidence Boys (n D 22) 4.31 0.48 0.31 .068 0.59
Girls (n D 19) 4.00 0.52
Pre survey gender equity Boys (n D 22) 4.24 0.88 0.44 .076 0.57
Girls (n D 19) 4.68 0.64
Post survey gender equity Boys (n D 22) 4.36 0.89 0.47 .038* 0.68
Girls (n D 19) 4.84 0.50

*p < .05.

Volume 48 Number 2 l Journal of Research on Technology in Education l 97

Mouza et al.

The results, however, indicated that most students utilized certain CS concepts, such as loops,
more than others, such as conditionals and data. Even fewer students utilized parallelism within or
across objects or more advanced concepts associated with operators. Although CS undergraduates
did not explicitly target concepts around operators, all other concepts were addressed either through
CS unplugged activities and short programming tasks or while working with Scratch for the design
of projects. These findings indicate that middle school students might not be ready to engage with
more challenging CS concepts, are less interested in programming projects that involve such con-
cepts, or need more explicit instruction before they can utilize these concepts in their projects.
Further, for the most part, findings from this work indicated positive changes in students’ atti-
tudes toward computing, although these changes were not statistically significant. The lack of statis-
tical significance can be attributed to the fact that initial attitudes toward computing were strong,
leaving little room for statistically significant improvements. This outcome is not surprising, given
that students voluntarily elected to participate in the CS after-school program. Findings, however,
indicated that there were no statistically significant differences in attitudes toward computing
between boys and girls. This is an important finding because it suggests that equitable CS practices
have the potential to level the playing field among boys and girls. Our findings did indicate that
boys exhibited higher confidence in their computing skills than girls, but this finding is consistent
with other work showing that boys have higher confidence in computing than girls (Ericson &
McKlin, 2012; Margolis & Fisher, 2002). On a more positive note, our findings also indicated that
girls exhibited stronger attitudes toward gender equity, indicating that they felt confident about their
abilities to succeed in computing in the same way as boys do.
The outcomes just described are important because prior research has found that students begin
to lose interest in STEM during the middle school years (American Association of University
Women [AAUW], 2000). Our findings suggest that equitable CS practices have the potential to sus-
tain student interest toward computing. These types of interventions early in the pipeline could have
a significant impact on the number of students who decide to persist in computing fields and eventu-
ally choose computing-related careers (Guzdial et al., 2014). Therefore, it is essential to provide
middle school students with early experiences, such as those described in this work, during this cru-
cial time in their development (Peckham et al., 2007).

Limitations
Three limitations are noted in this work. First, students were self-selected since they chose to attend
the program voluntarily. Therefore, their motivation to learn and attitudes toward computing might
differ from other students. Second, although our sample included gender diversity, it presented lim-
ited ethnic and socioeconomic diversity. Specifically, most students were coming from middle class
families and enjoyed wide access to technology at home. Third, there is a potential for internal
validity threat due to repeated testing, whereby students might remember questions or responses
from the pre/post Scratch assessment.

Implications for Practice

To become well-educated citizens in an increasingly computing-driven world and be prepared for
the demands of the 21st-century economy, students must develop a deeper understanding of the fun-
damentals of CS (Wilson et al., 2010). In this work, we presented one approach to helping students
acquire foundational CS concepts in an after-school program designed as part of a school–university
partnership. After-school programs present the first glimmer of opportunity to explore CS concepts
among middle school students, and pedagogy within those programs is key to broadening participa-
tion of females and other underrepresented minorities in computing (Repenning et al., 2015). Find-
ings from our work provided insights related to the design, implementation, and outcomes of an
after-school program that implemented equitable pedagogical practices to help middle school stu-
dents learn key CS concepts and computational practices, and maintain positive attitudes toward
computing.
98 l Journal of Research on Technology in Education l Volume 48 Number 2
 Equitable Computer Science After-School Program

In order to broaden participation and help all students become creators of computing innovations,
however, it is imperative to move beyond after-school programs to the integration of foundational
CS concepts into the core curriculum. Working in conjunction with a technology or computer
teacher for helping students learn how to use programming tools such as Scratch, classroom teach-
ers could have students apply their programming skills to illustrate scenes from a book (literacy),
create simulations (science), present geometric shapes (mathematics), or produce virtual exhibits
using animation and music (art), to name a few. These applications, in conjunction with pedagogical
strategies such as those described in this work, can help students learn important computational con-
cepts in a meaningful and motivating context while also gaining a better understanding of technolo-
gies they encounter in their daily lives.

Recommendations for Future Research

Future research should examine the benefits of CS after-school programs in more diverse contexts,
in terms of both gender and ethnicity. It should also focus on integrating such experiences within
regular instruction in diverse classrooms to indicate whether our findings persist among students
who do not voluntarily choose to engage with programming experiences. Further, future research
should include a control group to enhance the reliability and validity of the findings. Finally, future
work should include longitudinal designs to examine student persistence with computing, interest
and participation in similar initiatives. We are currently examining longitudinal data collected from
a subset of students who attended more than one CS after-school program during a 2-year period, in
order to present trajectories of participation, interest, and motivation in the field of computing.

Received: 6/26/15
Initial decision: 9/1/15
Revised manuscript accepted: 12/29/15

Acknowledgments. The authors thank the teacher, students, and undergraduates who partici-
pated in this work, as well as Partner4CS members James Atlas and Terry Harvey.
Declaration of Conflicting Interests. The authors declared no potential conflicts of interest
with respect to the research, authorship, and/or publication of this article.
Funding. Research reported in this article was supported by National Science Foundation under
award number: 1240905. The content is solely the responsibility of the authors and does not neces-
sarily represent the official views of the National Science Foundation.

Author Notes
Chrystalla Mouza is an associate professor in the School of Education at the University of Delaware.
Her research interests focus on the development of computational thinking in K–16 settings, teacher
technological pedagogical content knowledge, and emerging technologies for the classroom. Please
address correspondence regarding this article to Chrystalla Mouza, School of Education, University of
Delaware, 219D Willard Hall, Newark, DE 19716, USA. E-mail: [email protected]
Alison Marzocchi is an assistant professor in the Department of Mathematics at California State
University, Fullerton. Her research interests focus on recruitment and retention of underrepresented
students in mathematics and computing.
Yi-Cheng Pan is a doctoral candidate in the School of Education at the University of Delaware. His
research interests focus on the development of computational thinking and application of new tech-
nologies in K–16 settings.

Lori Pollock is a professor in the Department of Computer and Information Sciences at the Univer-
sity of Delaware. Her research interests focus on improving the participation of women and other
underrepresented groups in computer science.

Volume 48 Number 2 l Journal of Research on Technology in Education l 99

Mouza et al.

References
American Association of University Women. (2000). Tech-savvy: Educating girls in the new computer age. Washington, DC:
American Association of University Women Education Foundation.
Barr, V., & Stephenson, C. (2011). Bringing computational thinking to K–12: What is involved and what is the role of the
computer science education community? ACM Inroads, 2(1), 48–54.
Bell, T., Alexander, J., Freeman, I., & Grimley, M. (2008). Computer science without computers: New outreach methods
from old tricks. In Proceedings of the 21st Annual Conference of the National Advisory Committee on Computing Qualifi-
cations (NACCQ08), Auckland, New Zealand. Retrieved from http://www.cs.bris.ac.uk/~jason/pdf/NACCQ08.pdf
Brennan, K. (2013). Learning computing through creating and connecting. Computer, 46, 52–59. doi:10.1109/MC.2013.229
Brennan, K., & Resnick, M. (2012, April). New frameworks for studying and assessing the development of computational think-
ing. Paper presented at Annual Meeting of the American Educational Research Association Meeting, Vancouver, Canada.
Bringle, R. G., & Hatcher, J. A (1999). Reflection in service-learning: Making meaning of experience. Educational Horizons,
77(4), 179–185.
College Board. (2013). Computer science principles. Retrieved from http://www.collegeboard.com/html/computerscience/
Cuny, J. (2012). Transforming high school computing: A call to action. ACM Inroads, 3(2), 32–36.
Denner, J., Werner, L., & Ortiz, E. (2011). Computer games created by middle school girls: Can they be used to measure
understanding of computer science concepts? Computers and Education, 58, 240–259.
Ericson, B., & McKlin, T. (2012). Effective and sustainable computing summer camps. In 43rd SIGCSE technical symposium
on computer science education, February 29–March 3, 2012, Raleigh, NC (pp. 290–294). New York, NY: ACM.
Gal-Ezer, J., & Stephenson, C. (2014). A tale of two countries: Successes and challenges in K–12 computer science education
in Israel and the United States. ACM Transactions on Computing Education, 14(2), 8.1–8.18.
Grover, S., & Pea, R. (2013). Computational thinking in K–12 a review of the state of the field. Educational Researcher, 42
(1), 38–43.
Guzdial, M., Ericson, B., McKlin, T., & Engelman, S. (2014). Georgia Computes! An intervention in a US state, with formal
and informal education in a policy context. ACM Transactions on Computing Education, 14(2), 13.1–13.29.
Howland, K., & Good, J. (2015). Learning to communicate computationally with Flip: A bi-modal programming language
for game creation. Computers and Education, 80, 224–240.
Kafai, Y. B., Fields, D. A., & Burke, W. Q. (2010). Entering the clubhouse: Case studies of young programmers joining the
online Scratch communities. Journal of Organizational and End-User Computing, 22(2), 21–35.
Kelleher, C., & Pausch, R. (2005). Lowering the barriers to programming: A taxonomy of programming environments and
languages for novice programmers. ACM Computing Surveys, 37(2), 83–137.
Knezek, G. & Christensen, R. (1996, January). Validating the computer attitude questionnaire. Presentation to the Southwest
Educational Research Association Annual Conference, New Orleans, LA. Available from ERIC Document Reproduction
Service No. ED 398 243.
Maloney, J., Peppler, K., Kafai, Y., Resnick, M., & Rusk, N. (2008). Programming by choice: urban youth learning program-
ming with scratch. In Proceedings of 39th SIGCSE technical symposium on computer science education, March 12–15,
2008, Portland, OR (pp. 367–371). New York, NY: ACM.
Margolis, J., & Fisher, A., (2002). Unlocking the clubhouse—Women in computing. Cambridge, MA: MIT Press.
Meerbaum-Salant, O., Armoni, M., & Ben-Ari, M. (2010). Learning computer science concepts with Scratch. Proceedings of
ICER 2010, August 9–10, 2010, Aarhus, Denmark (pp. 69–76). Retrieved from http://ims.mii.lt/ims/konferenciju_med
ziaga/ICER'10/docs/p69.pdf
Merriam, S. (2009). Qualitative research: A guide to design and implementation. San Francisco, CA: Jossey-Bass.
National Center for Education Statistics. (2011). America’s high school graduates: Results of the 2009 NAEP high school
transcript study. IES, National Center for Education Statistics. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/
studies/2011462.pdf
National Research Council. (2010). Committee for the Workshops on Computational Thinking: Report of a workshop on the
scope and nature of computational thinking. Washington, DC: National Academies Press.
National Research Council. (2011). Committee for the workshops on computational thinking: Report of a workshop of peda-
gogical aspects of computational thinking. Washington, DC: National Academies Press.
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.
Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York, NY: Basic Books.
PCAST. (2010). Prepare and inspire: K–12 education in science, technology, engineering, and mathematics (STEM) for
America’s future. Washington, DC. Retrieved from http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-ste
medreport.pdf
Peckham, J., Stephenson, P. D., Harlow, L. L., Stuart, D. A., Silver, B., & Mederer, H. (2007). Broadening participation in
computing: Issues and challenges. In Proceedings of the 12th Annual Conference on Innovation and Technology in Com-
puter Science Education (ITiCSE’07), June 23–27, Dundee, Scotland, United Kingdom. Retrieved from http://web.uri.edu/
advance-women/files/iticse159-peckham.pdf
QSR International. (2010). NVivo qualitative data analysis software [Version 9].
Repenning, A., Webb, D. C., Koh, K. H., Nickerson, H., Miller, S. B., Brand, C., . . . Repenning, N. (2015). Scalable game
design: A strategy to bring systemic computer science education to schools through game design and simulation creation.
ACM Transactions on Computing Education, 15(2), 11.1–11.31.

100 l Journal of Research on Technology in Education l Volume 48 Number 2

 Equitable Computer Science After-School Program

Royal Society of Engineering. (2012). Shut down or restart: The way forward for computing in UK schools. Retrieved from
http://royalsociety.org/education/policy/computing-in-schools/report/
Seehorn, D., Carey, S., Lee, I., Moix, D., O’Grady-Cuniff, D., Boucher Owens, B., . . . Verno, A. (2011). CSTA K–12 com-
puter science standards. New York, NY: Computer Science Teachers Association.
Shah, N., Lewis, C. M., Caires, R., Khan, N., Qureshi, A., Ehsanipour, D., & Gupta, N. (2013). Building equitable computer
science classrooms: elements of a teaching approach. In Proceedings of the 44th ACM technical symposium on computer
science education, March 6–9, Denver, CO, USA (pp. 263–268). New York, NY: ACM.
Webb, D., Repenning, A., & Koh, K. H. (2012). Toward an emergent theory of broadening participation in computer science
education. In Proceedings of the 43rd ACM technical symposium on computer science education, February 29–March 3,
Raleigh, NC (pp.173–178). New York, NY: ACM.
Williams, L., Wiebe, E., Yang, K., Ferzli, M., & Miller, C. (2002). In support of paired programming in the introductory com-
puter science course. Computer Science Education, 12(3), 197–212.
Wilson, C., Sudol, L. A., Stephenson, C., & Stehlik, M. (2010). Running on empty: The failure to teach K-12 computer sci-
ence in the digital age. New York, NY: The Association for Computing Machinery and the Computer Science Teachers
Association.
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35.
Wylie, E. C., Lyon, C. J., & Goe, L. (2009). Teacher professional development focused on formative assessment: Changing
teachers, changing schools. ETS Research Report Series, 2009(1), 1–14.

Volume 48 Number 2 l Journal of Research on Technology in Education l 101

Mouza et al.

Appendix

Figure A1. Coding scheme for scoring scratch projects (Continued on next page).

102 l Journal of Research on Technology in Education l Volume 48 Number 2

 Equitable Computer Science After-School Program

Figure A1. (Continued).

Volume 48 Number 2 l Journal of Research on Technology in Education l 103

Mouza et al.

Figure A1. (Continued).

104 l Journal of Research on Technology in Education l Volume 48 Number 2