An interactive, graphical coding environment for

EarSketch online using Blockly and Web Audio API

Anand Mahadevan Jason Freeman Brian Magerko

Dolby Laboratories School of Music Digital Media Program
San Francisco, CA 94103 Georgia Institute of Georgia Institute of
[email protected] Technology Technology
Atlanta, GA 30332 Atlanta, GA 30332
[email protected] [email protected]

ABSTRACT 1.2 Related work

This paper presents an interactive graphical programming Other recent projects have also tried to engage students in
environment for EarSketch, using Blockly and Web Audio computing by connecting coding to artistic and creative con-
API. This visual programming element sidesteps syntac- texts. Alice, for example, is a 3D programming environment
tical challenges common to learning text-based languages, that allows students to create animated stories through code
thereby targeting a wider range of users in both informal while learning computer science principles such as object-
and academic settings. The implementation allows seamless oriented programming [10]. Blockly [1] and Scratch [22], on
integration with the existing EarSketch web environment, the other hand, teach programming through the creation
saving block-based code to the cloud as well as exporting it and sharing of games, animations and simulations. From
to Python and JavaScript. the standpoint of coding to create music, Wavepot offers a
multitude of APIs for novice and experienced musicians to
explore and compose music online [5]. Other visual program-
1. INTRODUCTION ming based educational projects such as PencilCode [8], use
a block editor called Droplet [7]. Droplet allows bidirectional
1.1 A brief overview of EarSketch switching between text and blocks-based authoring.
1
EarSketch [12] is a novel approach to teaching com-
puter science concepts via algorithmic music composition
and remixing in the context of a digital audio workstation 1.3 Why visual programming?
paradigm. It is aimed at satisfying both artistic and peda- The role of visual programming in computer science edu-
gogical goals of introductory courses in computer science and cation has been prominent in recent years. As mentioned by
computer music. EarSketch seeks to increase and broaden Deepak Kumar in [14], visual languages help introduce basic
participation in computing by creating an engaging and cul- computational thinking skills required to learn programming
turally relevant learning experience using a STEAM (sci- in a fun, interactive digital multimedia setting. One of the
ence, technology, engineering, arts and mathematics) ap- primary motivations of Scratch was to “add programmability
proach [19]. Students write code (in Python or JavaScript) to media-manipulation activities that are popular in youth
to creatively and algorithmically manipulate audio samples culture” [17]. There are many tools currently available that
from a loop library. EarSketch offers a tightly integrated make this possible for a variety of applications at varying
creative and pedagogical environment that captivates stu- levels of scalability. Max/MSP [3] and Pure Data [20], for
dents by making abstract computing concepts relevant in example, are the most commonly used visual programming
a context that borrows from the paradigm of digital audio languages by artists, composers and performers.
workstations and music production while remaining readily Good visual programming tools transcend human language
accessible to those without any prior experience with music barriers and are easily understood by people all over the
or music technology [18]. globe [23]. K.N. Whitley, in detail, examines empirical data
Since development of EarSketch began in 2011, it has been from studies that show how visual representation can in-
accessed by over 40,000 users in all 50 states in the US and crease comprehension compared to text [24]. The current
over 100 countries through summer camps and academic version of EarSketch has been very successful with students
courses. It has also been incorporated into a music tech- who are teenagers. However, anything from complex syntax
nology MOOC [11] that has reached over 35,000 students. to verbose error messages on the console can lead to distrac-
1
tion and loss of interest. Cyndi Rader, in [21], talks in depth
Most of the content for sections 1.1 and 1.2 was borrowed about the degrees of comprehension with regard to childrens’
from [16]
understanding of visual programming environments. In or-
der to make sure the learning is engaging and productive,
one needs to address the method of delivery. A graphical
environment has the ability to keep users captivated and
Licensed under a Creative Commons Attribution 4.0 International License (CC BY
4.0). Attribution: owner/author(s). thereby enables them to focus on the computational learn-
Web Audio Conference WAC-2016, April 4–6, 2016, Atlanta, USA
ing without worrying about low level intricacies related to

c 2016 Copyright held by the owner/author(s).
code syntax and semantics. It can also be thought of as a
link between e-learning and playing [6].
 Figure 1: The EarSketch Blockly Interface

2.1 EarSketch API abstraction

Figure 1 shows a sample script executed on EarSketch
with Blockly. Before proceeding further, it is worth review-
1.4 Relevant frameworks ing some of the core EarSketch APIs. The focus of this
EarSketch resides completely on the web and uses the Web discussion will be restricted to the following API functions
Audio API for all of its audio scheduling and processing. - setTempo(), fitMedia(), setEffect() and makeBeat().
EarSketch previously supported coding in either JavaScript The project tempo is set using the setTempo() function.
or Python through a text editor integrated into the browser- An audio file is placed on the multi-track timeline us-
based interface. To add support for visual programming to ing the fitMedia() function. Audio files are specified by
EarSketch, we needed a simple and robust framework to constants. EarSketch comes bundled with over 4000 audio
author and execute visual code directly in the browser with- loops in a variety of genres like hip-hop, soul, rock, techno
out any external plugins and dependencies. The approach and house that we commissioned from Richard Devine, an
to visual programming also needed to convey key concepts experimental electronic musician and sound designer, and
of computing like conditionals and loops which are essential Young Guru, an audio engineer and DJ best known for his
to the EarSketch curriculum as it is taught in schools. long-running collaboration with Jay Z. The remaining argu-
Scratch [4] and Blockly [1] were shortlisted to be the plau- ments to fitMedia() specify a track number, start measure,
sible options. Both Scratch and Blockly are open source, and end measure. Figure 2 shows the fitMedia() API in
customizable, client based, and completely cross browser. context, juxtaposed with the python equivalent . All the
Customization of blocks is imperative for EarSketch as it blocks have tool tips built in to guide the user filling the
provides functions that closely mimic popular operations in arguments of the block. The blocks also take care of type
a digital audio workstation (DAW) workflow such as placing checking. For example, a text field cannot be added to a
audio clips on a multi-track timeline, adding effects and step field that expects a number.
sequencing rhythms. Ultimately, Blockly was chosen, as it The setEffect() function adds an effect to a particular
allowed exporting from blocks to JavaScript, Python and track. For example, the following statement
Dart, had a stable code base that was conducive to expan- setEffect(1, VOLUME, GAIN, -60, 1, 0, 5)
sion and customization and could readily be injected into adds a volume effect on track 1 whose gain is being au-
the EarSketch UI. tomated between -60db and 0db between measures 1 and 5.
Figure 3, shows a subset of the supported effects in a drop
down fashion.
2. DESIGN AND WORK FLOW EarSketch’s makebeat() function allows users to create
The following section goes in depth into the design and rhythmic beats and phrases by using strings to piece to-
implementation of EarSketch APIs with Blockly. All the gether contents of different audio files at a 16th note reso-
EarSketch APIs were redesigned and implemented using the lution. Borrowing from Thor Magnusson’s ixi lang [15] and
Block Factory utility provided by Blockly. A UI button with Freeman and Van Troyer’s LOLC [13], our API uses a string
the icon “B” located on the top left corner, allows the user representation to sequence individual sixteenth notes over a
to switch between the Blockly editor and the code editor. full measure. The notation is fairly straightforward; a num-
 Figure 2: Blockly abstraction for fitMedia and equivalent Python code

Figure 3: Dropdown list of effects supported by setEffect

ber represents a single audio file or an index in a list of audio 3. CONCLUSION AND FUTURE WORK
files, a “+” sign extends the duration of the preceding sound In order to make this project appealing to a broader au-
by a sixteenth note, and a “-” sign indicates a rest. dience, further refinement and evaluation is required. The
Bolstered by the default libraries provided by Blockly (such criteria for evaluation must be chosen carefully. From the
as logic, loops, text and variables), the implemented frame- point of view of cognitive factors involved in visual program-
work is now capable of effectively leveraging the EarSketch ming, Blackwell et al.[9] lists out various intelligible ’dimen-
approach to teaching computer science in a visual program- sions’ such as Abstraction gradient, Closeness of mapping
ming context. and hidden dependencies. Their research highlights the im-
portance of the underlying paradigms of visual programming
languages (control flow versus data flow) and that it must
2.2 Saving and exporting be considered before performing empirical evaluations.
Blockly provides low level routines to switch between DOM There are several constraints with this environment that
and XML. The XML for these groups of blocks in the main need to be addressed. It is difficult to write complex scripts
workspace is the same as Blockly’s XML save format. The using Blockly due to limited available real estate on the can-
equivalent XML is stored just like any other JavaScript or vas. Although Blockly supports collapsing of blocks, it is not
Python script on the EarSketch server. When the user at- a scalable solution for larger projects. Unlike the code edi-
tempts to load his or her Blockly XML, the Blockly editor tor which supports multiple tabs, Blockly follows a singleton
is automatically brought up and populated with the corre- paradigm. Because of this, the user is limited to having one
sponding DOM. With a single click, the user can also export script open at a time. It is also challenging to address cer-
the Blockly code to Python or JavaScript code in a new tab tain basic computing concepts such as variable scope within
in the code editor. This can be particularly useful to illus- Blockly.
trate how abstractions translate to high level code. It also Despite the aforementioned constraints, there is opportu-
enables enthusiastic individuals to adopt different method- nity for improvement with regard to further abstraction and
ologies for composition, for example to design a skeletal improved user experience. The setEffect() API, for in-
framework via blocks and perform some advanced editing stance, can be broken down into separate entities for creating
through code. It should be noted however, that unlike Pen- the effect and applying automation envelopes on them. In-
cilCode [8] and Code.org’s App Lab [2], bidirectional con- telligently capturing logical errors like infinite loops and un-
version between blocks and text views of code is not yet reachable code sections leaves less room for confusion while
possible: users can only convert from blocks to text code debugging. Also, the notion of bidirectional conversion be-
but not the other way.
tween code and blocks opens a new avenue for EarSketch to [12] J. Freeman, B. Magerko, T. McKlin, M. Reilly,
explore. Animations between conversions and re-rendering J. Permar, C. Summers, and E. Fruchter. Engaging
unformatted text into clean blocks are some of the things underrepresented groups in high school introductory
users can benefit from. computing through computational remixing with
The project also stands as an example of the flexibility of earsketch. In Proceedings of the 45th ACM technical
authoring web applications with Web Audio API. Before the symposium on Computer science education, pages
advent of the Web Audio API, audio-intensive applications 85–90. ACM, 2014.
such as EarSketch could only be developed as standalone [13] J. Freeman and A. Van Troyer. Collaborative textual
desktop application software. The Web Audio API enables improvisation in a laptop ensemble. Computer Music
EarSketch to be deployed in the browser, where it can eas- Journal, (2):8, 2011.
ily integrate a variety of third-party APIs, frameworks, and [14] D. Kumar. Digital Playgrounds for Early Computing
web services to add new functionality to the platform much Education. ACM Inroads, 5(1):20–21, Mar. 2014.
more quickly than would be possible with desktop software. [15] T. Magnusson. The IXI lang: A SuperCollider parasite
The rapid integration of Blockly into EarSketch to enable a for live coding. International Computer Music
visual programming paradigm is one such example. Because Conference, pages 503–506. San Francisco,
web development tools and APIs make it relatively trivial to Huddersfield, International Computer Music
integrate new coding paradigms into the EarSketch environ- Association, Centre for Research in New Music
ment, our team has been able to concentrate its resources University of Huddersfield, 2011.
on exploring which programming paradigms map best onto [16] A. Mahadevan, J. Freeman, B. Magerko, and J. C.
digital audio workstation workflows and onto computer sci- Martinez. Earsketch: Teaching computational music
ence learning objectives, and on API and curricular design remixing in an online web audio based learning
that supports these approaches. environment. In Proceedings of the 2015 Web Audio
Conference, Paris, France, 2015.
4. ACKNOWLEDGMENTS [17] J. Maloney, M. Resnick, N. Rusk, B. Silverman, and
This material is based upon work supported by the Na- E. Eastmond. The scratch programming language and
tional Science Foundation under Grant Nos. CNS #1138469 environment. ACM Transactions on Computing
and DRL #1417835. Many thanks to the entire EarSketch Education (TOCE), 10(4):16, 2010.
project team (http://earsketch.gatech.edu/personnel). [18] S. McCoid, J. Freeman, B. Magerko, C. Michaud,
T. Jenkins, T. Mcklin, and H. Kan. EarSketch: An
integrated approach to teaching introductory
5. REFERENCES computer music. Organised Sound, 18(2):146–160,
[1] Blockly - https://blockly-games.appspot.com/ Aug. 2013.
accessed: 04/21/2015. [19] N. Park and Y. Ko. Computer Education’s
[2] Code.org’s app lab - https://code.org/educate/applab Teaching-Learning Methods Using Educational
accessed: 02/17/2016. Programming Language Based on STEAM Education.
[3] Max / msp - https://cycling74.com/ accessed: In J. J. Park, A. Zomaya, S.-S. Yeo, and S. Sahni,
04/21/2015. editors, Network and Parallel Computing, number
[4] Scratch - https://scratch.mit.edu/ accessed: 7513 in Lecture Notes in Computer Science, pages
04/21/2015. 320–327. Springer Berlin Heidelberg, Jan. 2012.
[5] Wavepot - http://wavepot.com/ accessed: 04/21/2015. [20] M. Puckette et al. Pure data: another integrated
[6] A. Baratè, M. G. Bergomi, and L. A. Ludovico. computer music environment. Proceedings of the
Development of Serious Games for Music Education. Second Intercollege Computer Music Concerts, pages
Journal of e-Learning and Knowledge Society, 9(2), 37–41, 1996.
May 2013. [21] C. Rader, C. Brand, and C. Lewis. Degrees of
[7] D. Bau. Droplet, a Blocks-based Editor for Text Code. comprehension: children’s understanding of a visual
J. Comput. Sci. Coll., 30(6):138–144, June 2015. programming environment. In Proceedings of the ACM
[8] D. Bau and D. A. Bau. A Preview of Pencil Code: A SIGCHI Conference on Human factors in computing
Tool for Developing Mastery of Programming. In systems, pages 351–358. ACM, 1997.
Proceedings of the 2nd Workshop on Programming for [22] M. Resnick, J. Maloney, A. Monroy-Hernández,
Mobile Touch, PROMOTO ’14, pages 21–24, New N. Rusk, E. Eastmond, K. Brennan, A. Millner,
York, NY, USA, 2014. ACM. E. Rosenbaum, J. Silver, B. Silverman, and Y. Kafai.
[9] A. F. Blackwell, K. N. Whitley, J. Good, and Scratch: Programming for All. Commun. ACM,
M. Petre. Cognitive factors in programming with 52(11):60–67, Nov. 2009.
diagrams. Artificial Intelligence Review, [23] N. C. Shu. Visual programming. Van Nostrand
15(1-2):95–114, 2001. Reinhold New York, 1988.
[10] S. Cooper, W. Dann, and R. Pausch. Teaching [24] K. N. Whitley. Visual programming languages and the
objects-first in introductory computer science. In empirical evidence for and against. Journal of Visual
ACM SIGCSE Bulletin, volume 35, pages 191–195. Languages & Computing, 8(1):109–142, 1997.
ACM, 2003.
[11] J. Freeman. Survey of music technology
https://www.coursera.org/course/musictech, October
2014.