A Framework for Developing Multitouch Applications to

Enhance K-12 Education

Students:
Robyn Krage
Computer Science
South Dakota School of Mines & Technology
Rapid City, SD
[email protected]

Lori Rebenitsch
Physics
South Dakota School of Mines & Technology
Rapid City, SD
[email protected]

Jaelle Scheuerman
Computer Science
South Dakota School of Mines & Technology
Rapid City, SD
[email protected]

Faculty Advisor:
Dr. Antonette Logar
Computer Science
South Dakota School of Mines & Technology
Rapid City, SD
[email protected]

Abstract:
Multitouch platforms offer new and exciting ways of interacting with computers.
Traditional input devices, such as the mouse or keyboard, only allow input from one user
at a time. However, multitouch gestures allow for multiple inputs and often resemble
physical interactions that are more natural and intuitive for users. Additionally, large
multitouch interfaces allow more than one person to collaborate on the same screen
easily. This encourages people to work together and communicate. Multitouch
applications have the potential for offering a new hands-on approach to learning. Their
interactive nature is engaging and appeals to a wide variety of learning styles, and the use
of large interactive multitouch screens encourages collaboration. Science and math
education, in particular, can benefit from the highly interactive nature of multitouch
technology. With access to a multitouch computer and applications, students can work
together to learn concepts and solve problems.

1
Because of the positive affect that interactive learning can have on students, teachers
from different backgrounds need to have access to multitouch technology. The goal of
this project is to give educators the ability to leverage multitouch technology in their
classrooms. The solution that was developed will allow users to create multitouch
presentations using images, video, text, and sound. These presentations will allow basic
gestures, including zoom, rotate, and pan. A Creator program will allow teachers to
develop the presentations on any computer running Windows XP or higher. After the
presentation is completed, a Viewer program will run the presentation on a multitouch
enabled computer running Windows 7 and process incoming touch information. This
toolset has been developed with accessibility and affordability in mind.
This application toolset was developed by a team of undergraduate students. It is written
in C# and makes use of the Windows Presentation Foundation and Windows Touch API.
A demonstration will be included with the presentation. This project is supported by the
Computing Research Association’s Status of Women in Computing Research, the NASA
Space Grant, and the South Dakota School of Mines and Technology.

Introduction
The Blob Multitouch Designer is an effort to bring emerging multitouch technology into
K-12 classrooms. It allows teachers to create interactive presentations that encourage
collaboration between students. The interactive presentations use images, text, video, and
audio. Using different types of media encourages students with many learning styles to
participate. The images, text, and video appeal to visual learners. Auditory learners will
appreciate the audio clips and video as well. The hands-on nature of the Multitouch
hardware engages kinesthetic learners. Finally, by bringing exciting technology into
classrooms, children and teenagers are inspired to pursue further education in math and
science.

The framework consists of three applications. First, the Designer allows teachers to create
presentations from a regular desktop computer. These presentations can then be loaded
into the Viewer program that is installed on the multitouch system. The third application
is a demo program that can be used in the classroom or as a tutorial for the teachers
creating the presentations.

Blob Multitouch Creator

2
User Interface

Figure 1: Creator Interface

The Blob Multitouch Creator is the first part of the Blob Multitouch Designer. Teachers
use the Creator program to design Multitouch presentations for their students.
Presentations consist of images, videos, audio, and text. The user interface is shown
above in Figure 1. The user interface has a clear, unobtrusive menu bar at the top which
easily allows teachers to choose the multimedia element to be added. The playback
control for audio and video is also located on the menu bar. Each button on the menu bar
is defined below.

File: Users can select to start a new scene, load an existing scene, or close the program.
Text: This allows text to be added to the workspace. The font type, size, and color can be
selected.
Image: The user can select an image to add to the workspace.
Video: The user can select a video to add to the workspace.
Sound: The user can select a sound file to play in the background of the presentation.
Properties: The user can change the properties of an existing object in the scene. The
properties include height, width, and allowable gestures.
Remove: Removes an element from the workspace.
Export: Saves the presentation in a format recognizable by the Viewer program.

Features and Functionality

The main function of the Creator program is to prepare presentations for the Viewer.
When images, video, and text are added a number of properties can be specified.
The filename can be chosen via the Browse button. Height and Width are specified from
the dialog box. Movement, Rotation, and Scaling refer to the gestures that will be
allowable to this object in the Viewer. If none are selected, the object will be static. Links

3
To will connect an object to another scene when that object is clicked. A XAML file is
chosen for this with the Browse button. The dialog box to add text has an additional
feature that allows the user to change the font style and size of the text.

Figure 2: Adding an Image to a Scene

Underlying Technology
The Blob Multitouch Creator uses C# with WPF. This increases its compatibility with the
Viewer program. WPF accommodates a wide variety of layout styles and incorporates
multimedia. Using WPF made it easy to allow images, video, and sound into
presentations.

The file is exported from the Creator in XAML format. The XAML format is necessary
for the Viewer to use WPF and the Touch libraries. The Export feature of the Creator
program also copies the multimedia files to a resources folder. This makes the
presentation more portable and increases its ease of use. While Windows 7 is necessary
for the Viewer program, the Creator program can run on any PC with Windows XP or
higher. This allows a greater deal of portability so that teachers can make a presentation
without the Multitouch hardware from work or home.

Multitouch Viewer

4
Overview:
The Viewer was developed in Microsoft Visual Studio 2010 using C#, .NET 4.0 and the
Windows Presentation Foundation. It uses the TouchUp event to handle clicks and the
Manipulation class available in .NET 4.0 to handle the panning, zoom and rotate
transformations that happen when a user interacts with the scene. It runs on multitouch-
enabled hardware running Windows 7 and imports scenes exported from the Creator to
display them. It is made up of two main components, a graphical user interface and a
scene loader.

Figure 3: Viewer Structure

Graphical User Interface

The graphical user interface displays a container for the loaded scene as well as an Import
Presentation button, gesture icons and a play/stop button for any audio that may be in the
scene. Gesture icons display when the user is interacting with an object in the scene to
visually show the gestures that are allowed for that object.

Scene Loader
The scene loader is responsible for importing a XAML file that was exported by the
Creator. It loads the file and displays it in the presentation window. It also parses
through the file looking for information about the gestures assigned to each object, and
attaches the appropriate event handlers. Finally, it copies the associated scenes, images,
video and text to a temporary location for easy access by the Viewer.

Supported Objects and Gestures

5
The Viewer supports all the objects and gestures that can be exported by the Creator.
Supported objects include images, video, text, and sound. These objects are listed in a
XAML file which must follow the expected format to display properly. Supported
gestures include drag, zoom, rotate, or any combination of these. A click action is also
supported, and this will load a specified scene into the Viewer on click.

Handling Events for Gestures

The Manipulation class includes several event handlers that can be programmed to handle
multitouch manipulations including Rotate, Scale, Translate, TranslateX, TranslateY or
all of the above. The two events used by the viewer include the
ManipulationStartingEvent and ManipulationDeltaEvent. The possible gesture types are
specified in the ManipulationStartingEvent. The Viewer application allows up to three
different gesture types per object. Specific actions to take while these manipulations are
being registered are covered in the ManipulationDeltaEvent. Any object that will be
using the manipulation events must have the IsManipulationEnabled event set to true.
The Viewer also supports the TouchUpEvent on objects that need to register a click and
load a new scene into the Viewer. The scene file location is saved in a hidden TextBox
field that follows the naming convention. When the Viewer program finds an object that
is clickable in the XAML file it searches for its associated text field to get the scene
location. When a user touches the clickable object, the Viewer proceeds to load the new
scene and display it.

Assigning Events to Dynamically Loaded XAML

XAML that is dynamically loaded into an application cannot have event handlers
associated with it, so an alternative method of attaching events for gestures had to be
developed. It was decided to use a naming convention for objects in the scene. The
naming convention would specify the types of gestures allowed for that object. The table
below summarizes the expected naming scheme. The ObjectName can be any
descriptive name for the object and the OptionsAllowed must be one of the values in the
table.
Example Object Name: [ObjectName]_[OptionsAllowed]

OptionsAllowed Description Constraints

All All gestures are allowed for the object. None
Translate Drag gesture is allowed for the object. None
Scale Zoom gesture is allowed for the object. None
Rotate Rotate gesture is allowed for the object. None
RotateTranslate Rotate and Drag gestures are allowed for None
the object.
Rotate Scale Rotate and Zoom gestures are allowed None
for the object.

6
TranslateScale Drag and Zoom gestures are allowed for None
the object.
Click Clicking on the object will load a newMust have an
scene associated TextBox
field.
Click_Scene Contains the location of a scene to load Must have an
when an associated object is clicked. associated object with
the Click option.
Table 1: Naming Conventions for Supporting Gestures in the Viewer

Once the scene loader has completed loading the file, the scene is displayed in the
presentation window. Users will be able to interact with the objects in the scene as
specified by the assigned gestures. If a video file was loaded, it will display in a
StackPanel that includes play, pause, and stop buttons. If an audio file was loaded, a play
button appears in the toolbar.

Demonstration
Now let’s take a look at how the Creator and Viewer can be utilized in a complete
presentation. For one’s benefit, the framework includes a demonstration program on a
Timeline of Women in Science. We will use this program to showcase the framework’s
capabilities.

Figure 4: Timeline Opening Scene

First, the demo program is opened to its initial page. One can see a bar of different time

7
periods set across the top of the screen. The time period currently selected is before the
1700s. Below the bar is a selection of images referring to a specific scientist of the
selected period, in this case, scientists who were born before the year 1700.

Figure 5: Specific Scientist's Scene

By touching the image of a specific scientist, that scientist’s scene is brought up. The
scene typically includes the image of the scientist, a small commentary about the scientist
and her achievements, and occasionally an activity relating to the scientist’s field of
study.

Figure 6: Interacting With a Scene

The text and image of the scientist’s scene can be resized for the user’s benefit. By
touching the back button the scene of all the scientists born in that time period are
redisplayed.

8
One can explore the other time periods by simply touching the respective time period on
the timeline bar at the top of the screen.

Figure 7: More Scenes

As one can see, the user(s) can explore a wide variety of women scientist at one’s own
pace and in the order of one’s choosing. By showcasing all these women scientists, girls
are hopefully encouraged to pursue higher education and consider careers in science and
technology.

Figure 8: Grace Hopper Scene

9
Benefits of Using Educational Multitouch Applications in the
Classroom
Multitouch applications have other benefits as well. By utilizing touch, kinesthetic
learners are more engaged in the program and thus the learning process. Visual learners
benefit from the visual aspect of the presentations with its images, videos, and text.
While auditory learners benefit from the videos and sounds added to the presentations.
This means that the basic three learning styles are all engaged in the multitouch
presentations.

Conclusion
In conclusion, the Blob Multitouch Designer is being developed to help educators
encourage students, particularly girls in K-12 schools to consider careers and higher
education in science and technology. It does this by giving the teachers the option to
create multitouch interactive presentations that engage students and promote interest in
learning. It consists of a Creator program which is used to build the presentations in a
user-friendly way. A Viewer application is used to display the presentations on
multitouch enabled hardware running Windows 7. Finally, a timeline of the history of
women in science and technology was created as an example of the capabilities of the
Blob Multitouch Designer.

10
References

Eden, Joel. Designing for Multi-Touch, Multi-User and Gesture-Based Systems Oct.
1995. Dr. Dobb’s World of Microsoft Developers. 3 Apr. 2009
<http://www.ddj.com/windows/216402697>.

Educause Learning Initiative. 7 Things You Should Know About Multi-Touch Interfaces.
May 2008. net.educause.edu. Educause. 23 Apr. 2009
<http://net.educause.edu/ir/library/pdf/ELI7037.pdf>.

Inkpen, Kori, Kellogg S. Booth, Maria Klawe, and Rena Upitis. Playing Together Beats
Playing Apart, Especially for Girls. Oct. 1995. University of British Columbia. 26
Apr. 2009 <http://www.cs.ubc.ca/>.

Riddle, Larry. "Women Mathematicians Chronological Index." Biographies of Women

Mathemeticians. Agnes Scott College, 3 Mar. 2010. Web.
<http://www.agnesscott.edu/lriddle/women/chronol.htm>.

Rodriguez, Jamie. "Introduction to WPF 4 Multitouch." MSDN Blogs. Microsoft

Developer Network, 4 Nov. 2009. Web.
<http://blogs.msdn.com/jaimer/archive/2009/11/04/introduction-to-wpf-4-
multitouch.aspx>.

SMART Technologies ULC. Creating Classrooms for Everyone: How Interactive

Whiteboards Support Universal Design for Learning. 2009. SMART
Technologies ULC. 23 Apr. 2009
<http://www2.smarttech.com/st/enUS/Resourcelibrary/Publications/white+papers
/>.

"Laura Bassi (1711-1778)." Sunshine for Women. Web.

<http://www.pinn.net/~sunshine/whm2001/bassi.html>.

"Time Travel Timeline." Women's Adventures in Science. Web.

<http://iwaswondering.com/timeline.html>.

Toole, Betty. "Ada Byron, Lady Lovelace." Biographies of Women Mathematicians.

Agnes Scott College, 29 Apr. 2009. Web.
<http://www.agnesscott.edu/lriddle/women/love.htm>.

Women Scientists in History. Historia. Web.

<http://www.women-scientists-in-history.com/historia.html>.

11