Julian Joseph University of Texas at Austin, Senior Julian88Tex@gmail.

com

Date: January 19th, 2012 Submitted to: Office of the Vice President for Research, Attn: URF 2011-2012

Descriptive Title
Kinect to iPad to Automated Fixtures: A New Performance Lighting Method with Motion

Thesis Statement
To create an interface between a motion-tracking device and an automated lighting fixture that is effective, easy to use, and affordable with direct application in live experimental performances, but with potential to expand to more conventional settings. In this proposal and project I will focus on the use of this interface by the performer on stage in a live situation.

Problem/Hypothesis
Stage lighting is becoming more complex as live entertainment grows in scale and creativity. With a modern automated light, a lighting technician must control a variety of parameters including position, color, size, pattern, and speed. However, recent developments in the gaming and mobile device industry allow for a certain amount of control to be given to performers in more experimental situations at a relatively low cost. A relationship between the Kinect, iPad, and automated lighting fixtures in particular has yet to be fully explored. By creating a way to interface with automated lighting through a Kinect motion-tracking device in conjunction with an iPad for remote control, the performer will be able to handle all of the parameters listed previously in a live setting with relative ease. In fact, musicians today, such as Skrillex, are already beginning to use the Kinect1. On his 2011 tour, Skrillex enhanced his concert with digital representations of himself while performing as seen in Figure 1. This project will explore expanding the use of a Kinect to include automated lighting fixtures. Figure 1 - Skrillex1 With this setup, performers can use the Kinect to follow their actions on stage and will be able to control lights. This means that a light can mimic their movements and/or change its color or shape based on the performers movements on the stage. Additionally, the Kinect will allow a light to follow the exact location of the performer. Essentially, the automated light could become a tracking spotlight, or followspot, without the need for a human operator.

Julian Joseph University of Texas at Austin, Senior [email protected]

The iPad serves a dual purpose. The first is that it allows the performer to switch between parameters of an automated fixture and among the fixtures that are going to be controlled. Having an interface that is inexpensive and simple to use is a must for a live environment. The iPad, in conjunction with the program Luminar, allows for this while still providing room for artistic freedom and creativity. More in depth explanation of this is included in the section on methodology. The second purpose for the iPad is that it currently provides the simplest interface between a Kinect and an automated fixture. Without it, complex programming would be required which is beyond the scope of most performers.

Methodology
To control an automated light with a Kinect (Figure 2), the light must be able to understand commands being given to it. In other words, a program (intermediary protocol) that translates Kinects computer language into a language that automated fixtures can understand is needed. The most functional intermediary protocol for this purpose is Open Sound Control (OSC)2. Open Sound Control is similar to Musical Instrument Digital Interface (MIDI) in that it connects two devices together so that they can trigger and sync with each other. OSCs main difference is that it connects two devices through IP address, similar to how two computers can connect to each other over a wireless network.

Figure 1 - Workflow of Kinect/iPad/Automated Light Interface

Julian Joseph University of Texas at Austin, Senior [email protected]

OSC data will be collected from the Kinect using a program called Synapse5 (Figure 3) on a Mac computer. Once OSC data is received by the computer, it will be manipulated using a programming language called Pure Data3. Pure Data is required in order to covert resolution of the Kinect to the resolution of a light. It is similar to how a laptop computer converts its display resolution when connecting to an external monitor or projector.

Figure 2 - Kinect4 After OSC data is collected and converted properly with Pure Data, it will be sent wirelessly over a network to an iPad running an application called Luminar6 (Figure 4) that can convert OSC data into lighting commands. The iPad will be wirelessly connected to a router and then an adapter which feeds directly to an automated lighting fixture. Essentially, the light will then be receiving commands indirectly from the Kinect. A diagram of this procedure is shown in Figure 1.

Figure 3 - Synapses Kinect output5

At this point light parameters can begin to be manipulated by the Kinect. For instance, the XY space that the Kinect sees can be mapped to the pan and tilt of a light. Taking one step to the left will trigger a numerical change in data being sent from the Kinect, which will be converted into light language and trigger the light to respond by moving left. This association of the XY axis of the Kinect to the pan and tilt on the light can be modified with the iPad to instead control color. So when the user steps to the left, instead of the light moving, it will turn more red or more blue. Of course, this can also be replicated for any other parameter as well. Through further experimentation, default presets for these parameters can be established and then easily tweaked on the spot in a performance by the user. Figure 4 - Luminar for the iPad6

Julian Joseph University of Texas at Austin, Senior [email protected]

Expected Outcome
The Kinect and iPad are easy to use, affordable, and have motion-tracking features that allow their users to control automated lighting on the fly in a way that is visually interesting and unique to an audience. This method will not likely replace lighting consoles and designers for tradition entertainment events like theatre, however, it is likely to enhance more experimental forms like dance, performance art, and live music. This methodology will be tested and used to develop new techniques for creating visual entertainment. This work will be documented and posted online so that others can replicate and improve upon it. Also, the following uses will be explored during this project in the hope of finding new applications: Programming a lighting show using body gestures instead of a control board Automatically focusing fixtures on an exact position of a person or object on stage Followspot control for a performance which automatically tracks the subjects position Combining real-time digital video from the Kinect with moving lights for a performance Combining body motion with other interactive devices like Wii controllers

Why a Fellowship is Necessary:

To realize this goal, some equipment must be purchased. Though the most expensive equipment, a robotic light, has been secured, the Kinect, iPad, Luminar, and several other intermediary tools are not freely available for this study. Additionally, once this area has been explored, the information and techniques will be shared with the students currently enrolled in the Automated Lighting Program at the Department of Theatre and Dance. An Undergraduate Research Fellowship will allow the purchase of equipment, assure thorough and accurate testing, and increase likelihood of the success of this project.
1Darren3179. Skrillex Ruffneck Bass @ Palladium Dallas.MP4 - YouTube. Digital image.YouTube - Broadcast Yourself. 29 Sept. 2011. Web. 22 Jan. 2012. <http://www.youtube.com/watch?v=K4USPVIGVEY>. 2"Introduction to OSC." Opensoundcontrol.org. Web. 20 Jan. 2012. <http://opensoundcontrol.org/introduction-osc>. 3Barknecht, Frank. "What Is Pd? PD Community Site." Pure Data PD Community Site. 04 May 2008. Web. 20 Jan. 2012. <http://puredata.info/docs/faq/whatispd>. 4Terdiman, Daniel. Microsoft Looks to Kinect as Game-changer | E3 2011 - CNET Reviews. Digital image. Product Reviews - Electronics Reviews, Computer Reviews & More - CNET Reviews. Microsoft. Web. 20 Jan. 2012. <http:// reviews.cnet.com/8301-21539_7-20007681-10391702.html>. 5Challinor, Ryan. "Synapse." Synapse for Kinect. 20 June 2012. Web. 20 Jan. 2012. <http://synapsekinect.tumblr.com/post/ 6610177302/synapse>. 6"Luminair for IPad - Multi-touch Wireless DMX Lighting Control." Synthe FX - Innovative Tools for Artists/designers Film/tv Architectural. Synthe FX, LLC, 2012. Web. 20 Jan. 2012. <http://synthe-fx.com/products/luminairipad>.