Oct.

5, 2012 Carolyn Remmler

Educational Software
Browser-based games
Multi-player games
Websockets (NodeJS, socket.io)
Other Javascript Libraries
- Box2dWeb.js
- Kinetic.js
- JSXGraph.js
- jQuery SVG Plug-in



High School Math Teacher
Stained Glass Artisan
Current Work
Past Work
What is WebRTC?

Web Browsers with Real-Time-Communication

Audio/Video Chat on the web.
Accessed through Javascript API.
Does not require plugins, downloads or installs.
Multiple browsers, multiple platforms.
http://www.webrtc.org/faq
How did we get here?

Graphic by Jimmy Lee / jimmylee.info
http://venturebeat.com/2012/08/13/webrtc-is-almost-here-and-it-will-change-the-web/
Javascript Session Establishment Protocol


Justin Uberti, Tech Lead, WebRTC,
http://www.youtube.com/watch?v=E8C8ouiXHHk
Peer-to-peer exchange of data

Justin Uberti, Tech Lead, WebRTC,
http://www.youtube.com/watch?v=E8C8ouiXHHk
We can use webRTC.io!


Who can use WebRTC? How?


Justin Uberti, Tech Lead, WebRTC,
http://www.youtube.com/watch?v=E8C8ouiXHHk
http://www.webrtc.org/running-the-demos
But wait... If WebRTC isn't on mobile, yet, what is
this guy doing?

It is an open source HTML5 Sip Client.


http://www.sipml5.org/index.html
Instead of using webRTC.io, they used Javascript to
implement the SIP Protocol.

Justin Uberti, Tech Lead, WebRTC,
http://www.youtube.com/watch?v=E8C8ouiXHHk
But we just want to implement audio/video chat in a
browser!

Let's use webRTC.io!

It's an abstraction layer for webRTC.

webRTC.io is to WebRTC as
socket.io is to WebSockets
https://github.com/webRTC/webRTC.io
Let's use webRTC.io to access WebRTC!


How does it work?

MediaStreams access to user's camera and mic
PeerConnection audio/video calls
DataChannels p2p application data transfer

Let's make a WebRTC mirror!
generative.edb.utexas.edu/webrtc-demos/mirror.html
MediaStreams PeerConnection DataChannels
<script src='webrtc.io.js'></script>
<!-- from https://github.com/webRTC/webrtc.io-client -->

MediaStreams PeerConnection DataChannels
<video id="me" autoplay></video>
Add a Video Tag.
Flip Video to show mirror image.
#me { -webkit-transform: rotateY(180deg); }
Display my video stream.
var PeerConnection = window.PeerConnection ||
window.webkitPeerConnection00;

if (PeerConnection) {
rtc.createStream({"video": true, "audio": true},
function(stream) {
rtc.attachStream(stream, 'me'); // <video id="me">
});
} else {
alert("Please use a WebRTC-enabled browser.");
}

MediaStreams PeerConnection DataChannels
Let's lighten and darken the mirror,
in real time!

MediaStreams PeerConnection DataChannels
<script src='webrtc.io.js'></script>
Lighten and darken the mirror.
MediaStreams PeerConnection DataChannels
$('#lighten').bind('click',function() {
if (videoOpacity > 0) {videoOpacity = videoOpacity - 0.1;}
$('#me').css('opacity',videoOpacity); // <video id="me">
});

$('#darken').bind('click',function() {
if (videoOpacity < 1) { videoOpacity = videoOpacity + 0.1;}
$('#me').css('opacity',videoOpacity);
});
<button id='lighten'>Lighten</button>
<button id='darken'>Darken</button>
Add buttons.
Let's make a WebRTC Photo Booth!

generative.edb.utexas.edu/webrtc-demos/photobooth.html
MediaStreams PeerConnection DataChannels
<script src='webrtc.io.js'></script>
generative.edb.utexas.edu/webrtc-demos/photobooth.html
MediaStreams PeerConnection DataChannels
Add video and canvas tags, and a snapshot
button.
<video id="me" autoplay></video>
<canvas id=snapshot></canvas>
<button id=snapPicture>Snapshot</button>
Create Video and Canvas Objects.
var myVideo = document.getElementById("me");
var snapshotCanvas = document.getElementById("snapshot");

Take a frame of the video. Add it to the canvas.
$('#snapPicture').bind('click', function() {
snapshotCanvas // to <canvas id='snapshot'>
.getContext('2d')
.drawImage(myVideo, 0, 0, // from <video id="me">
snapshotCanvas.width,
snapshotCanvas.height);
});
MediaStreams PeerConnection DataChannels
MediaStreams PeerConnection DataChannels
Let's make an Embossed WebRTC Mirror!
<script src='webrtc.io.js'></script>
<script src='seriously.js'></script>
<!-- from https://github.com/brianchirls/Seriously.js/ -->
generative.edb.utexas.edu/webrtc-demos/emboss.html
MediaStreams PeerConnection DataChannels
Add video and canvas tags.
<video id="me" autoplay></video>
<canvas id=snapshot></canvas>


seriously = new Seriously();
myVideo = seriously.source(#me); // source video
target = seriously.target('#snapshot'); // target canvas
emboss = seriously.effect('emboss');
emboss.source = myVideo; // emboss video frame
target.source = emboss; // draw to canvas
seriously.go();
Take video frame. Add emboss affect.
Draw it on canvas.
MediaStreams PeerConnection DataChannels
Let's Add Goofy WebRTC Glasses!
<script src='webrtc.io.js'></script>
<script src='ccv.js'></script>
<script src='face.js'></script>
<!-- from https://github.com/liuliu/ccv -->
MediaStreams PeerConnection DataChannels
Add video and canvas tags.
<video id="me" autoplay></video>
<canvas id=snapshot></canvas>
Load glasses image.
var glassses = new Image();
glasses.src = 'glasses2.png';
Keep detecting and drawing glasses.
playing = setInterval(function() {
showGlasses();
}, 200);
MediaStreams PeerConnection DataChannels
snapshotCanvas.getContext('2d')
.drawImage(myVideo,0,0,
snapshotCanvas.width, snapshotCanvas.height);

.
Draw video frame on canvas.
MediaStreams PeerConnection DataChannels
comp = ccv.detect_objects({
canvas: snapshotCanvas,
cascade: cascade,
interval: 4,
min_neighbors: 1
});



for (i = comp.length; i--; ) {
snapshotCanvas.getContext('2d')
.drawImage(glasses, comp[i].x, comp[i].y,
comp[i].width, comp[i].height);
}
.
Detect face(s).
Draw glasses on canvas.
MediaStreams PeerConnection DataChannels
Let's see the audio in real time!
<script src='webrtc.io.js'></script>
<!-- Chrome 23. Enable Web Audio Input. -->
generative.edb.utexas.edu/webrtc-demos/sound.html
MediaStreams PeerConnection DataChannels
Add video and canvas tags.
<video id="me" autoplay></video>
<canvas id=results></canvas>
Define Buffer and Create Canvas Objects.
var buflen = 1024;
var buf = new Uint8Array( buflen );
var canvas = document.getElementById('results');
var ctx = canvas.getContext('2d');
MediaStreams PeerConnection DataChannels
Connect it to another node for processing.
rtc.attachStream(stream, 'me'); // from video



audioContext = new webkitAudioContext();
mediaStreamSource =
audioContext.createMediaStreamSource( stream );



analyser = audioContext.createAnalyser();
analyser.fftSize = 2048;
mediaStreamSource.connect( analyser );
// output audio through analyzer

Attach the stream.

Create an audio node from the stream.
MediaStreams PeerConnection DataChannels
analyser.getByteTimeDomainData( buf );
// or analyser.getByteFrequencyData( buf );



ctx.clearRect(0, 0, canvas.width, canvas.height);
for (i=0;i<buf.length;i++) {
ctx.beginPath();
ctx.moveTo(i/2,200);
ctx.lineTo(i/2,200-buf[i]);
ctx.stroke();
}

Process buffer.
Draw buffer data on canvas.
MediaStreams PeerConnection DataChannels
function updateSound() {

// Add code to process buffer and draw to canvas.

rafID =
window.webkitRequestAnimationFrame( updateSound );
}
Keep updating sound.
MediaStreams PeerConnection DataChannels
By Jos Dirkson

1. Stream live audio with WebRTC

2. Record small WAV files with RecorderJS

3. Send WAV files to server with websockets

4. Convert WAV to FLAC with JavaFlacEncoder

5. Send audio in FLAC format to Undocumented Google API

6. Receive JSON string
WebRTC Speech Recognition
http://www.smartjava.org/content/record-audio-using-webrtc-chrome-and-speech-
recognition-websockets

Having fun?
having fun
Let's make a Video Call!
generative.edb.utexas.edu/webrtc-demos/videocall.html
MediaStreams PeerConnection DataChannels
<script src='webrtc.io.js'></script>
My Client
generative.edb.utexas.edu/webrtc-demos/videocall.html
MediaStreams PeerConnection DataChannels
Our Node.js Server
http://generative
.edb.utexas.edu
/webrtc-demos
/videocall.html
requires
webrtc.io.js

(client version)
http://generative
.edb.utexas.edu
:60020
requires
webrtc.io.js

(server version)
calls
if (PeerConnection) {
rtc.createStream({"video": true, "audio": true},
function(stream) {
rtc.attachStream(stream, 'me');
});
} else {
alert("Please use a WebRTC-enabled browser.");
}

Display my video stream.
MediaStreams PeerConnection DataChannels
<video id="me" autoplay></video>
<video id="you" autoplay></video>
Add our video tags.
Add a New Room button.
MediaStreams PeerConnection DataChannels
Create a new room.
$('#newRoom').bind('click', function() {
var randomString = createRandomString(); // = 'foo'
window.location.hash = randomString;
location.reload();
});
<button id='newRoom'>Create a New Room</button>
// From generative.edb.utexas.edu/webrtc-demos/videocall.html
// to generative.edb.utexas.edu/webrt-demos/videocall.html#foo
rtc.on('add remote stream', function(stream,socketId) {
console.log('peer ' + socketID + ' joined');
rtc.attachStream(stream,"you"); // <video id="you">
});
rtc.on('disconnect stream', function(data) {
console.log('peer ' + data + ' disconnected');
});

Find which room I'm in.
MediaStreams PeerConnection DataChannels
var room = window.location.hash.slice(1); // 'foo'
Connect to that room.
rtc.connect("ws://generative.edb.utexas.edu:60020",room);
Listen for a peer.
var app = require('express').createServer();

app.listen(60020); // port 60020, just because

var webRTC = require('webrtc.io').listen(app);
// from

Run my Node.js server.
MediaStreams PeerConnection DataChannels
webRTC.rtc.on('connect', function(rtc) {
console.log('user connected');
});

webRTC.rtc.on('send answer', function(rtc) {
console.log('answer sent');
});

webRTC.rtc.on('disconnect', function(rtc) {
console.log('user disconnected');
});

Listen to events on the Node.js server.
MediaStreams PeerConnection DataChannels
Send plain data with my Nurse Line App!
generative.edb.utexas.edu/webrtc/
MediaStreams PeerConnection DataChannels
I used socket.io for Websockets.
DataChannels are coming soon!
Can replace websockets connection for sending data.


It will have:
Peer-to-peer exchange of data
Low latency
High message rate/throughput
Optional unreliable semantics
Justin Uberti, Tech Lead, WebRTC, http://www.youtube.com/watch?v=E8C8ouiXHHk
MediaStreams PeerConnection DataChannels
Mesh
MediaStreams PeerConnection DataChannels
Justin Uberti, WebRTC
Tree
MediaStreams PeerConnection DataChannels
Justin Uberti, WebRTC
Comparing WebSockets and DataChannels!
MediaStreams PeerConnection DataChannels
WebSockets
Share events with server.

Broadcast to rooms.
Save client-specific data.
Send volatile messages.
Most current browsers.
Firewall issues.
Can run locally.
WebRTC
Share session info with server.
Share audio/video data with peer.
Broadcast to rooms.
Sure.
Send volatile messages.
Chrome, for now.
?
Can run locally.
generative.edb.utexas.edu/webrtc-demos/