JavaScript Speech Recognition
2 likes1,564 views
Speech recognition is the process of translating spoken words to text. It involves recording and digitizing audio, segmenting it into phonemes, applying a recognition model to analyze the phonemes against a lexicon and grammar, and returning a confidence-weighted transcript. Speech recognition accuracy is around 92% for English but lower for other languages. Mobile apps can use platform-specific APIs like Google Now on Android while the W3C specification allows cross-browser support. The related Speech Synthesis API can output responses by voice. Together these APIs enable interactive speech applications.
![Return a confidence weighted
result
[
{
"confidence":0.97335243225098,
"transcript":"hello"
},
{
"confidence":0.19940405040800,
"transcript":"helllow"
},
{
"confidence":0.19910827091000,
"transcript":"howlow"
}
]](https://image.slidesharecdn.com/javascriptspeechrecognition-150930203948-lva1-app6891/85/JavaScript-Speech-Recognition-23-320.jpg)
![Let's recognize some speech
varrecognition=newSpeechRecognition();
recognition.onresult=function(event){
if(event.results.length>0){
vartest1=document.getElementById("test1");
test1.innerHTML=event.results[0][0].transcript;
}
};
recognition.start();
Click to Speak
Replace me...](https://image.slidesharecdn.com/javascriptspeechrecognition-150930203948-lva1-app6891/85/JavaScript-Speech-Recognition-38-320.jpg)
![Let's do something a little less
trivial
recognition.onresult=function(event){
varresult=event.results[0][0].transcript;
varmusic=document.getElementById("music");
switch(result){
case"jazz":
music.src="jazz.mp3";
music.play();
break;
case"rock":
music.src="rock.mp3";
music.play();
break;
case"stop":
default:
music.pause();
}
};
Click to Speak](https://image.slidesharecdn.com/javascriptspeechrecognition-150930203948-lva1-app6891/85/JavaScript-Speech-Recognition-42-320.jpg)
JavaScript Speech Recognition
- 2. Who is this guy? @macdonst macdonst on Github simonmacdonald.com works at Adobe Apache Cordova core contributor nutty about speech recognition
- 3. The future won't be like Star Trek. Scott Adams, creator of Dilbert
- 5. Why do I care about speech rec?
- 8. Here's a conversation between two Cape Bretoners P1: jeet? P2: naw, jew? P1: naw, t'rly t'eet bye.
- 9. And here's the translation P1: jeet? P1: Did you eat? P2: naw, jew? P2: No, did you? P1: naw, t'rly t'eet bye. P1: No, it's too early to eat buddy.
- 10. Regular Alphabet 26 letters Cape Breton Alphabet 12 letters!
- 13. Speech recognition is the process of translating the spoken word into text.
- 14. The process of speech rec includes...
- 15. Record and digitize the audio data
- 17. Split data into phonemes
- 18. What is a phoneme? It is a perceptually distinct units of sound in a specified language that distinguish one word from another.
- 19. The English language has 44 distinct sounds Source: English language phoneme chart
- 20. By comparison, the Rotokas speakers in Papua New Guinea have 11 phonemes. But the !Xóõ speakers who mostly live in Botswana have 112 phonemes.
- 21. Apply the phonemes to the recognition model. This is a massive lexicon which takes into account all of the different ways words can be pronounced.
- 22. Analyze the results against the grammar
- 23. Return a confidence weighted result [ { "confidence":0.97335243225098, "transcript":"hello" }, { "confidence":0.19940405040800, "transcript":"helllow" }, { "confidence":0.19910827091000, "transcript":"howlow" } ]
- 24. Basically...
- 26. We want it to be like this 0:02
- 27. but more often than not... 0:25
- 28. Why is that? When two people talk comprehension rates are better than 97%
- 29. A really good english language speech recognition system is right 92% of the time
- 30. Where does that extra 5% in error rate come from? Vocabulary size and confusability Speaker dependence vs independence Isolated or continuous speech Initiated vs spontaneous speech Adverse conditions
- 31. Mobile Speech Recognition OS Application SDK Android Google Now Java API iOS Siri Many 3rd party Obj-C SDK's Windows Phone Cortana C# API
- 32. So how do we add speech rec to our app?
- 33. You may look at the W3C Speech API Specification
- 34. but only Chrome on the desktop has implemented that spec
- 35. But that's okay!
- 36. The spec looks like this: interfaceSpeechRecognition:EventTarget{ //recognitionparameters attributeSpeechGrammarListgrammars; attributeDOMStringlang; attributebooleancontinuous; attributebooleaninterimResults; attributeunsignedlongmaxAlternatives; attributeDOMStringserviceURI; //methodstodrivethespeechinteraction voidstart(); voidstop(); voidabort(); };
- 37. With additional event methods to control behaviour: attributeEventHandleronaudiostart; attributeEventHandleronsoundstart; attributeEventHandleronspeechstart; attributeEventHandleronspeechend; attributeEventHandleronsoundend; attributeEventHandleronaudioend; attributeEventHandleronresult; attributeEventHandleronnomatch; attributeEventHandleronerror; attributeEventHandleronstart; attributeEventHandleronend;
- 38. Let's recognize some speech varrecognition=newSpeechRecognition(); recognition.onresult=function(event){ if(event.results.length>0){ vartest1=document.getElementById("test1"); test1.innerHTML=event.results[0][0].transcript; } }; recognition.start(); Click to Speak Replace me...
- 40. ...if taking dictation gets you going
- 41. But I want to do something more exciting with the result
- 42. Let's do something a little less trivial recognition.onresult=function(event){ varresult=event.results[0][0].transcript; varmusic=document.getElementById("music"); switch(result){ case"jazz": music.src="jazz.mp3"; music.play(); break; case"rock": music.src="rock.mp3"; music.play(); break; case"stop": default: music.pause(); } }; Click to Speak
- 43. Which seems much cooler to me
- 44. Let's ask the web a question Click to Speak
- 46. Let's style our button with some CSS
- 48. by Nicholas Gallagher And we'll add some color using Speech Bubbles Pure-CSS-Speech-Bubbles
- 49. Then pull it all together!
- 51. But wait, why am I using my eyes like a sucker?
- 52. We'll output the answer using SpeechSynthesis
- 53. The SpeechSynthesis spec looks like this: interfaceSpeechSynthesis{ readonlyattributebooleanpending; readonlyattributebooleanspeaking; readonlyattributebooleanpaused; voidspeak(SpeechSynthesisUtteranceutterance); voidcancel(); voidpause(); voidresume(); SpeechSynthesisVoiceListgetVoices(); };
- 54. The SpeechSynthesisUtterance spec looks like this: interfaceSpeechSynthesisUtterance:EventTarget{ attributeDOMStringtext; attributeDOMStringlang; attributeDOMStringvoiceURI; attributefloatvolume; attributefloatrate; attributefloatpitch; };
- 55. With additional event methods to control behaviour: attributeEventHandleronstart; attributeEventHandleronend; attributeEventHandleronerror; attributeEventHandleronpause; attributeEventHandleronresume; attributeEventHandleronmark; attributeEventHandleronboundary;
- 57. Plugin repo's SpeechRecognitionPlugin - SpeechSynthesisPlugin - https://github.com/macdonst/SpeechRecognitionPlugin https://github.com/macdonst/SpeechSynthesisPlugin
- 58. * Working with Julio César (@jcesarmobile) to get iOS done Availability OS Recognition Synthesis Android ✓ ✓ iOS* Soonish Native to iOS 7.0+ Windows Phone × ×
- 60. For more information on hybrid applications Check out Nick Van Weerdenburg and Andrey Feldman presentation on Creating a Comprehensive Social Media App Using Ionic and Phone Gap 3:45pm today in 801A.
- 61. But wait, one more thing...
- 62. Speech recognition and speech synthesis are not well supported in the emulator and sometimes developing on the device can be a bit of a pain.
- 63. That's why I coded speechshim.js https://github.com/macdonst/SpeechShim
- 64. Chrome + speechshim.js = W3C Web Speech API on your desktop
- 65. Types of Speech Recognition Applications Voice Web Search Speech Command Interface Continuous Recognition of Open Dialog Domain Specific Grammars Filling Multiple Input Fields Speech UI present when no visible UI need be present Voice Activity Detection Speech Translation Multimodal Interaction Speech Driving Directions
- 67. THE END