Scribd
Upload
34 views14 pages

Slides - 02 01 estimNgramProbs

Uploaded by

man994412
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
34 views14 pages

Slides - 02 01 estimNgramProbs

Uploaded by

man994412
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

 

Language  
Modeling  
 
Es$ma$ng  N-­‐gram  
Probabili$es  
 
Dan  Jurafsky  

Es/ma/ng  bigram  probabili/es  


• The  Maximum  Likelihood  Es$mate  

count(w i"1,w i )
P(w i | w i"1 ) =
count(w i"1 )

c(w i"1,w i )
P(w i | w i"1 ) =
c(w i"1 )
!
Dan  Jurafsky  

An  example  

<s>  I  am  Sam  </s>  


c(w i"1,w i )
P(w i | w i"1 ) = <s>  Sam  I  am  </s>  
c(w i"1 ) <s>  I  do  not  like  green  eggs  and  ham  </s>  
 
Dan  Jurafsky  

More  examples:    
Berkeley  Restaurant  Project  sentences  

• can  you  tell  me  about  any  good  cantonese  restaurants  close  by  
• mid  priced  thai  food  is  what  i’m  looking  for  
• tell  me  about  chez  panisse  
• can  you  give  me  a  lis$ng  of  the  kinds  of  food  that  are  available  
• i’m  looking  for  a  good  place  to  eat  breakfast  
• when  is  caffe  venezia  open  during  the  day
Dan  Jurafsky  

Raw  bigram  counts  


• Out  of  9222  sentences  
Dan  Jurafsky  

Raw  bigram  probabili/es  


• Normalize  by  unigrams:  

• Result:  
Dan  Jurafsky  

Bigram  es/mates  of  sentence  probabili/es  


P(<s>  I  want  english  food  </s>)  =  
 P(I|<s>)        
   ×    P(want|I)      
 ×    P(english|want)        
 ×    P(food|english)        
 ×    P(</s>|food)  
             =    .000031  
Dan  Jurafsky  

What  kinds  of  knowledge?  


• P(english|want)    =  .0011  
• P(chinese|want)  =    .0065  
• P(to|want)  =  .66  
• P(eat  |  to)  =  .28  
• P(food  |  to)  =  0  
• P(want  |  spend)  =  0  
• P  (i  |  <s>)  =  .25  
Dan  Jurafsky  

Prac/cal  Issues  
• We  do  everything  in  log  space  
• Avoid  underflow  
• (also  adding  is  faster  than  mul$plying)  

p1 ! p2 ! p3 ! p4 = log p1 + log p2 + log p3 + log p4


Dan  Jurafsky  

Language  Modeling  Toolkits  


• SRILM  
• h_p://www.speech.sri.com/projects/srilm/  
Dan  Jurafsky  

Google  N-­‐Gram  Release,  August  2006  

…
Dan  Jurafsky  

Google  N-­‐Gram  Release  


• serve as the incoming 92!
• serve as the incubator 99!
• serve as the independent 794!
• serve as the index 223!
• serve as the indication 72!
• serve as the indicator 120!
• serve as the indicators 45!
• serve as the indispensable 111!
• serve as the indispensible 40!
• serve as the individual 234!

http://googleresearch.blogspot.com/2006/08/all-our-n-gram-are-belong-to-you.html
Dan  Jurafsky  

Google  Book  N-­‐grams  


• h_p://ngrams.googlelabs.com/  
 
Language  
Modeling  
 
Es$ma$ng  N-­‐gram  
Probabili$es  
 

You might also like