elasticsearch
Shifa Khan github/shifakhan @findshifa
Why do we need search options?
We need it to make our lives easier Find stuff that is relevant to us Find it faster
How do we add seach features to our projects?
ptions available!
Standard "atabase #uery $ucene libraries Solr Sphin% &lasticsearch
elasticsearch
'''
(opics for today)
Full (e%t Searching *&S(ful &S &lasticsearch in *ails +dvanced features of &S
,ote! &S - &lasticSearch
What is elasticsearch?
Database server Implemented with RESTful HTTP/JSON Easil scalable !hence the name #ased $n %ucene
elasticsearch"
Features of elasticsearch
Sche.a/free *eal/ti.e &asy to e%tend with a plugin syste. for new functionality +uto.atic discovery of peers in a cluster Failover and replication 0o..unity support! 1ultiple clients available in various (ire ge./ ruby client available for +ctive1odel integration
languages for easy integration
(er.inology
*elational database "atabase (able *ow 0olu.n Sche.a
&lasticsearch 2nde% (ype "ocu.ent Field 1apping
2nside &S
" 0/(45&
2,"&3
"ata Structure of &lasticsearch
+n 2nde% can have .ultiple types' " 0/(45& &ach type can have .ultiple docu.ents'
2,"&3
" 0/(45&
&ach docu.ent contains data in JSON for.at
6''7 6''7
"ocu.ent
Searches can also be perfor.ed in .ultiple indices " 0/(45&
2,"&3
" 0/(45&
How does elasticsearch work?
Full te%t searching
2nverted inde%ing
+nalysis
F8$$ (&3( S&+*0H
stars shine sky stars stars sky cloudy sky
9:;
9<;
9=;
F8$$ (&3( S&+*0H
stars shine sky stars stars sky cloudy sky
W *" stars shine sky cloudy
$ 0+(2 , :>< : :><>= =
F8$$ (&3( S&+*0H
stars shine sky stars stars sky cloudy sky
W *" stars shine sky cloudy
$ 0+(2 , :>< : :><>= =
F8$$ (&3( S&+*0H
stars shine sky stars stars sky cloudy sky
W *" stars shine sky
$ 0+(2 , : < : : < =
5 S2(2 , : :>< < = = < :
cloudy
F8$$ (&3( S&+*0H
stars shine sky stars stars sky cloudy sky
W *" stars shine sky
$ 0+(2 , : < : : < =
5 S2(2 , : :>< < = = < : +re these positions in the docu.ent are consecutive ?
cloudy
F8$$ (&3( S&+*0H
stars shine sky stars stars sky cloudy sky
W *" stars shine sky
$ 0+(2 , : < : : < =
5 S2(2 , : :>1 < = 2 < : We find the words in consecutive positions in "ocu.ent <
cloudy
+,+$4S2S
+naly?ing is e%tracting @ter.sA fro. given te%t' 5rocessing natural language to .ake it co.puter searchable'
*uby is a dyna.ic> reflective> general/purpose 5 language that co.bines synta% inspired by 5erl with S.alltalk/like features' *uby was first designed and developed in the .id/<BB:s by 4ukihiro C1at?C 1atsu.oto in Dapan'
Stopwords!/re.oval of words of less
se.antic significance
ruby dyna.ic reflective general purpose oop language co.bine synta% inspire perl s.alltalk feature first design develop .id <BB: yukihiro .at? .atsu.oto japan
*uby is a dyna.ic> reflective> general/purpose 5 language that co.bines synta% inspired by 5erl with S.alltalk/like features' *uby was first designed and developed in the .id/<BB:s by 4ukihiro C1at?C 1atsu.oto in Dapan'
$owercase and punctuation .arks
ruby dyna.ic reflective general purpose oop language co.bine synta% inspire perl s.alltalk feature first design develop .id <BB: yukihiro .at? .atsu.oto japan
*uby is a dyna.ic> reflective> general/purpose 5 language that co.bines synta% inspired by 5erl with S.alltalk/like features' *uby was first designed and developed in the .id/<BB:s by 4ukihiro C1at?C 1atsu.oto in Dapan'
Ste..er!/"eriving root of words
ruby dyna.ic reflective general purpose oop language co.bine synta% inspire perl s.alltalk feature first design develop .id <BB: yukihiro .at? .atsu.oto japan
&S +naly?er
+naly?er! 0onsists of one tokeni?er and .ultiple token filters eg! Whitespace> Snowball>etc
(okeni?er! 2t tokeni?es all words' Splits sentences into individual Eter.sE ' ,gra. and &dge,gra. highly useful for autoco.plete feature' 5ath hierarchy tokeni?ers'
(oken filter! +ctions on tokeni?ed words> basic lowercase to phonetic filters and ste..ers Favailable in .any languagesG
5opular +naly?ers
&S is easy to use' *eady.ade analy?ers for general usage'
Snowball H e%cellent for natural language
Standard tokeni?er> with standard filter> lowercase filter> stop filter> and snowball filter So.e fancy analy?ers! 5attern analy?ers For all the *egular e%pressions guys out there) CtypeC! CpatternC> CpatternC!CIIsJC
How do we access &lasticsearch?
How do we access &lasticsearch?
2ts *&S(ful
How do we access &lasticsearch?
2ts *&S(ful
K&(
5 S(
58(
"&$&(&
58( /inde%/type/id
action?
58( /inde%/type/id
where?
58( /twitterLdevelop.ent/type/id
58( /twitterLdevelop.ent/type/id
what?
58( /twitterLdevelop.ent/tweet/id
58( /twitterLdevelop.ent/tweet/id
which?
58( /twitterLdevelop.ent/tweet/<
curl /358( Elocalhost!B=::/twitterLdevelop.ent/tweet/<E /d E6 CtweetC ! C &lasticsearch is cool) C> Cna.eC ! C1r "eveloperC 7E
curl /358( Elocalhost!B=::/twitterLdevelop.ent/tweet/<E /d E6 CtweetC ! C &lasticsearch is cool) C> Cna.eC ! C1r "eveloperC 7E 6 CLinde%C!CtwitterLdevelop.entC> CLtypeC!CtweetC> CLidC!C<C> CLversionC! <> CokC!true
Si.ilarly>
curl /3K&( Ehttp!//localhost!B=::/twitter/tweet/<E
Si.ilarly>
curl /3K&( Ehttp!//localhost!B=::/twitter/tweet/<E
6 CLinde%C ! CtwitterC> CLtypeC ! CtweetC> CLidC ! C<C> CLsourceC ! 6 CtweetC ! C &lasticsearch is cool) C> Cna.eC ! C1r "eveloperC 7
+lso
K&( /inde%/Lsearch K&( /inde%<>inde%=/Lsearch K&( /indM/Lsearch K&( /inde%/type/Lsearch K&( /inde%/type<>type=/Lsearch K&( /inde%/typeM/Lsearch K&( /Lall/typeM/Lsearch
Search .ultiple or all indices FdatabasesG Search all types Fcolu.nsG
Si.ilarly>
"&$&(&
"elete docu.ent
curl /3"&$&(& Ehttp!//localhost!B=::/twitter/tweet/<E curl /3"&$&(& Ehttp!//localhost!B=::/twitter/E
"elete inde%
curl /3"&$&(& Ehttp!//localhost!B=::/twitter/tweet/LNueryE/d E6 Cter.C ! 6 Cna.eC ! CdeveloperC 7 7E
&lasticsearch in *ails
&S in your *+2$S app
<'2nstall &S fro. website' Start service' ='2n your browser check Ehttp!//localhost!B=::E for confir.ation' O'+dd EtireE to Ke.file' P'2n your .odel file Fsay Echwink'rbE G add/ include (ire!!1odel!!Search include (ire!!1odel!!0allbacks Q'*un rake environ.ent tire!i.port 0$+SS-0hwink R'2n rails console type 0hwink'searchF@worldAG or 0hwink'tire'searchF@worldAG S'*esults)
*eNuest for 1odel'search
curl /3K&( Ehttp!//localhost!B=::/chwinkLdevelop.ent/chwink/Lsearch? /d E6 CNueryC! 6 CNueryLstringC! 6 CNueryC! CworldC 7 7 7E
+utoco.plete feature i.ple.entation
and
So.e e%a.ples of ready.ade and custo. analy?ers
#uerying
1ore types of Nueries and use cases! Faceting ! +llowing .ultiple filters 5agination ! $i.iting per page results Sort ! Sorting results by relevance and scoring Fusing &lasticsearchEs scoring algorith.G
+dvanced features of &lasticsearch
8S5 of &lasticsearch
T2K) "ata +nalysis in *eal (i.e
+uto.atic discovery of peers in a cluster Failover and replication
+uto.atic "iscovery 1odule
,ode <
,ode =
,ode O
+<
T<
T=
T<
+=
+O
0<
+=
+O
+<
+=
T<
+O
T=
+=
T=
+<
T=
Shard
*eplica
When a new node is added'''
+uto.atic "iscovery 1odule
,ode <
,ode =
,ode O
,ode P
+<
T=
0<
T=
+<
T=
+O
+=
+= T< += +O T< +O T< T=
T<
+<
High +vailability
For each inde% you can specify!
,u.ber of shards &ach inde% has fi%ed nu.ber of shards Shards i.prove inde%ing perfor.ance ,u.ber of replicas &ach shard can have :/.any replicas> can be changed dyna.ically *eplicas i.prove search perfor.ance
(ips
1ake sure you .ake separate indices for test and develop.ent environ.ent &g! +dd inde%Lna.e C0hwinkLU6*ails'env7C to your .odel file
"uring tests whenever you save an object .ake sure you add ! 0hwink'tire'inde%'refresh after each test case'
1ake sure you delete and recreate the inde% after tests' &g! 4ou can add this to your *spec configuration file config'afterF!eachG do 0hwink'tire'0hwinkLtest'delete 0hwink'tire'createLelasticsearchLinde% end "ebug using log files ! +dd to (ire config file! logger CtireLU6*ails'env7'logC
*oo. for e%ploration'''
Tig "esk plugin)
5lug/in for analysis of your search data)
*oo. for e%ploration'''
Tig "esk plugin)
5lug/in for analysis of your search data)
*oo. for e%ploration'''
Tig "esk plugin)
5lug/in for analysis of your search data)
,ow you can go ahead and start e%ploring &lasticsearch for yourself))
(hank you)
#uestions are welco.e)
