solrとelasticsearchの比較

Transcript

1. elasticsearchͱSolrͷൺֱ ݉ࢁ ݩଠ @penguinana_ Monday, November 26, 12

2. ࣗݾ঺հ • ݉ࢁ ݩଠ @penguinana_ • ϨγϐݕࡧνʔϜ @ http://cookpad.com/ •

   Solr4.0 Monday, November 26, 12

3. SolrͷόʔδϣϯΞοϓΛ ݕ౼͍ͯ͠Δͱ͖... Monday, November 26, 12

4. Elasticsearch΋ ௐ΂ͨ΄͏͕͍͍ͷͰ͸ʁ Monday, November 26, 12

5. • Luceneϕʔε • HTTP API • ෼ࢄݕࡧOK • ೔ຊޠOK Monday,

   November 26, 12

6. • Luceneϕʔε • HTTP API • ෼ࢄݕࡧOK • ೔ຊޠOK طࢹײ

   Monday, November 26, 12

7. http://solr-vs-elasticsearch.com/ Monday, November 26, 12

8. ײ૝ • ػೳ໘Ͱෆ଍͸ͳ͍ • API͕։ൃऀʹ΍͍͞͠ • ༰қʹशಘͰ͖Δ • େن໛෼ࢄݕࡧҎ֎Ͱ΋༗༻ •

   SolrΛ࢖ͬͯͳ͚Ε͹ͬͪ͜Λຊ൪ʹ࢖͍ͨ ͍ʂ Monday, November 26, 12

9. αϯϓϧΛ࢖ͬͯ ͻͱ௨Γઆ໌͠·͢ Monday, November 26, 12

10. http://blog.livedoor.jp/techblog/archives/65836960.html Monday, November 26, 12

11. livedoorάϧϝ • Ϩετϥϯ৘ใ(21.4ສళ) • ళ໊ɺѻ͍ͬͯΔྉཧɺॅॴɺҢ౓ ܦ౓ɺΞΫηε਺ɺ࠷دΓฑߦ͖͔ Βͷڑ཭ɺetc... Monday, November 26,

    12

12. livedoorάϧϝ • ϨϏϡʔ৘ใ(20.5ສϨϏϡʔ) • ૯߹ධՁʢ5ஈ֊ʣ • งғؾɺ஋ஈɺαʔϏεɺຯ • ϨϏϡʔίϝϯτ Monday,

    November 26, 12

13. https://github.com/penguinco/ld_gourmet_search Monday, November 26, 12

14. ElasticsearchΛ࢖͏ • 1݅ొ࿥ͯ͠ɺ1݅ݕࡧ • ೔ຊޠͷѻ͍Λఆٛ • εΩʔϚఆٛ • औΓࠐΈ •

    ݕࡧ • είΞϦϯάͳͲͷௐ੔ Monday, November 26, 12

15. PUT curl -XPUT http://localhost:9200/twitter/tweet/1 -d ' { "user": "kimchy", "post_date":

    "2012-11-26T20:12:00", "message": "Trying out elasticsearch", "score": 5 } ' index type id Monday, November 26, 12

16. PUT curl -XPUT http://localhost:9200/twitter/user/kimchy -d ' { "name" : "Shay

    Banon" } ' index type id Monday, November 26, 12

17. GET curl -XGET http://localhost:9200/twitter/tweet/1 { "user": "kimchy", "post_date": "2012-11-26T20:12:00", "message":

    "Trying out elasticsearch", "score": 5 } } index type id Monday, November 26, 12

18. SEARCH curl -XGET http://localhost:9200/twitter/tweet/_search -d '{ "query" : { "term"

    : { "user": "kimchy" } } }' index type id { "user": "kimchy", "post_date": "2012-11-26T20:12:00", "message": "Trying out elasticsearch", "score": 5 } Monday, November 26, 12

19. REST API • υΩϡϝϯτͷ௥Ճɾ࡟আ • ઃఆͷ௥Ճɾ࡟আ • શ෦HTTP APIͰͰ͖Δ •

    εΩʔϚϑϦʔ Monday, November 26, 12

20. ೔ຊޠ $ curl -XGET 'localhost:9200/_analyze?pretty' -d 'ਆઘ' { "tokens" :

    [ { "token" : "ਆ", "start_offset" : 0, "end_offset" : 1, "type" : "<IDEOGRAPHIC>", "position" : 1 }, { "token" : "ઘ", "start_offset" : 1, "end_offset" : 2, "type" : "<IDEOGRAPHIC>", "position" : 2 } ] } Monday, November 26, 12

21. ೔ຊޠ AnalyzerΛมߋ͢Δ͜ͱͰରԠ kuromoji͕࢖͑·͢ʂ http://www.hirotakaster.com/archives/2012/11/ elasticsearch-kuromoji-plugin.php Monday, November 26, 12

22. kuromoji $ cd elasticsearch $ bin/plugin -install elasticsearch/elasticsearch-analysis-kuromoji/1.0.0 $ git

    clone git://github.com/elasticsearch/elasticsearch-analysis- kuromoji.git $ cd elasticsearch-analysis-kuromoji/ $ mvn clean package $ cp target/elasticsearch-analysis-kuromoji-1.2.0-SNAPSHOT.jar ../plugins/ analysis-kuromoji/elasticsearch-analysis-kuromoji-1.0.0.jar # restart elasticsearch Monday, November 26, 12

23. add analyzer $ curl -XPUT 'localhost:9200/test/' -d ' { "index":{

    "analysis":{ "tokenizer" : { "kuromoji" : { "type":"kuromoji_tokenizer", "mode":"search" } }, "analyzer" : { "kuromoji_analyzer" : { "type" : "custom", "tokenizer" : "kuromoji_tokenizer" } } } } } ‘ Monday, November 26, 12

24. kuromoji $ curl -XGET 'localhost:9200/test/_analyze? analyzer=kuromoji_analyzer&pretty' -d 'ਆઘ' { "tokens"

    : [ { "token" : "ਆઘ", "start_offset" : 0, "end_offset" : 2, "type" : "word", "position" : 1 } ] } Monday, November 26, 12

25. _analyze $ curl -XGET 'localhost:9200/test/_analyze? analyzer=kuromoji_analyzer&pretty' -d 'ؔ੢ࠃࡍۭߓ' { "tokens"

    : [ {"token" : "ؔ੢",}, {"token" : "ؔ੢ࠃࡍۭߓ",}, {"token" : "ࠃࡍ",}, {"token" : "ۭߓ",} ] } Monday, November 26, 12

26. kuromojiΛσϑΥϧτʹ • default͍ͬͯ͏໊લͰanalyzerΛએݴ Monday, November 26, 12

27. ಉٛޠ • Solrಉ༷ಉٛޠ͕ϑΝΠϧͰॻ͚Δ • +WordNetܗࣜ΋࢖͑Δ Monday, November 26, 12

28. analyzer Monday, November 26, 12

29. ೔ຊޠͷ৺഑͸͋Δఔ౓ย෇͍ͨʂ Monday, November 26, 12

30. εΩʔϚఆٛ • εΩʔϚϑϦʔʂ • JSONͷܕ͕࠾༻͞ΕΔ • ڧ੍తʹఆٛ΋Ͱ͖Δ(mapping) Monday, November 26,

    12

31. mappingྫ $ curl -XPUT 'http://localhost:9200/twitter/tweet/ _mapping' -d ' { "tweet"

    : { "properties" : { "message" : {"type" : "string", "store" : "yes"} } } } ' Monday, November 26, 12

32. Solrͱͷࠩ෼ • SolrͷDynamicFieldΑΓ΋؆୯ • type • 1ίΞ಺ʹෳ਺छྨͷdocΛೖΕΔ͜ ͱΛ૝ఆͯ͋ͬͯ͠ศར Monday, November

    26, 12

33. import(ruby) ratings = [] CSV.foreach("ratings.csv") do |row| ratings << {

    :id => row[:id].to_i, :restaurant_id => row[:restaurant_id].to_i, :body => row[:body], :type => 'rating' } end Tire.index 'livedoor_gourmet' do import ratings end Monday, November 26, 12

34. ݕࡧ curl -X GET 'http://localhost:9200/livedoor_gourmet/ restaurant/_search?pretty' -d ' { "query":{

    "query_string":{ "query":"ϥʔϝϯ" } }, "sort":[{"access_count":"desc"}], "filter":{ "term":{"closed":"0"} } } ' Monday, November 26, 12

35. Solrͱͷࠩ෼ • DSL͕݁ߏҧ͏ • filter, facet, grouping, highlight΋αϙʔτ • είΞϦϯά͸εΫϦϓτݴޠͰఆٛ

    Ͱ͖Δ Monday, November 26, 12

36. είΞϦϯά • PVॱͰฒ΂ͨΒ͏·͍ͬͨ͆͘ • ݱ࣮ͷ໰୊΋݁ߏ͜͏͍͏͜ͱଟ͍ Monday, November 26, 12

37. είΞϦϯά • ڵຯͷ͋Δํ͸ͥͻ • εΫϦϓτݴޠͰఆٛͰ͖Δ • google: elasticsearch guide scoring

    Monday, November 26, 12

38. ײ૝ • ػೳ໘Ͱෆ଍͸ͳ͍ • API͕։ൃऀʹ΍͍͞͠ • ༰қʹशಘͰ͖Δ • େن໛෼ࢄݕࡧҎ֎Ͱ΋༗༻ Monday,

    November 26, 12

39. API Monday, November 26, 12

40. config curl͚ͩͰͰ͖Δ →ΞϓϦέʔγϣϯʹఆٛΛஔ͚Δ Monday, November 26, 12

41. ίΞ௥Ճ curl͚ͩͰͰ͖Δ →։ൃऀͻͱΓͰ׬݁Ͱ͖Δ Monday, November 26, 12

42. ༰қʹशಘͰ͖Δ • ΄ͱΜͲͷૢ࡞͸curlͰ׬݁ • Solrͱڞ௨ͷ஌ࣝ΋ଟ͍ • luceneͷΫΤϦ͕࢖͑Δ • qury DSL͸ͪΐͬͱোน…

    Monday, November 26, 12

43. ෼ࢄݕࡧ Monday, November 26, 12

44. ෼ࢄݕࡧ • number_of_shards • number_of_replicas • replication • async/sync •

    write consistency(one, quorum, all) Monday, November 26, 12

45. multi-tenant • open/close index • write I/O throttling • merge

    policy control • shard allocation • number_of_replicas per index Monday, November 26, 12

46. plugin Monday, November 26, 12

47. plugin $ bin/plugin -install Aconex/elasticsearch-head Monday, November 26, 12

48. ύϑΥʔϚϯε • ࣄྫ͸ଟ͘ݟ͔ͭΔ • foursquare, soundcloud, bugsense ...etc • ΫΤϦΩϟογϡ͕ͳ͍

    • nginx, varnishͳͲͰΩϟογϡ͢Δ Monday, November 26, 12

49. ·ͱΊ • ෼ࢄݕࡧΛ࢖͏ͳΒelasticsearch • ෼ࢄݕࡧΛ࢖Θͳͯ͘΋ར఺͕ଟ͍ • ࠓޙ࢖ΘΕΔػձ͕͋Δ͔΋ Monday, November 26,

    12

50. see also... • http://www.elasticsearch.org/ • http://www.elasticsearch.org/guide/ • http://solr-vs-elasticsearch.com/ • github.com/elasticsearch

    • http://blog.sematext.com/ • #elasticsearch Monday, November 26, 12