Notebooks/08_ElasticSearch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Databases\n",
    "\n",
    "### Using elasticsearch\n",
    "\n",
    "Based on [this](https://medium.com/naukri-engineering/elasticsearch-tutorial-for-beginners-using-python-b9cb48edcedc) and [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html) posts\n",
    "\n",
    "Installing: https://www.willandskill.se/en/install-elasticsearch-6-x-on-ubuntu-18-04-lts/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!sudo pip install -U elasticsearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index\",\"resource.type\":\"index_expression\",\"resource.id\":\"_all\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index\",\"resource.type\":\"index_expression\",\"resource.id\":\"_all\"},\"status\":404}"
     ]
    }
   ],
   "source": [
    "!curl -XPUT -H \"Content-Type: application/json\" http://localhost:9200/_all/_settings -d '{\"index.blocks.read_only_allow_delete\": null}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Elasticsearch package \n",
    "import json\n",
    "from elasticsearch import Elasticsearch "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Elasticsearch?\n",
    "\n",
    "You know, for search (and analysis)\n",
    "\n",
    "Elasticsearch is the distributed search and analytics engine at the heart of the [Elastic Stack](https://www.elastic.co/pt/elk-stack). \n",
    "\n",
    "\n",
    "Elasticsearch provides near real-time search and analytics for all types of data. Whether you have structured or unstructured text, numerical data, or geospatial data, Elasticsearch can efficiently store and index it in a way that supports fast searches. You can go far beyond simple data retrieval and aggregate information to discover trends and patterns in your data. And as your data and query volume grows, the distributed nature of Elasticsearch enables your deployment to grow seamlessly right along with it.\n",
    "\n",
    "While not every problem is a search problem, Elasticsearch offers speed and flexibility to handle data in a wide variety of use cases:\n",
    "\n",
    "    Add a search box to an app or website\n",
    "    Store and analyze logs, metrics, and security event data\n",
    "    Use machine learning to automatically model the behavior of your data in real time\n",
    "    Automate business workflows using Elasticsearch as a storage engine\n",
    "    Manage, integrate, and analyze spatial information using Elasticsearch as a geographic information system (GIS)\n",
    "    Store and process genetic data using Elasticsearch as a bioinformatics research tool\n",
    "\n",
    "We’re continually amazed by the novel ways people use search. But whether your use case is similar to one of these, or you’re using Elasticsearch to tackle a new problem, the way you work with your data, documents, and indices in Elasticsearch is the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Elasticsearch([{'host': 'localhost', 'port': 9200}])>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Connect to the elastic cluster\n",
    "es = Elasticsearch([{'host':'localhost','port':9200}])\n",
    "es"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Elasticsearch is document oriented, meaning that it stores entire object or documents. It not only stores them, but also indexes the content of each document in order to make them searchable. In Elasticsearch you index, search,sort and filter documents.\n",
    "\n",
    "Elasticsearch uses JSON as the serialisation format for the documents.\n",
    "\n",
    "Now let’s start by indexing the employee documents.\n",
    "\n",
    "The act of storing data in Elasticsearch is called indexing. An Elasticsearch cluster can contain multiple indices, which in turn contain multiple types. These types hold multiple documents, and each document has multiple fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"about\": \"Love to climb\",\n",
      "  \"age\": 27,\n",
      "  \"first_name\": \"Renato\",\n",
      "  \"interests\": [\n",
      "    \"sports\",\n",
      "    \"music\",\n",
      "    \"literature\"\n",
      "  ],\n",
      "  \"last_name\": \"Souza\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "e1 = {\"first_name\":\"Renato\",\n",
    "      \"last_name\":\"Souza\",\n",
    "      \"age\": 27,\n",
    "      \"about\": \"Love to climb\",\n",
    "      \"interests\": ['sports','music','literature'],\n",
    "     }\n",
    "\n",
    "print(json.dumps(e1, indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inserting a document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Now let's store this document in Elasticsearch \n",
    "\n",
    "res = es.index(index='emap',\n",
    "               doc_type='employee',\n",
    "               id=1,\n",
    "               body=e1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"_id\": \"2\",\n",
      "  \"_index\": \"emap\",\n",
      "  \"_primary_term\": 1,\n",
      "  \"_seq_no\": 0,\n",
      "  \"_shards\": {\n",
      "    \"failed\": 0,\n",
      "    \"successful\": 1,\n",
      "    \"total\": 2\n",
      "  },\n",
      "  \"_type\": \"employee\",\n",
      "  \"_version\": 1,\n",
      "  \"result\": \"created\"\n",
      "}\n",
      "{\n",
      "  \"_id\": \"3\",\n",
      "  \"_index\": \"emap\",\n",
      "  \"_primary_term\": 1,\n",
      "  \"_seq_no\": 0,\n",
      "  \"_shards\": {\n",
      "    \"failed\": 0,\n",
      "    \"successful\": 1,\n",
      "    \"total\": 2\n",
      "  },\n",
      "  \"_type\": \"employee\",\n",
      "  \"_version\": 1,\n",
      "  \"result\": \"created\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Let's insert some more documents\n",
    "e2 = {\"first_name\" :  \"Jane\",\n",
    "      \"last_name\" :   \"Smith\",\n",
    "      \"age\" :         32,\n",
    "      \"about\" :       \"I like to collect rock albums\",\n",
    "      \"interests\":  [\"music\"]\n",
    "     }\n",
    "\n",
    "e3 = {\"first_name\" :  \"Douglas\",\n",
    "      \"last_name\" :   \"Fir\",\n",
    "      \"age\" :         35,\n",
    "      \"about\":        \"I like to build cabinets\",\n",
    "      \"interests\":  [\"forestry\"]}\n",
    "\n",
    "res = es.index(index='emap',\n",
    "               doc_type='employee',\n",
    "               id=2,\n",
    "               body=e2)\n",
    "\n",
    "print(json.dumps(res, indent=2, sort_keys=True))\n",
    "\n",
    "res = es.index(index='emap',\n",
    "               doc_type='employee',\n",
    "               id=3,\n",
    "               body=e3)\n",
    "\n",
    "print(json.dumps(res, indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"_id\": \"3\",\n",
      "  \"_index\": \"emap\",\n",
      "  \"_primary_term\": 1,\n",
      "  \"_seq_no\": 0,\n",
      "  \"_shards\": {\n",
      "    \"failed\": 0,\n",
      "    \"successful\": 1,\n",
      "    \"total\": 2\n",
      "  },\n",
      "  \"_type\": \"employee\",\n",
      "  \"_version\": 1,\n",
      "  \"result\": \"created\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res, indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieving a Document:\n",
    "\n",
    "This is easy in Elasticsearch. We simply execute an HTTP GET request and specify the address of the document — the index, type, and ID. Using those three pieces of information, we can return the original JSON document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"_id\": \"3\",\n",
      "  \"_index\": \"emap\",\n",
      "  \"_primary_term\": 1,\n",
      "  \"_seq_no\": 0,\n",
      "  \"_source\": {\n",
      "    \"about\": \"I like to build cabinets\",\n",
      "    \"age\": 35,\n",
      "    \"first_name\": \"Douglas\",\n",
      "    \"interests\": [\n",
      "      \"forestry\"\n",
      "    ],\n",
      "    \"last_name\": \"Fir\"\n",
      "  },\n",
      "  \"_type\": \"employee\",\n",
      "  \"_version\": 1,\n",
      "  \"found\": true\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "res = es.get(index='emap',\n",
    "             doc_type='employee',\n",
    "             id=3)\n",
    "\n",
    "print(json.dumps(res, indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will get the actual document in ‘_source’ field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"_id\": \"3\",\n",
      "  \"_index\": \"emap\",\n",
      "  \"_primary_term\": 1,\n",
      "  \"_seq_no\": 0,\n",
      "  \"_source\": {\n",
      "    \"about\": \"I like to build cabinets\",\n",
      "    \"age\": 35,\n",
      "    \"first_name\": \"Douglas\",\n",
      "    \"interests\": [\n",
      "      \"forestry\"\n",
      "    ],\n",
      "    \"last_name\": \"Fir\"\n",
      "  },\n",
      "  \"_type\": \"employee\",\n",
      "  \"_version\": 1,\n",
      "  \"found\": true\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res, indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting a document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deleted\n"
     ]
    }
   ],
   "source": [
    "res = es.delete(index='emap',\n",
    "                doc_type='employee',\n",
    "                id=3)\n",
    "\n",
    "print(res['result'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let’s validate the absence of the document in Elasticsearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got 0 hits:\n"
     ]
    }
   ],
   "source": [
    "res = es.search(index='emap',\n",
    "                body={'query':{'match_all':{}}})\n",
    "\n",
    "print('Got %d hits:' %res['hits']['total'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search Lite:\n",
    "\n",
    "A GET is fairly simple — you get back the document that you ask for. Let’s try something a little more advanced, like a simple search! \n",
    "\n",
    "[Ref](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "res = es.search(index='emap',\n",
    "                body={'query':{<your query comes here>}})\n",
    "\n",
    "print(json.dumps(res['hits']['hits'], indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let’s search for the user name who has nitin in his first name.\n",
    "\n",
    "### match operator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "res = es.search(index='emap',\n",
    "                body={'query':{'match':{'first_name':'nitin'}}})\n",
    "\n",
    "print(json.dumps(res['hits']['hits'], indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### bool operator:\n",
    "\n",
    "bool takes a dictionary containing at least one of must, should, and must_not, each of which takes a list of matches or other further search operators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "res = es.search(index='emap',\n",
    "                body={'query':{'bool':{'must':[{'match':{'first_name':'nitin'}}]}}}\n",
    "               )\n",
    "print(json.dumps(res['hits']['hits'], indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filter operator:\n",
    "\n",
    "Let’s make the search a little more complicated. We still want to find all employees with a first name of nitin, but we want only employees who are older than 30. Our query will change a little to accommodate a filter, which allows us to execute structured searches efficiently:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "res= es.search(index='emap',\n",
    "               body={'query':{'bool':{'must':{'match':{'first_name':'nitin'}},\n",
    "                                      'filter':{\"range\":{\"age\":{\"gt\":25}}}}}}\n",
    "              )\n",
    "\n",
    "print(json.dumps(res['hits']['hits'], indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full text search\n",
    "\n",
    "The searches so far have been simple.  \n",
    "Let’s try more advanced full text search. Before starting this next type of search let me insert one more document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"_id\": \"4\",\n",
      "  \"_index\": \"emap\",\n",
      "  \"_primary_term\": 1,\n",
      "  \"_seq_no\": 1,\n",
      "  \"_shards\": {\n",
      "    \"failed\": 0,\n",
      "    \"successful\": 1,\n",
      "    \"total\": 2\n",
      "  },\n",
      "  \"_type\": \"employee\",\n",
      "  \"_version\": 1,\n",
      "  \"result\": \"created\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "e4 = {\"first_name\":\"asd\",\n",
    "      \"last_name\":\"pafdfd\",\n",
    "      \"age\": 27,\n",
    "      \"about\": \"Love to play football\",\n",
    "      \"interests\": ['sports','music'],}\n",
    "\n",
    "res = es.index(index='emap',\n",
    "               doc_type='employee',\n",
    "               id=4,\n",
    "               body=e4)\n",
    "\n",
    "print(json.dumps(res, indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = es.search(index='emap',\n",
    "                doc_type='employee',\n",
    "                body={'query':{'match':{\"about\":\"play cricket\"}}})\n",
    "\n",
    "for hit in res['hits']['hits']:\n",
    "    print(hit['_source']['about']) \n",
    "    print(hit['_score'])\n",
    "    print('**********************')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phrase Search\n",
    "\n",
    "Finding individual words in a field is all well and good, but sometimes you want to match exact sequence of words of phrases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = es.search(index='emap',\n",
    "                doc_type='employee',\n",
    "                body={'query':{'match_phrase':{\"about\":\"play cricket\"}}})\n",
    "\n",
    "for hit in res['hits']['hits']:\n",
    "    print(hit['_source']['about']) \n",
    "    print(hit['_score'])\n",
    "    print('**********************')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregations\n",
    "\n",
    "Elasticsearch has functionality called aggregations, which allowed you to generate sophisticated analytics over your data. It is similar to Group By in SQL, but much more powerful.  \n",
    "\n",
    "[Ref1](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html)\n",
    "[Ref2](https://techoverflow.net/2019/03/17/how-to-fix-elasticsearch-fielddata-is-disabled-on-text-fields-by-default-for-keyword-field/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "res= es.search(index='emap',\n",
    "               doc_type='employee',\n",
    "               body={\"aggs\": {\"all_interests\": {\"terms\": {\"field\": \"interests.keyword\"}}}}\n",
    "              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"_shards\": {\n",
      "    \"failed\": 0,\n",
      "    \"skipped\": 0,\n",
      "    \"successful\": 5,\n",
      "    \"total\": 5\n",
      "  },\n",
      "  \"aggregations\": {\n",
      "    \"all_interests\": {\n",
      "      \"buckets\": [],\n",
      "      \"doc_count_error_upper_bound\": 0,\n",
      "      \"sum_other_doc_count\": 0\n",
      "    }\n",
      "  },\n",
      "  \"hits\": {\n",
      "    \"hits\": [],\n",
      "    \"max_score\": null,\n",
      "    \"total\": 0\n",
      "  },\n",
      "  \"timed_out\": false,\n",
      "  \"took\": 1\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(res, indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting an Index:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'acknowledged': True}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es.indices.delete(index='emap', ignore=[400, 404])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}