elasticsearch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b4f17848",
   "metadata": {},
   "source": [
    "# Elasticsearch mini tutorial\n",
    "\n",
    "👨‍🎓 Elasticsearch (ES) is one of those technologies you never hear about in DS courses, but it’s very common in industry.\n",
    "\n",
    "You'll find ES in places like GitHub, Uber, or Facebook.\n",
    "\n",
    "Not familiar with ES? Don't worry, I wrote a quick tutorial to get you started with it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "258148df",
   "metadata": {},
   "source": [
    "## 📝 What’s Elasticsearch in 30 secs or less?\n",
    "\n",
    "It's a  distributed, fast, and easy-to-scale search engine capable of handling all types of data.\n",
    "\n",
    "It’s frequently used as a search engine for apps, websites, and logs analytics.\n",
    "\n",
    "Ok, enough theory. Let's start writing some code!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4db6048b",
   "metadata": {},
   "source": [
    "## 0️⃣ Prerequisites\n",
    "\n",
    "1. Install `docker`\n",
    "\n",
    "2. Create a `virtual env` and install `pandas`, `elasticsearch`, and `notebook`\n",
    "\n",
    "```shell\n",
    "$ pip install pandas notebook elasticsearch\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6749f19c",
   "metadata": {},
   "source": [
    "## 1️⃣ Run an ES cluster\n",
    "\n",
    "The easiest way to run elasticsearch locally is by using docker.\n",
    "\n",
    "Open a terminal and run this code to start a single-node ES cluster you can use for local development:\n",
    "\n",
    "```shell\n",
    "$ docker run --rm -p 9200:9200 -p 9300:9300 -e \"xpack.security.enabled=false\" -e \"discovery.type=single-node\" docker.elastic.co/elasticsearch/elasticsearch:8.3.3\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b67935d7",
   "metadata": {},
   "source": [
    "## 2️⃣ Connect to your cluster\n",
    "\n",
    "Create a new jupyter notebook, and run the following code, to connect to your newly created ES cluster.\n",
    "\n",
    "If everything went well, you should see an output similar to mine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7cabaf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from elasticsearch import Elasticsearch\n",
    "\n",
    "es = Elasticsearch(\"http://localhost:9200\")\n",
    "es.info().body"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1afa8910",
   "metadata": {},
   "source": [
    "## 3️⃣ Read the dataset\n",
    "\n",
    "Read the dataset and extract a sample from it.\n",
    "\n",
    "You'll need some data to use in ES, so download this dataset: https://www.kaggle.com/datasets/jrobischon/wikipedia-movie-plots\n",
    "\n",
    "Read it using pandas and extract a sample from it (if you don't take a sample, the next steps may take a long time to run):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63814645",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = (\n",
    "    pd.read_csv(\"wiki_movie_plots_deduped.csv\")\n",
    "    .dropna()\n",
    "    .sample(5000, random_state=42)\n",
    "    .reset_index(drop=True)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b46d321b",
   "metadata": {},
   "source": [
    "## 4️⃣ Create an index\n",
    "\n",
    "An index is a collection of documents that ES stores and represents through a very efficient data structure called an inverted index.\n",
    "\n",
    "This process is what allows ES to perform very fast full-text searches.\n",
    "\n",
    "You can create a new index like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a55cf54",
   "metadata": {},
   "outputs": [],
   "source": [
    "mappings = {\n",
    "        \"properties\": {\n",
    "            \"title\": {\"type\": \"text\", \"analyzer\": \"english\"},\n",
    "            \"ethnicity\": {\"type\": \"text\", \"analyzer\": \"standard\"},\n",
    "            \"director\": {\"type\": \"text\", \"analyzer\": \"standard\"},\n",
    "            \"cast\": {\"type\": \"text\", \"analyzer\": \"standard\"},\n",
    "            \"genre\": {\"type\": \"text\", \"analyzer\": \"standard\"},\n",
    "            \"plot\": {\"type\": \"text\", \"analyzer\": \"english\"},\n",
    "            \"year\": {\"type\": \"integer\"},\n",
    "            \"wiki_page\": {\"type\": \"keyword\"}\n",
    "    }\n",
    "}\n",
    "\n",
    "es.indices.create(index=\"movies\", mappings=mappings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab3416f2",
   "metadata": {},
   "source": [
    "## 5️⃣ Add data to your index\n",
    "\n",
    "You can use `.index()` or `.bulk()` to add data to an index.\n",
    "\n",
    "`.index()` adds one item at a time, while `.bulk()` lets you add multiple items at the same time.\n",
    "\n",
    "You can use any of the two methods to add data to your index:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8e8fc47",
   "metadata": {},
   "source": [
    "### Using `.index()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c1cab64",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, row in df.iterrows():\n",
    "    doc = {\n",
    "        \"title\": row[\"Title\"],\n",
    "        \"ethnicity\": row[\"Origin/Ethnicity\"],\n",
    "        \"director\": row[\"Director\"],\n",
    "        \"cast\": row[\"Cast\"],\n",
    "        \"genre\": row[\"Genre\"],\n",
    "        \"plot\": row[\"Plot\"],\n",
    "        \"year\": row[\"Release Year\"],\n",
    "        \"wiki_page\": row[\"Wiki Page\"]\n",
    "    }\n",
    "            \n",
    "    es.index(index=\"movies\", id=i, document=doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61d64b15",
   "metadata": {},
   "source": [
    "### Using `.bulk()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "018ae3db",
   "metadata": {},
   "outputs": [],
   "source": [
    "from elasticsearch.helpers import bulk\n",
    "\n",
    "bulk_data = []\n",
    "for i,row in df.iterrows():\n",
    "    bulk_data.append(\n",
    "        {\n",
    "            \"_index\": \"movies\",\n",
    "            \"_id\": i,\n",
    "            \"_source\": {        \n",
    "                \"title\": row[\"Title\"],\n",
    "                \"ethnicity\": row[\"Origin/Ethnicity\"],\n",
    "                \"director\": row[\"Director\"],\n",
    "                \"cast\": row[\"Cast\"],\n",
    "                \"genre\": row[\"Genre\"],\n",
    "                \"plot\": row[\"Plot\"],\n",
    "                \"year\": row[\"Release Year\"],\n",
    "                \"wiki_page\": row[\"Wiki Page\"],\n",
    "            }\n",
    "        }\n",
    "    )\n",
    "bulk(es, bulk_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e338a2c",
   "metadata": {},
   "source": [
    "### Check the number of documents indexed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "398b55c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "es.indices.refresh(index=\"movies\")\n",
    "es.cat.count(index=\"movies\", format=\"json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54947e1a",
   "metadata": {},
   "source": [
    "## 6️⃣ Make searches in your ES index\n",
    "\n",
    "Finally, you'll want to start running searches using your index.\n",
    "\n",
    "ES has a powerful DSL that lets you build many types of queries.\n",
    "\n",
    "Here's an example of a search that looks for movies starring Jack Nicholson, but whose director isn't Roman Polanski:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c843a253",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = es.search(\n",
    "    index=\"movies\",\n",
    "    query={\n",
    "        \"bool\": {\n",
    "            \"must\": {\n",
    "                \"match\": {\n",
    "                    \"cast\": \"jack nicholson\",\n",
    "                }\n",
    "            },\n",
    "            \"filter\": {\"bool\": {\"must_not\": {\"match_phrase\": {\"director\": \"roman polanksi\"}}}},\n",
    "        },\n",
    "    },\n",
    ")\n",
    "resp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3140222c",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8bdb907",
   "metadata": {},
   "source": [
    "## 7️⃣ Delete documents from the index\n",
    "\n",
    "This will delete the document 9140 from the index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "846541e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "es.delete(index = \"movies\", id = \"2500\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f0c47ad",
   "metadata": {},
   "source": [
    " ## 8️⃣ Delete an index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64063ed6",
   "metadata": {},
   "source": [
    "If for whatever reason you'd like to delete an index (and all of its documents), here's how you do it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77d561d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "es.options(ignore_status=[400,404]).indices.delete(index='movies')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}