1-Parallelize-your-python-code_Futures_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3786371b-cdd0-473b-89bd-94573b3676b5",
   "metadata": {},
   "source": [
    "<img src=\"http://dask.readthedocs.io/en/latest/_images/dask_horizontal.svg\"\n",
    "     align=\"right\"\n",
    "     width=\"30%\"\n",
    "     alt=\"Dask logo\\\">\n",
    "     \n",
    "# Parallelize your python code\n",
    "\n",
    "In this lesson you will learn how to parallelize custom python code using Dask using the Futures API.\n",
    "\n",
    "## Futures: a low-level collection.\n",
    "\n",
    "Dask low-level collections are the best tools when you need to have fine control to build custom parallel and distributed computations. \n",
    "\n",
    "The `futures` interface (derived from the built-in `concurrent.futures`) provides fine-grained real-time execution for custom situations. It allows you to submit arbitrary functions for computation in a parallelized, eager, and non-blocking way. \n",
    "\n",
    "### Why use Futures?\n",
    "\n",
    "The `futures` API offers a work submission style that can easily emulate the map/reduce paradigm. If that is familiar to you then futures might be the simplest entrypoint into Dask.\n",
    "\n",
    "The other big benefit of futures is that the intermediate results, represented by futures, can be passed to new tasks without having to pull data locally from the cluster. The **call returns immediately**, giving one or more *futures*, whose status begins as \"pending\" and later becomes \"finished\". There is no blocking of the local Python session. With futures, as soon as the inputs are available and there is compute available, the computation starts. \n",
    "\n",
    "### When do we us Futures?\n",
    "\n",
    "One of the most common cases where you can use `Futures` is when you have a for loop. For example, you need to apply a **read-transform-write** function over multiple files. Your serial code will look something like:\n",
    "\n",
    "\n",
    "```python\n",
    "# Serial code\n",
    "def process_file(filename):\n",
    "    data = read_a_file(filename)\n",
    "    data_transformed = do_a_transformation(data)\n",
    "    destination = f\"results/{filename}\"\n",
    "    write_out_data(data_transformed, destination)\n",
    "    return destination\n",
    "\n",
    "files = [\"file_1\", \"file_2\", \"file_3\", ..., \"file_n\"] #list of files\n",
    "new_files = [] #where we save the destination file names\n",
    "\n",
    "for f in files:\n",
    "    new_files.append(process_file(f)            \n",
    "```\n",
    "\n",
    "Notice that every call of `process_file` is independent from each other, this is what is call an embarrassingly parallel problem. You can do this in parallel with Dask by doing\n",
    "\n",
    "```python\n",
    "#Parallel code\n",
    "futures = []\n",
    "for f in files:\n",
    "    future = client.submit(process_file, f)\n",
    "    futures.append(future)\n",
    "    \n",
    "futures\n",
    "```\n",
    "\n",
    "## Example: Get SO get questions page title \n",
    "\n",
    "During this lesson, you will be working with the Stack Overflow question pages. To start let's see how to grab the title of each page, and how for multiple pages we can perform this in parallel. \n",
    "\n",
    "If you go to https://stackoverflow.com/questions/ you will see a list of the newest posts, if you go to the bottom of the page you can switch to the next page. For example, the top of page number two at the moment of the creation of this notebook, looked like \n",
    "\n",
    "<center>\n",
    "<img src=\"https://raw.githubusercontent.com/coiled/dask-tutorial/main/images/SO_page.png\"\n",
    "     width=\"65%\"\n",
    "     alt=\"SO page\\\">\n",
    "</center>\n",
    "\n",
    "### Get the title\n",
    "\n",
    "The title of the page is what is showed in the tab. The following function gets the title of a page given its page number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6bebd77-c616-4899-9f38-d1a5d33b554d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup as bs\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "485fe9fe-da20-4e17-98be-3572f32077d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_questions_page_title(page_num):\n",
    "    \"\"\" Get title of a SO questions page\n",
    "    \"\"\"\n",
    "    url = f\"https://stackoverflow.com/questions?tab=newest&page={page_num}\"\n",
    "    req = requests.get(url)\n",
    "    html = bs(req.text, \"html.parser\")\n",
    "\n",
    "    return html.title.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "033f1c7e-0769-439b-a640-b3bfeb3b3969",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "page_2 = get_questions_page_title(2)\n",
    "page_2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "486c8e08-7044-4a8e-a1b4-0b4ea93b7d30",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Serial code to get 8 pages would be:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57f2a7db-3b6b-4a85-bf09-f02c91fa5e05",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "page_title = []\n",
    "for p in range(1,9): #page numbers start in 1\n",
    "    page_title.append(get_questions_page_title(p))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c903a112-7802-417b-9565-32457fa663c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "page_title[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a06d3c08-b560-4abe-9d7b-40cdd41bba2a",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Run the code in parallel, using futures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27785fcb-93fe-4197-8133-6744cc02e710",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask.distributed import Client, wait"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9467a843-1364-4458-b4b4-8d68060c256a",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Client(n_workers=4)\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "810b134d-c7af-4d34-a1e8-48aef0f16312",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Solution\n",
    "futures = []\n",
    "for p in range(1,9):\n",
    "    future = client.submit(get_questions_page_title, p)\n",
    "    futures.append(future)\n",
    "    \n",
    "futures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6514a138-a39f-49b2-ae7c-e7b786ea4511",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "futures[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31f5aad8-1d2c-4a14-9c19-8c2a6e179b3d",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "futures[0].result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84ddfbd4-484d-403f-a219-c848b9982d56",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "results = [future.result() for future in futures]\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d809a1f0-b209-4ad8-b62a-01104eee28a0",
   "metadata": {},
   "source": [
    "**Extra:**\n",
    "\n",
    "To be able to `%%time` the cell and compare times with the serial version, you will need to wait for the futures to finish doing `wait(futures)`. If you try to do that and re run the cell, you will notice it is immediate, this is because by default, distributed assumes that all functions are pure. Pure functions:\n",
    "\n",
    "- always return the same output for a given set of inputs\n",
    "- do not have side effects, like modifying global state or creating files\n",
    "\n",
    " \n",
    "You can use the `pure=False` keyword argument in the `client.submit()`. Modify your solution to match this code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe36ccb1-0248-456e-9370-6f9e0bee2ef6",
   "metadata": {
    "tags": []
   },
   "source": [
    "```python\n",
    "%%time\n",
    "futures = []\n",
    "for p in range(1,9):\n",
    "    future = client.submit(get_questions_page_title, p, pure=False)\n",
    "    futures.append(future)\n",
    "    \n",
    "wait(futures)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4019bbbe-de28-49e9-bf4b-ca18aa4afba2",
   "metadata": {},
   "source": [
    "**`client.map()`**\n",
    "\n",
    "With `client.submit()` you can submit individual functions for evaluation with one set of inputs, and together with a `for-loop` you can also evaluate over a sequence of inputs. `client.map()` provides a simpler interface to perform the former, let's see how to perform the example above now using `client.map()`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8f3eb84-ab8a-491d-a961-fa2bab0bd8c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "futures = client.map(get_questions_page_title, range(1, 9))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a758e9f-a391-4eba-875a-8fd56ab03b03",
   "metadata": {},
   "source": [
    "`client.map()` returns a list of futures, you can block on the computation and gather the result by doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "668dfa0e-5dd3-4304-b6d7-e54eadfa5550",
   "metadata": {},
   "outputs": [],
   "source": [
    "res = client.gather(futures)\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2829f110-5843-4174-8d2a-b59d0de6b308",
   "metadata": {},
   "source": [
    "### Futures are great...\n",
    "\n",
    "The other big benefit of `futures` is that the intermediate results, represented by `futures`, can be passed to new tasks without having to pull data locally from the cluster. New operations can be setup to work on the output of previous jobs that have not even begun yet.\n",
    "\n",
    "Let's brake our steps into multiple functions\n",
    "\n",
    "- `request_html_page`: given a url returns the html of that page\n",
    "- `get_page_html_links`: given a SO questions page number returns a the html for that page number.\n",
    "- `get_post_links_per_page`: given a SO questions html page, returns a list with the posts of that page.\n",
    "\n",
    "\n",
    "<center>\n",
    "<img src=\"https://raw.githubusercontent.com/coiled/dask-tutorial/main/images/dask_SO_posts_links.png\"\n",
    "     width=\"65%\"\n",
    "     alt=\"dask post links\\\">\n",
    "</center>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "524bb7f1-49bd-4045-b3d8-f0a79228064b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def request_html_page(url):\n",
    "    \"\"\"Given a url returns the html of that page\n",
    "    \"\"\"\n",
    "    req = requests.get(url)\n",
    "    html = bs(req.text, \"html.parser\")\n",
    "    return html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25a592e2-b30e-4c5e-8811-854426e34d1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_page_html_links(page_num, tag=\"dask\", query_filter = \"MostVotes\"):\n",
    "    \"\"\"Given a SO questions page number returns a the html for that page number\n",
    "    for a tag and query_filter.\n",
    "    \"\"\"\n",
    "    base_url = \"https://stackoverflow.com/questions/tagged/\"\n",
    "    \n",
    "    page_url = f\"{base_url}{tag}?sort={query_filter}&page={page_num}\"\n",
    "\n",
    "    page_html = request_html_page(page_url)\n",
    "    \n",
    "    return page_html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7230061-0551-44fd-9014-d75ffc362d3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_post_links_per_page(html_page):\n",
    "    \"\"\"Given a SO questions html page, returns a list with the posts of that page.\n",
    "    \"\"\"\n",
    "    question_href = html_page.find_all(\"a\", class_=\"s-link\")[2:-1]\n",
    "    \n",
    "    question_link = [f\"https://stackoverflow.com{q['href']}\" for q in question_href]\n",
    "    \n",
    "    return question_link"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78e6bf98-faef-4343-afb3-2e800c3e416e",
   "metadata": {},
   "source": [
    "### Explore the functions:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "614af67b-1868-4c3b-a212-559ab8136e7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "page_number = 3\n",
    "\n",
    "page_3_html = get_page_html_links(page_num=page_number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3318156f-a299-43e1-a08f-d585503938d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "post_links_page_3 = get_post_links_per_page(page_3_html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ce191a0-7746-4fd7-93c5-620e69453c56",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(post_links_page_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cc7fe9d-5129-431d-92e5-939b8e777252",
   "metadata": {},
   "outputs": [],
   "source": [
    "post_links_page_3[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "359aeb5d-8c2e-4334-88c8-90d3d7fdcdd2",
   "metadata": {},
   "source": [
    "### Get post links for multiple pages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "532930c8-a46a-4788-990d-0079b149a70b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# serial code\n",
    "page_posts_links = []\n",
    "for page in range(1, 5):\n",
    "    page_html = get_page_html_links(page_num=page)\n",
    "    posts_links = get_post_links_per_page(page_html)\n",
    "    \n",
    "    page_posts_links.append(posts_links)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7377b99-415b-400d-bc41-38e8142da3c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(page_posts_links)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ed0ada1-8875-417f-a8b6-a887706b8eb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "[len(l) for l in page_posts_links]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d189e69c-648f-4e43-88df-08b3222b943c",
   "metadata": {},
   "source": [
    "**Parallel code: using `client.map()`**\n",
    "\n",
    "We can get first the futures for every page html, and pass those futures as the iterator to get the links per page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7417901f-f351-48ff-9083-685404f28742",
   "metadata": {},
   "outputs": [],
   "source": [
    "pages_html_futures = client.map(get_page_html_links, range(1,5))\n",
    "wait(pages_html_futures) #wait until completed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8443c00-6592-4dbd-b660-80d0453872a8",
   "metadata": {},
   "source": [
    "**`wait()`**\n",
    "\n",
    "Notice that here we used `wait()`, you can wait on a future or collection of futures using the `wait` function, which blocks until all futures are finished or have erred. This is useful when you need the all the futures to be completed to proceed with your computations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f9741ae-24e6-406c-ac69-e52ff5937d8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "pages_html_futures[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b14229c3-e063-4ebd-996f-ca3b9f449c5f",
   "metadata": {},
   "source": [
    "### Exercise:\n",
    "\n",
    "Using `client.map()` and the `pages_html_futures` you just got, to get the post's links for the four pages, in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0c18aaf-dce4-4a18-965f-3fc37e4532b0",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Solution\n",
    "posts_links_futures = client.map(get_post_links_per_page, pages_html_futures)\n",
    "posts_links_futures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6060850e-0647-431d-ab91-a7bf5015a4f8",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "posts_links_futures[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a456089-a046-42f0-8dc5-ce76ea6d6496",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "posts_links_futures[0].result()[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7242493f-9026-40e8-80aa-48e16c8cbff7",
   "metadata": {},
   "source": [
    "**`as_completed()`**\n",
    "\n",
    "In the example above we waited for the the `pages_html_futures` to finish before we proceed to get the `posts_links_futures`. However, we can get the `post_links_futures` for every page as the `pages_html_futures` finish. \n",
    "\n",
    "`as_completed()` return futures in the order in which they complete. It returns an iterator that yields the input future objects in the order in which they complete. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf892eab-26f8-4fc5-9367-7711d52468f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask.distributed import as_completed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11a3bb79-0116-40a4-8c14-9bbdbfb96652",
   "metadata": {},
   "outputs": [],
   "source": [
    "pages_html_futures = client.map(get_page_html_links, range(1,5), pure=False) #use pure=False to re-compute\n",
    "\n",
    "post_links_futures = []\n",
    "for p in as_completed(pages_html_futures):\n",
    "    post_links_futures.append(client.submit(get_post_links_per_page, p))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38a1dd26-699b-4800-b8c7-1229d3dcae00",
   "metadata": {},
   "outputs": [],
   "source": [
    "post_links_futures"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16467916-008c-4a21-bb1e-d374c530ee57",
   "metadata": {},
   "source": [
    "## Grown-up example: Scrape, crawl and get SO data\n",
    "\n",
    "Let's use all what we've learn in the examples above, to do something a bit more advanced. In this section, we graduate to a grownup example. You will learn how to parallelize a scrapping, crawling and get data workflow.\n",
    "\n",
    "Up to now, you learned how to scrape multiple pages from https://stackoverflow.com/questions/, and to get a list of the post links for every page. Let's go a step further and get some data of each post. For example we can \n",
    "\n",
    "- Title\n",
    "- Question body\n",
    "- Most voted answer\n",
    "- Number of votes for the best answer\n",
    "- Who answer the most voted answer\n",
    "\n",
    "<center>\n",
    "<img src=\"https://raw.githubusercontent.com/coiled/dask-tutorial/main/images/data_from_post.png\"\n",
    "     width=\"65%\"\n",
    "     alt=\"data to extract\">\n",
    "</center>\n",
    "\n",
    "\n",
    "#### Data insights\n",
    "\n",
    "For every page we will end up with one dictionary per post that contains the information above, we can convert them into a dataframe and for example find useful aggregated information like:\n",
    "\n",
    "- Which username gets the most \"best answers\"?\n",
    "- Which of the best answer usernames is the most voted?\n",
    "\n",
    "\n",
    "<center>\n",
    "<img src=\"https://raw.githubusercontent.com/coiled/dask-tutorial/main/images/sketch_bar_plots.png\"\n",
    "     width=\"85%\"\n",
    "     alt=\"bar plots sketch\">\n",
    "</center>\n",
    "\n",
    "**Note about throttling:**\n",
    "\n",
    "When scrapping directly from the pages and not using the API, it is not clear what are the throttling limitations, but from experience we run into them pretty quickly.\n",
    "\n",
    "The following examples, work as they are, if you change the number of pages you will likely hit a limit and be banned for few minutes. We will work around this towards the end, in the meantime avoid changing the number of pages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d22c3321-d821-4c89-b90d-a1620312db0b",
   "metadata": {},
   "source": [
    "## Scrape, crawl, get data, and plot\n",
    "\n",
    "Below you have our set of functions that we use above, plus function that will allow us to scrape the data needed to get some insights.\n",
    "\n",
    "You will see this functions in action in serial and together we will use all what we learned about futures to run things in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c4faad5-4bf2-4534-b3e1-26e6b2eb0432",
   "metadata": {},
   "outputs": [],
   "source": [
    "def request_html_page(url):\n",
    "    \"\"\"Given a url returns the html of that page\n",
    "    \"\"\"\n",
    "    req = requests.get(url)\n",
    "    html = bs(req.text, \"html.parser\")\n",
    "    return html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "308a11f4-69cc-412c-bcf9-e9994435a70c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_page_html_links(page_num, tag=\"dask\", query_filter = \"MostVotes\"):\n",
    "    \"\"\"Given a SO questions page number returns a the html for that page number\n",
    "    for a tag and query_filter.\n",
    "    \"\"\"\n",
    "    base_url = \"https://stackoverflow.com/questions/tagged/\"\n",
    "    \n",
    "    page_url = f\"{base_url}{tag}?sort={query_filter}&page={page_num}\"\n",
    "\n",
    "    page_html = request_html_page(page_url)\n",
    "    \n",
    "    return page_html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccdb5443-9067-40c2-a56c-a5cc204acade",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_post_links_per_page(html_page):\n",
    "    \"\"\"Given a SO questions html page, returns a list with the posts of that page.\n",
    "    \"\"\"\n",
    "    question_href = html_page.find_all(\"a\", class_=\"s-link\")[2:-1]\n",
    "    \n",
    "    question_link = [f\"https://stackoverflow.com{q['href']}\" for q in question_href]\n",
    "    \n",
    "    return question_link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bd88acf-eb3b-4d7d-8735-8449fc53b788",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(post_link):\n",
    "    \"\"\"Get data from a SO post as a dictionary\n",
    "    \n",
    "    - Title\n",
    "    - Question body\n",
    "    - Number of votes for the best answer\n",
    "    - Who answer the most voted answer\n",
    "    \n",
    "    \"\"\"\n",
    "    html_post = request_html_page(post_link)\n",
    "    post_info = {}\n",
    "    \n",
    "    \n",
    "    post_info[\"title\"] = html_post.title.text\n",
    "    post_info[\"question\"] = html_post.find(\"div\", class_=\"s-prose js-post-body\").text\n",
    "    \n",
    "    answ = html_post.find(\"div\", class_=\"answer\") #this will gets us the first/most voted answer\n",
    "    \n",
    "    if answ:\n",
    "        post_info[\"best_answer_votes\"] = int(answ[\"data-score\"])\n",
    "    \n",
    "        best_answer_author_obj = answ.find(\"span\", itemprop=\"name\")\n",
    "        \n",
    "        if best_answer_author_obj:\n",
    "            best_answer_author = best_answer_author_obj.text\n",
    "        else:\n",
    "            best_answer_author = \"comunity_post\"\n",
    "\n",
    "        post_info[\"best_answer_usrname\"] = best_answer_author\n",
    "    else:\n",
    "        post_info[\"best_answer_votes\"] = 0\n",
    "        post_info[\"best_answer_usrname\"] = \"no-answer\"\n",
    "\n",
    "    \n",
    "    return post_info"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ea45c41-734c-4dc0-bc64-cd3b401eb9a6",
   "metadata": {},
   "source": [
    "## Serial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2402f42a-e528-45dd-bce8-0b3bc64022f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "df_list =[]\n",
    "for page_num in range(1, 3): #more than 2 pages and get trhottling issues\n",
    "    page_html = get_page_html_links(page_num)\n",
    "    posts_links = get_post_links_per_page(page_html)\n",
    "    list_post_data = []\n",
    "    \n",
    "    for link in posts_links:\n",
    "        p_data = get_data(link)\n",
    "        list_post_data.append(p_data)\n",
    "\n",
    "    df = pd.DataFrame(list_post_data)\n",
    "    df_list.append(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a8a0cf1-4b0c-48d9-a037-d43306ff510c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_list[0].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9dc8afc-5eb4-4717-8f02-7dd15d5fadae",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_answers = df_list[0][\"best_answer_usrname\"].value_counts()\n",
    "most_answers[:3].plot.bar(title=\"Most Best Answers\",\n",
    "                       ylabel=\"Votes\",\n",
    "                       xlabel=\"Usernames\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a04e6d05-f001-4b6c-b4db-1257970867d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_voted = df_list[0].groupby(\"best_answer_usrname\").best_answer_votes.sum().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aab4547d-3abb-4559-abe9-dfbd3ec8fcd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_voted[:3].plot.bar(title=\"Most Voted Usernames\",\n",
    "                       ylabel=\"Votes\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2958d205-f854-44e2-ab8f-437032fc905a",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_answers_series = []\n",
    "most_voted_series = []\n",
    "for df in df_list:\n",
    "    most_answ = df[\"best_answer_usrname\"].value_counts()\n",
    "    most_vot = df.groupby(\"best_answer_usrname\").best_answer_votes.sum()\n",
    "    \n",
    "    most_answers_series.append(most_answ)\n",
    "    most_voted_series.append(most_vot)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b090fabf-66bd-4883-88e5-6a37c6aac57d",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_answers_tot = pd.concat(most_answers_series, axis=1).sum(axis=1).sort_values(ascending=False)\n",
    "most_voted_tot = pd.concat(most_voted_series, axis=1).sum(axis=1).sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a165ce45-54bb-46e0-9aa7-1683348c5fdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_answers_tot[:5].plot.bar(title=\"Most Best Answers\",\n",
    "                       ylabel=\"Votes\",\n",
    "                       xlabel=\"Usernames\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1e58f58-5368-4232-8428-91fd828cef86",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_voted_tot[:5].plot.bar(title=\"Most Voted Usernames\",\n",
    "                       ylabel=\"Votes\");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19cc84dc-bd05-475a-be57-398e0fb391af",
   "metadata": {},
   "source": [
    "## Parallel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7870844e-5711-45fc-8b8c-81a397be48e5",
   "metadata": {},
   "source": [
    "### Get pages and links of posts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7792eff1-7013-4db8-bf0c-a5e13fbf8828",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "pages_futures = client.map(get_page_html_links, range(1,3))\n",
    "wait(pages_futures)\n",
    "\n",
    "posts_links_futures = client.map(get_post_links_per_page, pages_futures)\n",
    "crawling = as_completed(posts_links_futures)\n",
    "\n",
    "dfs_data = []\n",
    "for future in crawling:\n",
    "    list_links = future.result() # list of links per page\n",
    "    df_data = []\n",
    "    for link in list_links:\n",
    "        fut_data = client.submit(get_data, link) \n",
    "        df_data.append(fut_data)\n",
    "\n",
    "    dfs_data.append(df_data)\n",
    "_ = wait(dfs_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0592207-b6ec-4dc3-880e-cf0dd34da5ac",
   "metadata": {},
   "source": [
    "### Parallel is fast\n",
    "\n",
    "Well that was ~9x on a laptop with 4 cpu cores. But we have only processed 2 pages, let's tackle more pages!\n",
    "\n",
    "**Note:** If you are running this from Binder, the limitation on Binder resources will end up on lower speedups."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27c2fce7-3a4d-4d8c-9d39-1bb76bb581fc",
   "metadata": {},
   "source": [
    "## Avoid throttling using a cluster. \n",
    "\n",
    "In the example above, if we try to work with more pages, you will hit throttling issues. This is a problem, but we can solve it by scaling out to a lot of machines. When using a cluster each worker has its own public ip-address so it is like we are requesting from different machines. \n",
    "\n",
    "Let's create a coiled cluster and scrape a bigger number of pages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a66cab38-f727-454d-b901-33f70b86a08b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Shutdown LocalCluster\n",
    "client.shutdown()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab088ce0-b8f7-47f0-9a64-153c90dbebb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "### coiled login \n",
    "#!coiled login --token ### --account dask-tutorials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b81208e-1075-412a-ac48-e2193c984acb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import coiled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8bbdbe4-f486-49dc-9d0b-0e6230c5ad2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#we use this to avoid re-using clusters on a team\n",
    "import uuid \n",
    "id_cluster = uuid.uuid4().hex[:4]\n",
    "id_cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d91a1f5-b9e1-4553-b9c4-3da14127c59f",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = coiled.Cluster(name=f\"my-cluster-{id_cluster}\",\n",
    "                        n_workers=10, \n",
    "                        package_sync=True,\n",
    "                        scheduler_port=443) #port needed for binder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4334ae4e-43a2-450e-a6fc-7f1b9c302f3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#When running from binder dask-lab extension won't work, use link to dashboard\n",
    "client =  Client(cluster)\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "401e1817-fe7a-4d3c-927e-44db2f5be74e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "pages_10_futures = client.map(get_page_html_links, range(1,11))\n",
    "wait(pages_10_futures)\n",
    "\n",
    "posts_links_futures = client.map(get_post_links_per_page, pages_10_futures)\n",
    "crawling = as_completed(posts_links_futures)\n",
    "\n",
    "dfs_data = []\n",
    "for future in crawling:\n",
    "    list_links = future.result() # list of links per page\n",
    "    df_data = []\n",
    "    for link in list_links:\n",
    "        fut_data = client.submit(get_data, link) \n",
    "        df_data.append(fut_data)\n",
    "\n",
    "    dfs_data.append(df_data)\n",
    "_ = wait(dfs_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01a64300-e971-4235-b9a9-b16c778c1069",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(dfs_data) #10 pages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fda0881-f0a1-47d0-b019-8f7c4270b5f7",
   "metadata": {},
   "source": [
    "At this point, we have the data to build each page dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67513dcf-0ee9-4981-a948-8f3697d0db0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs_data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fa7fdfb-4a50-43ef-b662-76ccc892c141",
   "metadata": {},
   "source": [
    "### Let's get some dataframes:\n",
    "In the serial code you got some insights on the data by manipulation the pandas dataframes. At the moment our futures do not have dataframes, but we can convert them by mapping pandas.Dataframe into our `dfs_data` futures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d13dedd5-18c7-41b4-b0a7-755502554b52",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_futures = client.map(pd.DataFrame, dfs_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df97d442-f647-4a86-82f9-78bf85efd327",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_futures[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0f1de10-3712-4a87-902e-3e26ecc8e276",
   "metadata": {},
   "source": [
    "Now we have some pandas dataframes!!\n",
    "\n",
    "### Exercise\n",
    "\n",
    "Write a function that returns the `value_counts` of the `best_answer_usrname`, and apply it to the dataframe futures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b5e1d0f-f71b-4aaf-8fa5-2c7505afde70",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "###Solution\n",
    "def best_ans_val_counts(df):\n",
    "    return df.best_answer_usrname.value_counts()\n",
    "\n",
    "best_ans_val_counts_futures = client.map(best_ans_val_counts, df_futures)\n",
    "best_ans_val_counts_futures[0].result()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f51a90b3-036b-4f44-a03d-982e97646af8",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "Write a function that calculates the total amount of votes by best answer, apply it to the dataframe futures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa871753-e011-4987-89f2-e4b8fedfa58a",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#solution\n",
    "def most_votes(df):\n",
    "    return df.groupby(\"best_answer_usrname\")['best_answer_votes'].sum()\n",
    "\n",
    "most_votes_futures = client.map(most_votes, df_futures, pure=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "690fe19b-0484-4984-8646-f6d98bfdebb8",
   "metadata": {},
   "source": [
    "## Results for 10 pages aggregation\n",
    "\n",
    "Now we have 10 futures that each of them is a `pd.Series`. We can bring this to the client, concatenate them and re-do our plots.\n",
    "\n",
    "### Exercise\n",
    "\n",
    "Gather the results into a a list of `pd.Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bde999e-bc8d-477b-9be2-04c4f778e058",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Solution\n",
    "best_ans_count_res = client.gather(best_ans_val_counts_futures)\n",
    "most_votes_res = client.gather(most_votes_futures)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cadf3e1d-a4da-46fa-a914-82a5e1d56dcc",
   "metadata": {},
   "source": [
    "### Let's Plot!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44e7962c-019b-471c-a3ca-fba7fbe11f7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_answers_tot = pd.concat(best_ans_count_res, axis=1).sum(axis=1).sort_values(ascending=False)\n",
    "most_voted_tot = pd.concat(most_votes_res, axis=1).sum(axis=1).sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5d50ca2-ad8e-4b74-aa46-cd7d8c7ec0c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_answers_tot[:5].plot.bar(title=\"Most Best Answers\",\n",
    "                       ylabel=\"Votes\",\n",
    "                       xlabel=\"Usernames\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e5b6813-e234-4f07-83a6-f083132098d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "most_voted_tot[:5].plot.bar(title=\"Most Voted Usernames\",\n",
    "                       ylabel=\"Votes\");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3b05936-5863-4583-b0f0-44fe6fa44c00",
   "metadata": {},
   "source": [
    "### Dask dataframes API\n",
    "\n",
    "Now we are on dataframe world, we can do pandas-like operations, for example."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a749893-30d1-4427-a293-ed91ef14ab5a",
   "metadata": {},
   "source": [
    "We can do multiple operations on these dataframes using `futures` but at this point since we are working with dataframes we can use `dask.dataframes`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4429b8f4-79f5-4af4-a3cb-30d1730d82d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.dataframe as dd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b63bef3e-9532-4fbe-b58e-2699633074bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_so = dd.from_delayed(df_futures)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e17b6c1d-84e8-4160-a0b3-c0a38469fefe",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_so"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4f19f68-1dee-4e4a-a748-df96260843aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_so.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "158c1bce-23af-4310-ba78-cadfede71968",
   "metadata": {},
   "source": [
    "We can check which of the user tht got a best answer, has the most \"best answers\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8c70813-5717-49c7-86ba-f705e1f32520",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_so.best_answer_usrname.value_counts().compute()[:6]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14ffeb25-b404-419f-a1f9-5a1d70b0616e",
   "metadata": {},
   "source": [
    "We can also check how many votes, these users got:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90e8e18a-3004-4c5f-8e26-6a92fc65768f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_so.groupby(\"best_answer_usrname\")['best_answer_votes'].sum().compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57f28d2f-2b44-428c-afca-402b7988a6ea",
   "metadata": {},
   "source": [
    "## Extra:\n",
    "\n",
    "Repeat the analysis we did for the `tag=\"dask\"` for a different one, like `tag=\"python\"`.\n",
    "\n",
    "You will need to modify this portion of the code to\n",
    "\n",
    "```python\n",
    "pages_futures = client.map(get_page_html_links, range(1,3))\n",
    "wait(pages_futures)\n",
    "```\n",
    "to use `client.map()` with `lambda` functions like:\n",
    "\n",
    "```python\n",
    "# Solution\n",
    "pages_py_futures = client.map(lambda p: get_page_html_links(p, tag=\"python\"), range(1, 3))\n",
    "wait(pages_py_futures)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62a7a6ef-b94b-46f8-af23-1a7ef27725f4",
   "metadata": {},
   "source": [
    "### Useful links\n",
    "\n",
    "- https://tutorial.dask.org/05_futures.html\n",
    "\n",
    "**Useful links**\n",
    "\n",
    "* [Futures documentation](https://docs.dask.org/en/latest/futures.html)\n",
    "* [Futures screencast](https://www.youtube.com/watch?v=07EiCpdhtDE)\n",
    "* [Futures examples](https://examples.dask.org/futures.html)\n",
    "\n",
    "### Next lesson\n",
    "\n",
    "Register [here](https://www.coiled.io/tutorials) for reminders. \n",
    "\n",
    "In the next lesson, we’ll learn some best practices around working with larger-than-memory datasets. We’ll use the Uber/Lyft dataset to:\n",
    "\n",
    "- Manipulate Parquet files and optimize queries\n",
    "- Navigate inconvenient file sizes and data types\n",
    "- Extract useful features with Dask Dataframe\n",
    "\n",
    "By the end, we’ll learn the advantages of working with the Parquet file format and how to efficiently perform an exploratory analysis with Dask."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}