2-Get_better-at-dask-dataframes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "036478b0-c89d-4051-a513-f4436f48968f",
   "metadata": {},
   "source": [
    "<img src=\"http://dask.readthedocs.io/en/latest/_images/dask_horizontal.svg\"\n",
    "     align=\"right\"\n",
    "     width=\"30%\"\n",
    "     alt=\"Dask logo\\\">\n",
    "\n",
    "# Get better at Dask Dataframes\n",
    "\n",
    "In this lesson, you will learn the advantages of working with the parquet data format and best practices when working with big data. You will learn how to manipulate inconvenient file sizes and datatypes, as well as how to make your data easier to manipulate. You will be exploring the Uber/Lyft dataset and learning some key practices of feature engineering with Dask Dataframes.\n",
    "\n",
    "## Dask Dataframes \n",
    "\n",
    "<img src=\"https://docs.dask.org/en/stable/_images/dask-dataframe.svg\"\n",
    "     align=\"right\"\n",
    "     width=\"30%\"\n",
    "     alt=\"Dask DataFrame is composed of pandas DataFrames\"/>\n",
    "\n",
    "At its core, the `dask.dataframe` module implements a \"blocked parallel\" `DataFrame` object that looks and feels like the `pandas` API, but for parallel and distributed workflows. One Dask `DataFrame` is comprised of many in-memory pandas `DataFrame`s separated along the index. One operation on a Dask `DataFrame` triggers many pandas operations on the constituent pandas `DataFrame`s in a way that is mindful of potential parallelism and memory constraints.\n",
    "\n",
    "Dask dataframes are very useful, but how to get the most out of them can be tricky.  Where your data is stored, the format your data was saved in, the size of each file and the data types, are some examples of things you need to care when it comes to working with dataframes. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4307fa56-f051-467b-bf1f-e3acae08e3a8",
   "metadata": {},
   "source": [
    "### Work close to your data\n",
    "\n",
    "To get started when you are working with data that is in the cloud it's always better to work close to your data to minimize the impact of IO networking. \n",
    "\n",
    "In this lesson, we will use Coiled Clusters that will be created on the same region that our datasets are stored. (the region is `\"us-east-2\"`)\n",
    "\n",
    "**NOTE:**\n",
    "If you do not have access to a Coiled Cluster you, can follow along just make sure you use the smaller dataset (use the `\"0.5GB-\"` ones). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a52c65b-f0bc-4341-9841-4de07f710dce",
   "metadata": {},
   "source": [
    "## Parquet vs CSV\n",
    "\n",
    "Most people are familiarized with **csv** files, but when it comes to working with data, working with **parquet** can make a big difference. \n",
    "\n",
    "### Parquet is where is at!!\n",
    "\n",
    "The Parquet file format is column-oriented and it is designed to efficiently store and retrieve data. Columnar formats provide better compression and improved performance, and enable you to query data column by column. Consequently, aggregation queries are faster compared to row-oriented storage.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/coiled/dask-tutorial/main/images/storage-files.png\"\n",
    "     align=\"right\"\n",
    "     width=\"50%\"\n",
    "     alt=\"Dask DataFrame is composed of pandas DataFrames\"/>\n",
    "     \n",
    "     \n",
    "- **Column pruning:** Parquet lets you read specific columns from a dataset without reading the entire file.\n",
    "- **Better compression:**  Because in each column the data types are fairly similar, the compression of each column is quite straightforward. (saves on storage)\n",
    "- **Schema:** Parquet stores the file schema in the file metadata.\n",
    "- **Column metadata:** Parquet stores metadata statistics for each column, which can make certain types of queries a lot more efficient.\n",
    "\n",
    "    \n",
    "### Small motivation example: \n",
    "\n",
    "Let's see an example where we compare reading the same data but in one case it is stored as `csv` files, while the other as `parquet` files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cb22fca-88e1-4019-9b26-d1f398b176bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#data dictionary\n",
    "data ={\"0.5GB-csv\": \"s3://coiled-datasets/h2o-benchmark/N_1e7_K_1e2/*.csv\",\n",
    "       \"0.5GB-pq\": \"s3://coiled-datasets/h2o-benchmark/N_1e7_K_1e2_parquet/*.parquet\",\n",
    "       \"5GB-csv\": \"s3://coiled-datasets/h2o-benchmark/N_1e8_K_1e2/*.csv\",\n",
    "       \"5GB-pq\": \"s3://coiled-datasets/h2o-benchmark/N_1e8_K_1e2_parquet/*.parquet\",}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e205e7c7-1d13-4f16-9035-ade902f43fad",
   "metadata": {},
   "outputs": [],
   "source": [
    "### coiled login\n",
    "#!coiled login --token ### --account dask-tutorials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bea1905f-09aa-4865-844e-d752639e29c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import coiled\n",
    "import dask\n",
    "from dask.distributed import Client\n",
    "import dask.dataframe as dd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1162a7b-969a-46c7-a6b2-bfef92cade59",
   "metadata": {},
   "outputs": [],
   "source": [
    "#we use this to avoid re-using clusters on a team\n",
    "import uuid \n",
    "id_cluster = uuid.uuid4().hex[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0ac86a4-5336-4c28-98e9-64323e450863",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "cluster = coiled.Cluster(name=f\"csv-vs-parquet-{id_cluster}\",\n",
    "                        n_workers=10,\n",
    "                        package_sync=True,\n",
    "                        backend_options={\"region_name\": \"us-east-2\"},\n",
    "                        );\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "254390a1-e055-446c-ab44-c8d76121928d",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Client(cluster)\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6829081-0fe8-432b-a126-254ba8139f82",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_csv = dd.read_csv(data[\"5GB-csv\"], storage_options={\"anon\": True})\n",
    "ddf_pq = dd.read_parquet(data[\"5GB-pq\"], storage_options={\"anon\": True})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78de69de-fe56-46e4-b8b7-63936c7de7f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1a9742b-a915-4e7f-8d81-59a127bf9e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_pq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f938ab3c-30da-416c-87a4-58128c7973b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "ddf_csv.groupby(\"id1\").agg({\"v1\": \"sum\"}).compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84a6f2fb-b1ba-4b8d-ade5-be1c2c6f4a93",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "ddf_pq.groupby(\"id1\").agg({\"v1\": \"sum\"}).compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0530b707-cad2-4b43-8921-1eb821616822",
   "metadata": {},
   "source": [
    "### Memory usage \n",
    "\n",
    "Notice that the `parquet` version without doing much it is already ~4X faster. Let's take a look at the memory usage as well as the `dtypes` in both cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4be1cc4f-4656-4565-bd83-dfec45e613fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "## memory usage for 1 partition\n",
    "ddf_csv.partitions[0].memory_usage(deep=True).compute().apply(dask.utils.format_bytes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "011266f8-b9d7-46c4-b88d-574582f5528b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_pq.partitions[0].memory_usage(deep=True).compute().apply(dask.utils.format_bytes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bf78bee-7a52-4bdb-8aaa-badf392e4d02",
   "metadata": {},
   "source": [
    "**Discuss: What do you notice?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5aa3b67-328d-409a-8b9a-8930ca0da53a",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.shutdown()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e323f48-9bbb-46bd-af08-7d6ded1ccead",
   "metadata": {},
   "source": [
    "## Uber/Lyft data transformation\n",
    "\n",
    "In the example above we saw that the format in which the data is stored, already makes a big difference. \n",
    "\n",
    "**Working with parquet** \n",
    "\n",
    "Let's use the Uber/Lyft dataset , as an example of a `parquet` dataset to learn how to troubleshoot the nuances of working with real data. The data comes from [High-Volume For-Hire Services](https://www.nyc.gov/site/tlc/businesses/high-volume-for-hire-services.page)\n",
    "\n",
    "_Data dictionary:_\n",
    "\n",
    "https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_hvfhs.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fda99d96-4b61-4bd8-882b-d7b457279276",
   "metadata": {},
   "outputs": [],
   "source": [
    "#inspect data\n",
    "import s3fs\n",
    "\n",
    "s3 = s3fs.S3FileSystem()\n",
    "files = s3.glob(\"nyc-tlc/trip data/fhvhv_tripdata_*.parquet\")\n",
    "files[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3af054b4-894c-4ff0-b5f2-e1dd9d117066",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(files)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "676338c2-52d0-4106-b81c-a8a7b3f23ce1",
   "metadata": {},
   "source": [
    "### Let's get a cluster \n",
    "\n",
    "From experience we know that we will need a cluster where the workers have plenty of memory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d64ee00a-f2e8-4aee-b587-0b807801d78e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "cluster = coiled.Cluster(\n",
    "    n_workers=10,\n",
    "    name=f\"nyc-uber-lyft-{id_cluster}\",\n",
    "    package_sync=True,\n",
    "    backend_options={\"region\": \"us-east-2\"}, \n",
    "    worker_memory=\"64 GiB\", #we know we need a lot of memory from experience\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "183c584b-bf15-4808-9dcb-a76a105e9c8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Client(cluster)\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b48f56f-2937-4b6d-8477-7c07721981c9",
   "metadata": {},
   "source": [
    "**Inspect the data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "882652d4-cc6b-454a-be90-d6b8111186ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf = dd.read_parquet(\n",
    "    \"s3://nyc-tlc/trip data/fhvhv_tripdata_*.parquet\",\n",
    ")\n",
    "ddf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3727d3f4-3544-4889-8499-d4ea867bbf85",
   "metadata": {},
   "outputs": [],
   "source": [
    "#inspect dtypes\n",
    "ddf.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70ea4e9a-d812-4d50-9f73-c34054b34007",
   "metadata": {},
   "outputs": [],
   "source": [
    "#inspect memory usage of 1 partition\n",
    "ddf.partitions[0].memory_usage(deep=True).compute().apply(dask.utils.format_bytes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0547a9bf-436a-4a49-8cdd-7d79f4c281f0",
   "metadata": {},
   "source": [
    "### Challenges:\n",
    "\n",
    "- Big partitions\n",
    "- Inefficient data types\n",
    "\n",
    "### Recommendations and best practices:\n",
    "\n",
    "**Partition size**\n",
    "\n",
    "In general we recommend starting with partitions that are in the order of ~100MB (in memory). However, the choice of the partition size can vary depending on the worker memory that you have available. \n",
    "\n",
    "For documentation on partition sizes visit the [repartition docs](https://docs.dask.org/en/stable/generated/dask.dataframe.DataFrame.repartition.html) as well as the repartition section in the Dask Dataframe [best practices](https://docs.dask.org/en/stable/dataframe-best-practices.html#repartition-to-reduce-overhead)\n",
    "\n",
    "**Data Types**\n",
    "\n",
    "- Avoid object types for strings: use `\"string[pyarrow]\"`\n",
    "- Reduce int/float representation if possible\n",
    "- Use categorical dtypes when possible (avoid high cardinality).\n",
    "- Consider using Nullable dtypes (very new/experimental)\n",
    "\n",
    "### Create conversions dictionary\n",
    "\n",
    "Based on these recommendations, let's work on better `dtypes`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac85d0af-b961-4fff-be71-b14b60cd52bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9bccadb-6f7a-4dba-a4fd-113bd96fbbc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "conversions = {}\n",
    "for column, dtype in ddf.dtypes.items():\n",
    "    if dtype == \"object\":\n",
    "        conversions[column] = \"string[pyarrow]\"\n",
    "    if dtype == \"float64\":\n",
    "        conversions[column] = \"float32\"\n",
    "    if dtype == \"int64\": \n",
    "        conversions[column] = \"int32\"\n",
    "    if \"flag\" in column:\n",
    "        conversions[column] = pd.CategoricalDtype(categories=[\"Y\", \"N\"])\n",
    "    if column == \"airport_fee\":\n",
    "        conversions[column] = \"float32\"  #noticed that this has floats and the <NA> is making it an object\n",
    "conversions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53645d0d-8cd9-4fe8-892a-b4c11f2b787d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#use new dtypes this takes a bit of time\n",
    "ddf = ddf.astype(conversions)\n",
    "ddf = ddf.persist() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eed8a8ed-abb8-4fa4-8686-3a7a68bd28df",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.partitions[0].memory_usage(deep=True).compute().apply(dask.utils.format_bytes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5970a21-6c8c-40dd-869e-07d998e7c2e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "dask.utils.format_bytes(\n",
    "    ddf.partitions[0].memory_usage(deep=True).compute().sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec42ac57-72fd-4ff3-8416-dcdb287e25b7",
   "metadata": {},
   "source": [
    "### Repartition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4815f73c-0ae4-4b8b-84f2-1750cbaa969a",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf = ddf.repartition(partition_size=\"128MB\").persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6c13761-4906-47a6-b138-1ec5875e1242",
   "metadata": {},
   "outputs": [],
   "source": [
    "dask.utils.format_bytes(\n",
    "    ddf.memory_usage(deep=True).compute().sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "329c807b-1dfe-4173-9d34-73bc8fc38695",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.npartitions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b208ec5-4c1e-45e7-921e-04a90b60f37e",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf = ddf.set_index(\"request_datetime\").persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b056e6e8-f393-4bae-906a-bdc411c9afb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.divisions[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d214655b-2850-49bd-8e6b-70f51ef761b7",
   "metadata": {},
   "source": [
    "### Sort and one-day partitioning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf67360c-83b8-4dbb-ac2c-340a51e329e6",
   "metadata": {},
   "source": [
    "Look like they are a bit longer than a day, we might as well repartition them witha  1-day frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58c66047-19cc-4348-a58f-2a7e1b10330a",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf = ddf.repartition(freq=\"1d\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37c174e7-9022-4afc-8fd4-571d17245c6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.divisions[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2ca0e10-fc36-476f-8721-54efb6c51370",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.npartitions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9603233c-7e0f-41bf-aa70-84c4d55a7f12",
   "metadata": {},
   "source": [
    "### Save data to and S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3de701dd-2f55-418f-908a-3ae8839f2352",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Clever name for files when to_parquet\n",
    "divisions = ddf.divisions\n",
    "\n",
    "def name_file(index: int) -> str:\n",
    "    return str(divisions[index].date()) + \".parquet\"\n",
    "\n",
    "name_file(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b0d899c-2155-4872-ade7-d80a19f5190e",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.to_parquet(\n",
    "    \"s3://coiled-datasets/uber-lyft-tlc/\", \n",
    "    name_function=name_file,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "287e76d2-04ce-4f5c-a620-39017f8b05a0",
   "metadata": {},
   "source": [
    "### Read data back\n",
    "\n",
    "After you save your data, you will want to read it back to do some data analysis or train a model. When reading data back, there are some caveats regarding the `dtypes`.\n",
    "\n",
    "- **Roundtriping for string pyarrow dtype** is not yet supported in pandas/dask. Hence when you read your data you need to tell pandas/dask to cast those columns as \"string[pyarrow]\" otherwise they'll be \"string[python]\". \n",
    "- **Nullable dtypes:** Using nullable dtypes is a fairly new feature and still under development, consider this experimental. Available in `dask >= 2022.12.0`\n",
    "\n",
    "**What are nullable dtypes?**\n",
    "\n",
    "Pandas (hence Dask) primarily uses NaN to represent missing data. Because NaN is a float, this forces an array of integers with any missing values to become floating point. In some cases, this may not matter much. But if your integer column is, say, an identifier, casting to float can be problematic. \n",
    "\n",
    "Nullable dtypes, allow you to work around this issue. \n",
    "\n",
    "If you want to read more about nullable dtypes, check the pandas [missing data docs](\n",
    "https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#missing-data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71a340ae-4f3b-4714-bce4-2534a65c3b9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf = dd.read_parquet(\n",
    "    \"s3://coiled-datasets/uber-lyft-tlc/\", \n",
    ").astype({\"hvfhs_license_num\": \"string[pyarrow]\", \n",
    "         \"dispatching_base_num\": \"string[pyarrow]\",\n",
    "         \"originating_base_num\": \"string[pyarrow]\",\n",
    "         }).persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9a28eda-ee5f-452f-9941-6bbbc8f9aa99",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0001de44-e6fd-4f45-aa3b-3d51531e48d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.hvfhs_license_num.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1264d50b-8d5f-461c-9927-de62ef16978f",
   "metadata": {},
   "outputs": [],
   "source": [
    "dask.utils.format_bytes(\n",
    "    ddf.memory_usage(deep=True).sum().compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c4291e8-0dce-467b-95ce-badd2513b10c",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.shutdown()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beeff184-6c3d-44bb-86f3-f071c1ab5781",
   "metadata": {},
   "source": [
    "## Let's do some data analysis\n",
    "\n",
    "Now we are at a stage that our whole dataset is ~80GB in memory. When it comes to exploring data we do not necessarily need the whole data set, we can work with a sample, as well as only select a subset of columns. One of the beauties of the parquet file format is **column pruning**\n",
    "\n",
    "Note: Keep in mind, that if you will do feature engineering, your data size will increase and having extra memory can help."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3476887b-5643-4e4a-9cdb-d2488692cdf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "cluster = coiled.Cluster(name=f\"uber-lyft-{id_cluster}\", \n",
    "                         n_workers=20, \n",
    "                         package_sync=True,\n",
    "                         backend_options={\"region_name\": \"us-east-2\"},\n",
    "                         worker_memory=\"32GB\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "304ba45b-5604-4bdd-8a33-7433cd9df26d",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Client(cluster)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5a715c5-b809-4305-93d1-466a1008326c",
   "metadata": {},
   "outputs": [],
   "source": [
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15530dc1-703e-4491-9eec-c391cd663a34",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = dd.read_parquet(\n",
    "    \"s3://coiled-datasets/uber-lyft-tlc/\", \n",
    ").astype({\"hvfhs_license_num\": \"string[pyarrow]\", \n",
    "         \"dispatching_base_num\": \"string[pyarrow]\",\n",
    "         \"originating_base_num\": \"string[pyarrow]\",\n",
    "         })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e2900ba-cbdf-46fc-9ee4-9fbc69ea881c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80eb7dc4-e84e-40b9-9842-dad6670dad3b",
   "metadata": {},
   "source": [
    "## Memory usage \n",
    "\n",
    "```python\n",
    "dask.utils.format_bytes(\n",
    "    df.memory_usage(deep=True).sum().compute()\n",
    ")\n",
    "```\n",
    "'74.82 GiB'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "834816b0-5899-43a8-853e-ffc91a0a7da8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dff7b58b-97c5-4e06-9fdf-f2fe4fc979a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d52ca6d7-4030-4103-b128-4263f95044c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "#len(df)\n",
    "## 707_786_079"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1262fb9-adaa-4e23-9b5f-7e50e62128ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "##let's count to see NaN\n",
    "df.count().compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb55c7c8-bb9b-4fe5-9729-6b91f263a3ac",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Create a column tip > 0 = True \n",
    "df[\"tip_flag\"] = df.tips > 0\n",
    "\n",
    "df_small = df[[\"hvfhs_license_num\", \"tips\", \"base_passenger_fare\", \n",
    "               \"driver_pay\", \"trip_miles\", \"trip_time\", \n",
    "               \"shared_request_flag\",\"tip_flag\"]].persist()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54707176-6ddf-4c3c-9870-7cdfd03f8335",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb809b7b-9cdf-4d54-b191-305a3c89eb97",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.base_passenger_fare.sum().compute() / 1e9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "286fc4f5-003b-47a9-8a56-fc227141e670",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.driver_pay.sum().compute() / 1e9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ca23fa1-ab0c-4426-8165-7742bcae7891",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.tips.sum().compute() / 1e6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fcaad19-5ad8-4ce8-a5e6-669a802c37e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de556d9d-3b0c-4fea-bd5b-cac5b5fa16b7",
   "metadata": {},
   "source": [
    "### Average `trip_time` by provider, shared vs non-shared trips\n",
    "\n",
    "[dask pivot table docs](https://docs.dask.org/en/stable/generated/dask.dataframe.DataFrame.pivot_table.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d0b5bbc-a9fb-4075-a0e6-856911bd0c5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "##try a pivot table\n",
    "dd.pivot_table(df_small, index=\"hvfhs_license_num\", \n",
    "               columns=\"shared_request_flag\", #this one needs to be a category\n",
    "               values=\"trip_time\", aggfunc='mean').compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2df5069-719f-4ff7-a8e5-55443a667e62",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2282206b-f800-4bcf-92f1-19676cb65bd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.shared_request_flag.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9f72e0b-5ef0-42bb-8891-b83b295f1cdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.shared_request_flag = df_small.shared_request_flag.cat.as_known()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62cea8e7-4cd3-4c9a-8533-b82e88e95414",
   "metadata": {},
   "outputs": [],
   "source": [
    "trip_time_avg = dd.pivot_table(df_small, index=\"hvfhs_license_num\", \n",
    "                               columns=\"shared_request_flag\",\n",
    "                               values=\"trip_time\", aggfunc='mean').compute()\n",
    "trip_time_avg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9530ff16-da3c-4ede-8a2a-08ac6204236d",
   "metadata": {},
   "source": [
    "**Note**  \n",
    "We have a NaN, this is probably because that provider does not have the feature to share the trip. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b38289b5-549b-468f-b47e-35e6d16c4807",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df_small\n",
    " .loc[df_small.shared_request_flag==\"Y\"]\n",
    " .hvfhs_license_num.unique().compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc2110ff-d2fe-4a57-8556-f8381cd12d48",
   "metadata": {},
   "source": [
    "### Are New Yorker's tippers? \n",
    "\n",
    "Let's see how many trips have tip by provider using a pivot table. \n",
    "\n",
    "https://docs.dask.org/en/stable/generated/dask.dataframe.DataFrame.pivot_table.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac3192f3-7913-41e6-9b7b-ed01bb8a96c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "tip_counts = df_small.groupby(['hvfhs_license_num']).tip_flag.value_counts().compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13062d36-4168-41b7-b2c5-d6c8e62de44e",
   "metadata": {},
   "outputs": [],
   "source": [
    "tip_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fac3870d-0b82-47e9-856b-e15e77cf3425",
   "metadata": {
    "tags": []
   },
   "source": [
    "**From the data dictionary we know:**\n",
    "\n",
    "As of September 2019, the HVFHS licensees are the following:\n",
    "\n",
    "- HV0002: Juno  \n",
    "- HV0003: Uber  \n",
    "- HV0004: Via  \n",
    "- HV0005: Lyft  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28c54f0e-0526-447e-8f5c-19fc6e6b1962",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(tip_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd1d5b68-5e2e-487d-9db7-b1bc0c5b8e2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "## this is a pandas \n",
    "tip_counts = tip_counts.unstack(level=\"tip_flag\")\n",
    "tip_counts/1e6"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e01315d8-0a69-47d1-a443-64bfa5d84aa7",
   "metadata": {},
   "source": [
    "### Percentage of total rides that tip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "554de773-2424-44c5-b711-d1d4e4a6bad6",
   "metadata": {},
   "outputs": [],
   "source": [
    "tip_counts[True] * 100 / (tip_counts[True]+tip_counts[False])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c42ee9b-b7db-4fde-8125-84c5aca7483c",
   "metadata": {},
   "source": [
    "### sum and mean of tips by provider "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b10f437-ba9d-41d3-bc43-0d4e96e9dcdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_total = (df_small.loc[lambda x: x.tip_flag]\n",
    "              .groupby('hvfhs_license_num')\n",
    "              .tips.agg([\"sum\", \"mean\"])\n",
    "              .compute()\n",
    "             )\n",
    "tips_total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef8c80f8-0d20-4bf4-a4ee-fa4a5325ce26",
   "metadata": {},
   "outputs": [],
   "source": [
    "provider = {\"HV0002\":\"Juno\", \n",
    "            \"HV0005\":\"Lyft\" ,\n",
    "            \"HV0003\":\"Uber\" ,\n",
    "           \"HV0004\":\"Via\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "776e2689-133d-4cfb-bd08-0ca34ee05f31",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_total = (tips_total\n",
    "              .assign(provider=lambda df: df.index.map(provider))\n",
    "              .set_index(\"provider\")\n",
    "             )\n",
    "tips_total"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e63f9aad-c1e0-49c6-9de2-8d99e4bd44e2",
   "metadata": {},
   "source": [
    "### What percentage of the passenger fare is the tip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cec9faaf-f329-4ebb-8e60-d1fe95d22f2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "tip_percentage = df_small.tips / df_small.base_passenger_fare \n",
    "df_small[\"tip_percentage\"] = tip_percentage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b14e09d-7a4b-454f-ab39-12cbcd5d7ef4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small = df_small.persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ab6fa5d-1b7b-4cd6-aa02-f5986784cd49",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_perc_mean = (df_small\n",
    "                  .loc[lambda x: x.tip_flag]\n",
    "                  .groupby('hvfhs_license_num')\n",
    "                  .tip_percentage.mean()\n",
    "                  .compute()\n",
    "                 )\n",
    "tips_perc_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c16d4260-46a4-4a5a-842d-b80f3e0cc76b",
   "metadata": {},
   "outputs": [],
   "source": [
    "(tips_perc_mean.to_frame()\n",
    " .set_index(tips_perc_mean.index.map(provider))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42ba5d51-9537-47ad-86db-3aa7a98a885a",
   "metadata": {},
   "source": [
    "### Base pay per mile per - by provider\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70eaed48-4b26-48ce-8036-fc26f4e13bad",
   "metadata": {},
   "outputs": [],
   "source": [
    "dolars_per_mile = df_small.base_passenger_fare / df_small.trip_miles\n",
    "df_small[\"dolars_per_mile\"] = dolars_per_mile\n",
    "df_small = df_small.persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb6f2544-c41f-45dc-8f8a-f3863f369044",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df_small.groupby('hvfhs_license_num')\n",
    " .dolars_per_mile.agg([\"min\", \"max\", \"mean\", \"std\"])\n",
    " .compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "910754f2-478c-4eeb-99a5-8ba4b2cf5d75",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df_small.loc[lambda x: x.tip_flag]\n",
    " .groupby('hvfhs_license_num')\n",
    " .dolars_per_mile.agg([\"min\", \"max\", \"mean\", \"std\"])\n",
    " .compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5508b41a-e186-4af6-8a35-2a659c3a6975",
   "metadata": {},
   "source": [
    "### Get insight on the data\n",
    "\n",
    "We are seeing weird numbers, let's try to take a deeper look and remove some outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1831c64-86dc-46cb-bbdd-5ffa415da91d",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df_small[[\"trip_miles\", \"base_passenger_fare\", \"tips\"]]\n",
    " .describe()\n",
    " .compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78f9d49e-89a9-43d6-af77-89920defcddf",
   "metadata": {},
   "source": [
    "### Exercise:\n",
    "Get stats on the columns we just analyzed but only for the trips that tipped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15df0aab-84fb-45bc-af6d-d2f9892b38be",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Solution\n",
    "(df_small[[\"trip_miles\", \"base_passenger_fare\", \"tips\", \"tip_flag\"]]\n",
    " .loc[lambda x: x.tip_flag]\n",
    " .describe().compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e970925-c96d-40e5-a906-176a6f69a44c",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "- How would you get more insights on the data?\n",
    "- Can you visualize it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21a73002-ab6b-4627-b5af-7ecdeedd0d78",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#needed to avoid plots from breaking\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1abd1c5-8d60-4772-b1a9-4a180790c0d8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Solution, take a sample \n",
    "#let's get 0.1% of the data to plot ~700_000 rows (go smaller if needed depending on your machine)\n",
    "df_tiny = (df_small.loc[lambda x: x.tip_flag]\n",
    "           [[\"trip_miles\", \"base_passenger_fare\", \"tips\"]]\n",
    "           .sample(frac=0.001).compute()\n",
    "          )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a46b9993-3009-4e5d-a0e4-2c8368ec0613",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Solution: box plot\n",
    "df_tiny.boxplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "500530b7-0f24-4566-b234-1f1a5acc2cc3",
   "metadata": {},
   "source": [
    "### Exercise:\n",
    "\n",
    "- Play with the pandas dataframe `df_tiny`to get insights on good filters for the bigger dataframe. \n",
    "\n",
    "Hint: think about quantiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d674a25-104a-4506-8602-25e214023954",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#solution\n",
    "df_tiny.tips.quantile([0.25, 0.75])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "905ed15c-106a-42eb-bfee-59f6d12b84d6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#solution\n",
    "df_tiny.base_passenger_fare.quantile([0.25, 0.75])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f08228ab-5f07-460a-b268-d176e77aef94",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#solution\n",
    "df_tiny.trip_miles.quantile([0.25, 0.75])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "469ccd8f-b119-40ae-9f89-7a087eb64dec",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#solution\n",
    "tips_condition = (df_tiny.tips.between(*df_tiny.tips.quantile([0.25, 0.75]).values))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1133eddf-b478-4f61-9889-3f41e9050689",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#solution\n",
    "tips_condition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89d5bdc4-1685-471d-898e-9ed239ad281b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Solution\n",
    "tips_condition = (df_tiny.tips.between(*df_tiny.tips.quantile([0.25, 0.75]).values))\n",
    "\n",
    "fares_condition = (df_tiny.base_passenger_fare.between(*df_tiny.base_passenger_fare.quantile([0.25, 0.75]).values))\n",
    "\n",
    "trips_condition = (df_tiny.trip_miles.between(*df_tiny.trip_miles.quantile([0.25, 0.75]).values))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a82012b-ee3a-4df9-b24b-877f114a207a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#solution\n",
    "df_tiny.loc[(tips_condition & fares_condition) & trips_condition].boxplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1337177-5c12-4f73-bc68-f9156c1bbc6c",
   "metadata": {},
   "source": [
    "## Filtering our big dataset based on the insights\n",
    "\n",
    "Based on these numbers let's go back to our `df_small` dataset and try to filter it.\n",
    "\n",
    "**Note:**\n",
    "\n",
    "Sometimes when you are trying to filter and you have been doing feature engineering, you might get a divisions not known error.\n",
    "If that's the case you can do \n",
    "\n",
    "```python\n",
    "df_small = df_small.reset_index()\n",
    "df_small = (df_small\n",
    "            .set_index(\"column_to_be_the_index\")\n",
    "            .persist()\n",
    "           )\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6db09194-2001-4887-9623-39e31d022330",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_q1q3 = (2, 5.66)\n",
    "trips_q1q3 = (1.71 , 7.09)\n",
    "fares_q1q3 = (10.74, 29.80)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75a5a9be-5e13-4f0d-8c76-13ab7caea857",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_condition = df_small.tips.between(*tips_q1q3)\n",
    "trips_condition = df_small.trip_miles.between(*trips_q1q3)\n",
    "fares_condition = df_small.base_passenger_fare.between(*fares_q1q3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "879314c1-b9de-483f-af5d-59347db07f61",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small = (df_small\n",
    "            .loc[(tips_condition & fares_condition) & trips_condition]\n",
    "            .persist()\n",
    "           )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39de8d6c-b94f-4b26-9f5e-5758f37efe1c",
   "metadata": {},
   "source": [
    "### Stats on `dollar_per_mile`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e33c8db5-e8f1-4559-bc02-deb716e19fcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df_small.groupby('hvfhs_license_num')\n",
    " .dolars_per_mile.agg([\"min\", \"max\", \"mean\", \"std\"])\n",
    " .compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4933836-e833-4787-b561-f2d0bb5f76ee",
   "metadata": {},
   "source": [
    "### Let's look at the `tip_percentage` again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e745175-5725-4b92-a58d-4f870a3d3ace",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_perc_avg = (df_small\n",
    "                 .groupby('hvfhs_license_num')\n",
    "                 .tip_percentage.mean()\n",
    "                 .compute()\n",
    "                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46be3c7a-8128-4ab1-b8f1-3b398faa4238",
   "metadata": {},
   "outputs": [],
   "source": [
    "(tips_perc_avg.to_frame()\n",
    " .set_index(tips_perc_mean.index.map(provider))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1538a243-8de1-4dde-82f0-dd5434b21f11",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3665fc2-8b8e-4e0e-a656-e47cefbafb86",
   "metadata": {},
   "source": [
    "### Exercise:\n",
    "\n",
    "Compute the basic statistics about trip time per provider. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "176f6aa4-8eec-4d4b-9959-678fa61d50ea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#solution\n",
    "trips_time_avg = (df_small.groupby('hvfhs_license_num')\n",
    "                  .trip_time.agg([\"min\", \"max\", \"mean\", \"std\"])\n",
    "                  .compute()\n",
    "                 )\n",
    "trips_time_avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3715c5a-7520-42c3-a41c-c1295a19d5ff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#solution in minutes\n",
    "trips_time_avg.set_index(tips_perc_mean.index.map(provider)) / 60\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf7fdf77-320b-4d87-802c-1f260867a141",
   "metadata": {},
   "source": [
    "## What we've learn\n",
    "- Most New Yorkers do not tip\n",
    "- But it looks like of those who tip, it is common to tip around 20% regardless of the provider. Unless it's Via, they tend to tip slightly less.\n",
    "- The trip_time column needs some cleaning of outliers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "451a0ec9-fc40-4f23-bd8a-69803f0a6fe2",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.shutdown()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}