Notebooks/1_Preprocessing/6_Strings_And_Regex.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mo-BdJBktJvG"
   },
   "source": [
    "# 1. String Manipulation Methods (READ-AND-PLAY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uqhBU97FMOTM"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "import pandas as pd\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mtx0_hpyPgul"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "string = '    pen,pineapple,apple, pen   '\n",
    "print(string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HHiSy3yauCvS"
   },
   "source": [
    "`split()` method \n",
    "\n",
    "- This method splits a string into a list where each word is a list item.\n",
    "- We need to specify the `separator` to use when splitting the string.\n",
    "- We can specifiy how many splits to do by setting the `maxsplit` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-6mbfhS1tZ4A"
   },
   "outputs": [],
   "source": [
    "# Split the string using separator ','\n",
    "string.split(',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nKWOm3mcu2c2"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "string_2 = 'summer#autumn   #spring  # winter'\n",
    "print(string_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pzwduWpWvHAv"
   },
   "outputs": [],
   "source": [
    "# Split the string using separator '#'\n",
    "string_2.split('#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "or4k7UHDwYcO"
   },
   "outputs": [],
   "source": [
    "# Split string_2 and set the maxsplit parameter to 2 (this should return a list with 3 elements)\n",
    "x = string_2.split('#', 2)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HNFNcPHZuYJk"
   },
   "source": [
    "`strip()` method\n",
    "\n",
    "- It removes whitespaces at the beginning and at the end of the string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GNSmSGqdtfUR"
   },
   "outputs": [],
   "source": [
    "# Remove whitespaces in the variable our_string\n",
    "our_string = '     There is a lot of space at the beginning and at the end of this sentence, let`s remove it.       '\n",
    "our_result = our_string.strip()\n",
    "print(our_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SfeLbKcyz1Ec"
   },
   "source": [
    "`join()` method\n",
    "\n",
    "- This method takes all items in an iterable and joins them into one string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Wfhe7fv-t0E5"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "my_list = ['Please', 'join', 'these', 'items.']\n",
    "'_'.join(my_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dI1D72nU0ifj"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "my_tuple = ('We','are', 'joining', 'again.')\n",
    "'-'.join(my_tuple)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_Dkd1sRW12Hv"
   },
   "source": [
    "In the case of a dictionary, `join()` tries to join keys of the dictionary, not values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "j_EG_Pvo0_xq"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "my_dictionary = {'Key_1':'1',\n",
    "                 'Key_2':'2'}\n",
    "'#'.join(my_dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Te0DU0ifZuW8"
   },
   "source": [
    "`index()` method\n",
    "\n",
    "- It returns the position of the first character in a substring if the substring is found in the string.\n",
    "- It raises a `ValueError` if nothing is found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RoWymu99Y2Hk"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "string_3 = 'That is my string'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CPbHqh3jZnwT"
   },
   "outputs": [],
   "source": [
    "# Find the position of 'm' using `index()`\n",
    "string_3.index('m')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xsbDTSoTan6w"
   },
   "source": [
    "`replace()` method\n",
    "\n",
    "- This method replaces occurences of a substring with another string.\n",
    "- It is commonly used to remove characters by passing an empty string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GFfXgwHzatme"
   },
   "outputs": [],
   "source": [
    "# Replacing string in string_3\n",
    "string_3.replace('is','was')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-W9fG7q6BbZL"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "string_4 = 'Why is here a semicolon; ?'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9VPIr-tKBnzh"
   },
   "outputs": [],
   "source": [
    "# Replacing character\n",
    "string_4.replace(';','')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-q2T36c3NwHC"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "string_5 = 'Banana, avocado, pineapple, artichoke'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4X0eFJP4cySC"
   },
   "outputs": [],
   "source": [
    "# TASK 1 >>>> Use .replace() method to replace 'a' with 'A' in string_5 and store it in variable result_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uq0ODoCtM2IP"
   },
   "source": [
    "`upper()` method\n",
    "\n",
    "- This method converts all lowercase characters in a string into uppercase characters and returns it.\n",
    "\n",
    "`lower()` method\n",
    "- This method converts all upercase characters in a string into lowercase characters and returns it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NXHlYKltMHQI"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "string_to_upper = \"Make this uppercase\"\n",
    "print(string_to_upper.upper())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "czYHrFYlMolH"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "string_to_lower = 'THIS SHOULD BE ALL LOWERCASE'\n",
    "print(string_to_lower.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``find()`` method\n",
    "- This method is similar to `index()`.\n",
    "- If the substring is found, this method returns the index of first occurrence of the substring.\n",
    "- If the substring is not found, -1 is returned.\n",
    "- This function is **case sensitive**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this code\n",
    "quote = \"Data Science is cool\"\n",
    "\n",
    "print(\"The quote is: \" + quote)\n",
    "\n",
    "# first occurance of 'Data Science'\n",
    "result = quote.find('Data Science')\n",
    "print(\"Substring 'Data Science':\", result)\n",
    "\n",
    "# what happens when we neglect the case sensitivity \n",
    "result = quote.find('data science')\n",
    "print(\"Substring 'data science':\", result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find returns -1 if substring not found\n",
    "result = quote.find('RBI')\n",
    "print(\"Substring 'RBI':\", result)\n",
    "\n",
    "# How to use find()\n",
    "if (quote.find('is') != -1):\n",
    "    print(\"Substring is found\")\n",
    "else:\n",
    "    print(\"Substring is not found\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you go to the [Documentation of `find()`](https://python-reference.readthedocs.io/en/latest/docs/str/find.html), you will see that ``find()`` can accept three parameters. One is compulsory, and the others are optional. \n",
    "\n",
    "The general syntax looks like this:\n",
    "````\n",
    "string.find(value, start, end)\n",
    "````\n",
    "\n",
    "|Parameter|Characteristics|Description|Default|\n",
    "|---------|-----|------------- |-----|\n",
    "|sub| Required|The string that you are searching for| (no default)|\n",
    "|start|Optional|Specify the start position|Default is 0, corresponds to beginning of the string|\n",
    "|end|Optional|Specify the end position|Default is the end of the string|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this code\n",
    "quote = \"Data Science is so cool, I love Data Science!\"\n",
    "\n",
    "print(\"The new quote is:\" + quote)\n",
    "\n",
    "# Where in the text is the first occurrence of the substring \"Data\" when you only want to search between position 10 and 40?\n",
    "result = quote.find(\"Data\",10,40)\n",
    "\n",
    "print(\"Substring 'Data' from position 10 to 40: \", result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fo572Tu8XsHd"
   },
   "source": [
    "# 2. Project: Cleaning Column Names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UiEgHOWu1cTg"
   },
   "outputs": [],
   "source": [
    "# Import Pandas library\n",
    "import pandas as pd\n",
    "data = pd.read_csv('Data/avocado.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fOftEkPKX_zU"
   },
   "source": [
    "If we take a look at the column names, we notice that they need some cleaning, such as removing the whitespaces. Some systems and data pipelines can have issues with these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XR27PVus3Yxy"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "data_2015 = data[data['year'] == 2015]\n",
    "data_2015.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mBbdudOpYs3O"
   },
   "source": [
    "Let's use a lambda function and three of the methods which we have just learned - strip, lower and replace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7xDOz2sC4cR6"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "data_2015.rename(columns = lambda x: x.strip().lower().replace(' ','_'), inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uLqZJoyZmbSr"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "data_2015.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jU1GAwF_ZMiS"
   },
   "source": [
    "One column is still ugly. It would not be worth it to attempt and write a specific function for it. We address it manually via a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6SxcRhdg422u"
   },
   "outputs": [],
   "source": [
    "# BONUS TASK - Hints: use .rename() method and specify columns through dictionary, i.e. 'column_name_to_clean':'new_column_name'\n",
    "#                   specify inplace = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gDgPRwHzumw5"
   },
   "source": [
    "# 3. Cleaning Text Column (READ-ONLY)\n",
    "\n",
    "Imagine we have 2 possible categories of avocado (A and B) in the same row for the same day separated with '/'. \n",
    "It would be an issue for us if we'd like to explore and visualize data based on the avocado's category. \n",
    "\n",
    "We can use `str.split()` method to resolve this issue in few steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BJByg_FC7kGc"
   },
   "outputs": [],
   "source": [
    "# Run this code - don't bother what it does for now\n",
    "\n",
    "data_avo = {'day':'Monday', 'category':'A/B', 'type':'organic'}\n",
    "monday_data = pd.DataFrame(data_avo, range(10))           "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Pd0U4OwOa233"
   },
   "source": [
    "Let's now examine the special altered dataset which we created. You will notice that in the 'category' column. we have A and B symbols. These represent avocado types, which means that in **every row we have stored 2 observations**. That is not good and we need to split each row into 2 separate rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GXJv_Ccq8JFa"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "monday_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "z49j8IjMbtdQ"
   },
   "source": [
    "At first, we use `split` the method to create a list of two objects from the original element in the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KHGSbu257v1Q"
   },
   "outputs": [],
   "source": [
    "# Firstly, split the 'category' column with separator '/'\n",
    "\n",
    "monday_data['category'] = monday_data['category'].str.split('/')\n",
    "monday_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZQ1upehYyd4n"
   },
   "source": [
    "As the next steps:\n",
    "\n",
    "- next we use `apply()` function on `monday_data` that return Series: use lambda function `lambda x:` to create new Series - we also need to specify `axis = 1` which returns a new column for avocado's type\n",
    "- after the `apply()` part add `stack()` - to stack avocado's category "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d6vVYwZs1TDo"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "\n",
    "series_2 = monday_data.apply(lambda x: pd.Series(x['category']), axis = 1).stack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D89FzDV093iG"
   },
   "source": [
    "As you can see below, **categories are now separated into new rows**: 10 observation for Monday. However there is also new level (another index) for A and B that we don't need anymore. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5ZGhU1RXsmhR"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "series_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6Xshd7QI-lEE"
   },
   "source": [
    "We can remove this index using `reset_index()`: \n",
    "- use `drop = True`\n",
    "- set `level = 1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "905FcOjt9vtt"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "series_2 = monday_data.apply(lambda x: pd.Series(x['category']), axis = 1).stack().reset_index(level = 1, drop = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bBo9HevS_Hi2"
   },
   "source": [
    "- give the Series (it will be a new column) a name 'avocado_category'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3mhMrQUntjca"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "series_2.name = 'avocado_category' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bzOPCvTh_QJ2"
   },
   "source": [
    "- drop the column 'category' from `new_data` (this is the column that contain A/B), set axis = 1\n",
    "- join `series_2` where we have separated categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "F-_2tJIQszxa"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "new_data = monday_data.drop('category', axis = 1).join(series_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HkwheozRtoJJ"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "new_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C3j0M_IscRbA"
   },
   "source": [
    "# 4. Project: Cleaning Text Column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "i4bAOPRoohA_"
   },
   "outputs": [],
   "source": [
    "# Run the code\n",
    "import numpy as np\n",
    "data_1 = pd.read_csv('Data/movie_metadata.csv')\n",
    "movie_data = data_1.iloc[:,np.r_[1:3, 8:13]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_JZCumYgqkgv"
   },
   "outputs": [],
   "source": [
    "# Display first 5 rows of movie_data and look at the genres column\n",
    "movie_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TBG24MHR_nWX"
   },
   "source": [
    "Now we use the same way to split genres of movies, the only difference is the separator '|'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k5_nOjgFqsl1"
   },
   "outputs": [],
   "source": [
    "# Split the 'genres' column with separator '/'\n",
    "\n",
    "movie_data.genres = movie_data.genres.str.split('|')\n",
    "movie_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lS0Yda1VroXH"
   },
   "outputs": [],
   "source": [
    "# Create a new Series for genres using a lambda function and apply it to movie_data\n",
    "\n",
    "series_genres = movie_data.apply(lambda x: pd.Series(x['genres']), axis = 1).stack().reset_index(level = 1,drop = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VGNhJBl8r4E6"
   },
   "outputs": [],
   "source": [
    "# Print the new Series\n",
    "\n",
    "print(series_genres)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cH59Oh_kr6bs"
   },
   "outputs": [],
   "source": [
    "# Give the Series (new column) the name 'genre'\n",
    "series_genres.name = 'genre'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JyGTwgDwr-ND"
   },
   "outputs": [],
   "source": [
    "# TASK 2 >>>> Drop the old column 'genres' from movie_data on axis = 1\n",
    "#             Join to the new Series 'series_genres'. \n",
    "#             Assign it to our_movie_data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "e5WQqTjEuZi2"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "\n",
    "print(our_movie_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hLLlkBhcBORo"
   },
   "source": [
    "# 5. Regular expressions\n",
    "\n",
    "- provide a flexible way to serach or match string patterns in text\n",
    "- a single expression, commonly called a **regex**, is a string formed according to the regular expression language\n",
    "- using built-in module `re` we can apply regular expressions to strings\n",
    "\n",
    "Run the following cell showing example of regular expression for validating an email $^{1}$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this code\n",
    "Image('Images/regex.PNG')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NcvzGMaqBQ2F"
   },
   "outputs": [],
   "source": [
    "# Import re module\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2sqN2mcoILRn"
   },
   "source": [
    "Regex Methods\n",
    "\n",
    "There is a set of methods that allows us to search a string for a match such as:\n",
    "\n",
    "`findall`\n",
    "- returns a list that contain all matches\n",
    "\n",
    "`match`\n",
    "- if zero or more characters at the beginning of string match this regular expression, return a corresponding match object\n",
    "\n",
    "`search`\n",
    "- scan through string looking for the first location where regular expression produces a match and return a corresponding match object\n",
    "\n",
    "`split`\n",
    "- breaks string into pieces at each occurence of pattern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NhduL71wHyRD"
   },
   "outputs": [],
   "source": [
    "# Split string called 'sentence' by whitespaces \n",
    "sentence = 'This  sentence contains     whitespace'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4GzAhCYcL0NO"
   },
   "source": [
    "To split this string we need to call `re.split()`. \n",
    "\n",
    "Within this method we specify regex `'\\s+'` describing one or more whitespace character and string to split (in our case 'sentence').\n",
    "\n",
    "Firstly, the regex is compiled and then the `split` function is called on the passed string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BE6KVu6uLoA2"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "re.split('\\s+', sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Se5uzdW7NeqJ"
   },
   "source": [
    "With `re.compile()` we can combine a regular expression pattern into pattern objects which can be used for pattern matching\n",
    "- this approach is recommended if you intend to apply the same expression to many strings "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TVV7FuY0LniI"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "our_regex = re.compile('\\s+')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WLGs6mbaOCu9"
   },
   "outputs": [],
   "source": [
    "# Split string 'sentence' using regex object 'our_regex'\n",
    "our_regex.split(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Wd_BwQBCOF13"
   },
   "outputs": [],
   "source": [
    "# Get the list of all patterns that match regex using findall() method\n",
    "our_regex.findall(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KhQAznGNOKgv"
   },
   "outputs": [],
   "source": [
    "# Create regex object that match pattern contain 'e'\n",
    "another_regex = re.compile('e')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RA8APTGbOgcP"
   },
   "outputs": [],
   "source": [
    "# Run the code\n",
    "sentence_2 = 'Learning RegEx is fun'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "z67ELANtPnEd"
   },
   "outputs": [],
   "source": [
    "# Return the list that contain all matches in string 'sentence_2'\n",
    "another_regex.findall(sentence_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zvpokw-KQ6U8"
   },
   "source": [
    "As you can see, the regex object performed case-sensitive matching and matched lowercase letters only. \n",
    "\n",
    "We can also define a case insensitive regex object during the pattern compile using `flags = re.IGNORECASE`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_2WvI5ONPqjH"
   },
   "outputs": [],
   "source": [
    "# Create regex object that is not case sensitive using re.IGNORECASE\n",
    "regex_sensitive = re.compile('e', flags = re.IGNORECASE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UAjnTFLMQHOG"
   },
   "outputs": [],
   "source": [
    "# Run this code\n",
    "regex_sensitive.findall(sentence_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DMAFKdrAVzbK"
   },
   "outputs": [],
   "source": [
    "text = 'Regex, Regex pattern, Expressions'\n",
    "\n",
    "# Create a regex object with the matche pattern 's'\n",
    "pattern = re.compile('s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "q7AdilW4WI3D"
   },
   "outputs": [],
   "source": [
    "# Check for a match anywhere in the string using .search()\n",
    "\n",
    "pattern.search(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KykGwLzQeZJp"
   },
   "source": [
    "As you can see `search` returns only the start and end position of the pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DgbJogv0WLRV"
   },
   "outputs": [],
   "source": [
    "# Check for a match only at the beginning of the string using .match()\n",
    "\n",
    "pattern.match(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8t2Loe5SYy7m"
   },
   "outputs": [],
   "source": [
    "# Run this line of code\n",
    "\n",
    "email = 'Email addresses of our two new employees are first.example@gmail.com and second_example@gmail.com'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1qbwQG_SbKCE"
   },
   "outputs": [],
   "source": [
    "# Write a regex to match email addresses\n",
    "\n",
    "email_pattern = r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DoDGF_tpac_a"
   },
   "outputs": [],
   "source": [
    "# Create a regex object that matches email addresses and make it case-insensitive\n",
    "\n",
    "rege = re.compile(email_pattern, flags = re.IGNORECASE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pRpjpeb2a-bU"
   },
   "outputs": [],
   "source": [
    "# Get list of email addresses from 'email' string\n",
    "\n",
    "rege.findall(email)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bEx78TP7coAF"
   },
   "outputs": [],
   "source": [
    "# Search for the position of the first email address in the string 'email'\n",
    "\n",
    "rege.search(email)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DRWIFb9xiIVH"
   },
   "outputs": [],
   "source": [
    "text = 'The average price of the avocados was $1.35 last year, hopefully, this year the price don`t exceed $1.50 for a piece!'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xsnutzsBhTaT"
   },
   "outputs": [],
   "source": [
    "# TASK 3 >>>> Google for Regex patterns to match decimal numbers and assign it to the variable decimal_number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fy8xLeOXhWYR"
   },
   "outputs": [],
   "source": [
    "# Regex object that match decimal number - won't work if TASK 3 is not completed\n",
    "\n",
    "pattern_dec = re.compile(decimal_number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0-fSFbhIhdsx"
   },
   "outputs": [],
   "source": [
    "# Run this code - won't work if TASK 3 is not completed\n",
    "\n",
    "pattern_dec.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find many Regular Expressions Cheat Sheets on the web, like [this one](https://cheatography.com/mutanclan/cheat-sheets/python-regular-expression-regex/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hint**\n",
    "\n",
    "If we want to find some pattern (decimal numbers for example) within the string of a Series, we can also use the pandas function `str.contains`. For more information check the [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.contains.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix\n",
    "\n",
    "Data Source 1: https://www.kaggle.com/neuromusic/avocado-prices\n",
    "\n",
    "License: Database: Open Database, Contents: © Original Authors\n",
    "\n",
    "\n",
    "Data source 2: https://www.kaggle.com/orgesleka/imdbmovies\n",
    "\n",
    "License: CC0: Public Domain\n",
    "\n",
    "# References\n",
    "\n",
    "$^{1}$ BreatheCode. 2017. Regex Tutorial. [ONLINE] Available at: https://content.breatheco.de/en/lesson/regex-tutorial-regular-expression-examples. [Accessed 14 September 2020].\n",
    "\n",
    "pandas. pandas.Series.str.contains. [ONLINE] Available at: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.contains.html. [Accessed 14 September 2020].\n",
    "\n",
    "Material adapted for RBI internal purposes with full permissions from original authors. Source: https://github.com/zatkopatrik/authentic-data-science"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyNF7HG16Du5GggJrLyG3bqp",
   "collapsed_sections": [],
   "name": "5. Strings_And_Regex.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}