Exercises/Exercise_9_Regular_Expressions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regular Expression Assignment Practice Lab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to construct and debug regular expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell defines a large html string that we will use to test some of regular expressions.  When writing a program that is going to depend on accurately extracting instances of certain patterns from text or HTML, you need to create the regular expressions first, testing them on realistic example strings.  You need your expressions to do two things:\n",
    "\n",
    "1. Match the strings you trying to extract, and possibly some context around them, to guarantee you\n",
    "   are extracting the right information;\n",
    "2. If your expression matches context as well as the information you are trying to extract,\n",
    "   (and often it will have to) you need to identify the target part of  the expression.  This is done by placing the target part of \n",
    "   the pattern in parentheses (illustrated below).\n",
    "   \n",
    "The homework assignment asks you to extract the baby name year in the html file.  The line containing the relevant information looks like this\n",
    "     \n",
    "     <h3 align=\"center\">Popularity in 1990</h3>\n",
    "     \n",
    "One regular expression that will match the year is the following:\n",
    "\n",
    "       '\\d\\d\\d\\d'\n",
    "\n",
    "The code below tries out this idea.  Evaluate it and report on the  success of the idea in the markdown cell below the code cell.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8601\n",
      "2005\n",
      "2005\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "html_string = \"\"\"\n",
    "<head><title>Popular Baby Names</title>\n",
    "<meta name=\"dc.language\" scheme=\"ISO639-2\" content=\"eng\">\n",
    "<meta name=\"dc.creator\" content=\"OACT\">\n",
    "<meta name=\"lead_content_manager\" content=\"JeffK\">\n",
    "<meta name=\"coder\" content=\"JeffK\">\n",
    "<meta name=\"dc.date.reviewed\" scheme=\"ISO8601\" content=\"2005-12-30\">\n",
    "<link rel=\"stylesheet\" href=\"../OACT/templatefiles/master.css\" type=\"text/css\" media=\"screen\">\n",
    "<link rel=\"stylesheet\" href=\"../OACT/templatefiles/custom.css\" type=\"text/css\" media=\"screen\">\n",
    "<link rel=\"stylesheet\" href=\"../OACT/templatefiles/print.css\" type=\"text/css\" media=\"print\">\n",
    "</head>\n",
    "<body bgcolor=\"#ffffff\" text=\"#000000\" topmargin=\"1\" leftmargin=\"0\">\n",
    "<table width=\"100%\" border=\"0\" cellspacing=\"0\" cellpadding=\"4\">\n",
    "  <tbody>\n",
    "  <tr><td class=\"sstop\" valign=\"bottom\" align=\"left\" width=\"25%\">\n",
    "      Social Security Online\n",
    "    </td><td valign=\"bottom\" class=\"titletext\">\n",
    "      <!-- sitetitle -->Popular Baby Names\n",
    "    </td>\n",
    "  </tr>\n",
    "  <tr bgcolor=\"#333366\"><td colspan=\"2\" height=\"2\"></td></tr>\n",
    "  <tr><td class=\"graystars\" width=\"25%\" valign=\"top\">\n",
    "       <a href=\"../OACT/babynames/\">Popular Baby Names</a></td><td valign=\"top\"> \n",
    "      <a href=\"http://www.ssa.gov/\"><img src=\"/templateimages/tinylogo.gif\"\n",
    "      width=\"52\" height=\"47\" align=\"left\"\n",
    "      alt=\"SSA logo: link to Social Security home page\" border=\"0\"></a><a name=\"content\"></a>\n",
    "      <h1>Popular Names by Birth Year</h1>September 12, 2007</td>\n",
    "  </tr>\n",
    "  <tr bgcolor=\"#333366\"><td colspan=\"2\" height=\"1\"></td></tr>\n",
    "</tbody></table>\n",
    "<table width=\"100%\" border=\"0\" cellspacing=\"0\" cellpadding=\"4\" summary=\"formatting\">\n",
    "  <tr valign=\"top\"><td width=\"25%\" class=\"greycell\">\n",
    "      <a href=\"../OACT/babynames/background.html\">Background information</a>\n",
    "      <p><br />\n",
    "      &nbsp; Select another <label for=\"yob\">year of birth</label>?<br />      \n",
    "      <form method=\"post\" action=\"/cgi-bin/popularnames.cgi\">\n",
    "      &nbsp; <input type=\"text\" name=\"year\" id=\"yob\" size=\"4\" value=\"1990\">\n",
    "      <input type=\"hidden\" name=\"top\" value=\"1000\">\n",
    "      <input type=\"hidden\" name=\"number\" value=\"\">\n",
    "      &nbsp; <input type=\"submit\" value=\"   Go  \"></form>\n",
    "    </td><td>\n",
    "<h3 align=\"center\">Popularity in 1990</h3>\n",
    "<p align=\"center\">\n",
    "\"\"\"\n",
    "re1 = r'\\d\\d\\d\\d'\n",
    "re1_revised = r'[12]\\d\\d\\d'\n",
    "match = re.search(re1,html_string)\n",
    "match_two = re.search(re1_revised,html_string)\n",
    "# match object tells you positions in string where match begins and ends (match.start() and match.end()).  \n",
    "# Let's look at  this span\n",
    "\n",
    "#match = None\n",
    "#match_two = None\n",
    "if match:\n",
    "   print(html_string[match.start():match.end()])\n",
    "if match_two:\n",
    "   print(html_string[match_two.start():match_two.end()])\n",
    "   print(match_two.group())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Discuss how well this regular expression worked at extracting the year. If it failed, explain why.\n",
    "You may edit this cell.\n",
    "\n",
    "This exercise should have convinced you needed to amend the regular expression to provide some contexts; 4 digits in a row, even if the first is required to be 1 or 2, won't do it.  In the next cell, define and test a new regular expression that does\n",
    "the job. You may want to try some of the exercises in the following sections first, to get some practice with regular expressions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the next html string, you want to find ALL the triples of the form RANK, MALE NAME, FEMALE NAME.\n",
    "Your output should look like this:\n",
    "\n",
    "   [('1', 'Jacob', 'Emma'), ('2', 'Michael', 'Isabella'), ('3', 'Ethan', 'Emily')]\n",
    "   \n",
    "You can get this using `re.findall`.  The next cell gives you a pretty helpful example of how to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['<tr align=\"right\"><td>1</td>',\n",
       "  '<tr align=\"right\"><td>2</td>',\n",
       "  '<tr align=\"right\"><td>3</td>'],\n",
       " ['1', '2', '3'])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "html_str2 = \"\"\"<tr align=\"center\" valign=\"bottom\">\n",
    "  <th scope=\"col\" width=\"12%\" bgcolor=\"#efefef\">Rank</th>\n",
    "  <th scope=\"col\" width=\"41%\" bgcolor=\"#99ccff\">Male name</th>\n",
    "<th scope=\"col\" bgcolor=\"pink\" width=\"41%\">Female name</th></tr>\n",
    "<tr align=\"right\"><td>1</td><td>Jacob</td><td>Emma</td>\n",
    "</tr>\n",
    "<tr align=\"right\"><td>2</td><td>Michael</td><td>Isabella</td>\n",
    "</tr>\n",
    "<tr align=\"right\"><td>3</td><td>Ethan</td><td>Emily</td>\n",
    "</tr>\"\"\"\n",
    "\n",
    "res1 = re.findall(r'<tr\\s+.+><td>\\d+</td>',html_str2)\n",
    "res2 = re.findall(r'<tr\\s+.+><td>(\\d+)</td>',html_str2)\n",
    "\n",
    "(res1, res2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the very different results you get with very similar `findall` requests.  The function `findall` is written so as to retrieve the **groups** in your regular expression. The groups in your regular expression are defined by parentheses.  If there are no groups (no parentheses), `findall` returns a list of complete matches.  So the first result above is what you get for a regular expression with no groups, and the second is what you get for a regular expression with one group.  If your regular expression contains multiple groups, you get a list of tuples.  Each tuple member corresponds to one group in the pattern.  Since you're being asked for a result that is a list of triples, you want a regular expression with 3 groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solving crosswords (requires NLTK)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following example is adapted from [the NLTK Book, Ch. 3.](http://www.nltk.org/book/ch03.html)\n",
    "\n",
    "Let's say we're in the midst of doing a cross word puzzle and we need an 8-letter word\n",
    "whose third letter is *j* and whose sixth letter is *t* which means\n",
    "*sad*.    We want\n",
    "words that match the following regular expression pattern::\n",
    "\n",
    "   '^..j..t..$'\n",
    "\n",
    "Notice that this specifies a string of exactly 8 characters because\n",
    "of the `^` and the `$`, which mark the beginning\n",
    "and ending of the string, respectively.  Each `.` is a wildcard\n",
    "which matches exactly one character but will match any character.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package words to /home/rsouza/nltk_data...\n",
      "[nltk_data]   Package words is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "print(nltk.__version__)\n",
    "nltk.download('words')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "236736\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['abjectly',\n",
       " 'adjuster',\n",
       " 'dejected',\n",
       " 'dejectly',\n",
       " 'injector',\n",
       " 'majestic',\n",
       " 'objectee',\n",
       " 'objector',\n",
       " 'rejecter',\n",
       " 'rejector',\n",
       " 'unjilted',\n",
       " 'unjolted',\n",
       " 'unjustly']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "from nltk.corpus import words\n",
    "wds = words.words()\n",
    "print(len(wds))\n",
    "cands = [w for w in wds if re.search('^..j..t..$',w)]\n",
    "cands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we check our list and there it is: *dejected*.\n",
    "Will you ever be stumped by a crossword puzzle again?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Textonyms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [NLTK Book, Ch. 3](http://www.nltk.org/book/ch03.html>) introduces the following\n",
    "concept of **textonym** with this definition:\n",
    "\n",
    "   The T9 system is used for entering text on mobile phones: Two or more words that are \n",
    "   entered with the same sequence of keystrokes are known as textonyms. For example, \n",
    "   both *hole* and *golf* are entered by pressing the sequence `4653`. What other words could \n",
    "   be produced with the same sequence? \n",
    "\n",
    "   Here we  could use the regular expression `'^[ghi][mno][jlk][def]$'`.  \n",
    "\n",
    "    >>> [w for w in wds if re.search('^[ghi][mno][jlk][def]$', w)]\n",
    "    ['gold', 'golf', 'hold', 'hole']\n",
    "\n",
    "Try the following.  Find all words that can be spelled out with the sequence\n",
    "`3456`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['dilo', 'film', 'filo']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[w for w in wds if re.search('^[def][ghi][jkl][mno]$', w)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['gold', 'golf', 'hold', 'hole', 'gold', 'hole']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[w for w in wds if re.search('^[ghi][mno][jlk][def]$', w)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regular expression practice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\\w\\w\\w\n",
      "\\w\\w\\w\n",
      "<re.Match object; span=(0, 3), match='bcd'>\n",
      "<re.Match object; span=(0, 3), match='1bd'>\n",
      "<re.Match object; span=(0, 3), match='b1d'>\n",
      "None\n",
      "None\n",
      "None\n",
      "<re.Match object; span=(0, 3), match='bda'>\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "pat = r'a|b|c'\n",
    "pat2 = r'[abc]'\n",
    "pat3 = r'\\w\\w\\w'\n",
    "print(pat3)\n",
    "pat4 = '\\\\w\\\\w\\\\w'\n",
    "print(pat4)\n",
    "print(re.match(pat3,'bcd'))\n",
    "print(re.match(pat3,'1bd'))\n",
    "print(re.match(pat3,'b1d'))\n",
    "print(re.match(pat3,'b-d'))\n",
    "print(re.match(pat3,'b?d'))\n",
    "print(re.match(pat3,'b d'))\n",
    "print(re.match(pat3,'bda '))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Edit this cell and after each regular expression, describe the class of strings it matches.  Check your answer examining the output of the code cell that follows.\n",
    "\n",
    "1.  [a-zA-Z]+\n",
    "2.  [A-Z][a-z]*\n",
    "3.  \\d+(\\.\\d+)?\n",
    "4.  ([bcdfghjklmnpqrstvwxyz][aeiou][bcdfghjklmnpqrstvwxyz])*\n",
    "5.  \\w+|[^\\w\\s]+ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "re2 [a-zA-Z]+\n",
      "=============\n",
      "\n",
      "   1. abracadabra                                    abracadabra\n",
      "   2. 1billygoat                                     None\n",
      "   3. billygoat1                                     billygoat\n",
      "   4. 43.1789                                        None\n",
      "   5. 43x1789                                        None\n",
      "   6. 43.                                            None\n",
      "   7. 43                                             None\n",
      "   8. road_runner                                    road\n",
      "   9.  road_runner                                   None\n",
      "  10. bathos                                         bathos\n",
      "  11. The little dog laughed to see such a sight.    The\n",
      "  12. socrates                                       socrates\n",
      "  13. Socrates                                       Socrates\n",
      "  14. *&%#!?                                         None\n",
      "  15. IBM                                            IBM\n",
      "\n",
      "re3 [A-Z][a-z]+\n",
      "===============\n",
      "\n",
      "   1. abracadabra                                    None\n",
      "   2. 1billygoat                                     None\n",
      "   3. billygoat1                                     None\n",
      "   4. 43.1789                                        None\n",
      "   5. 43x1789                                        None\n",
      "   6. 43.                                            None\n",
      "   7. 43                                             None\n",
      "   8. road_runner                                    None\n",
      "   9.  road_runner                                   None\n",
      "  10. bathos                                         None\n",
      "  11. The little dog laughed to see such a sight.    The\n",
      "  12. socrates                                       None\n",
      "  13. Socrates                                       Socrates\n",
      "  14. *&%#!?                                         None\n",
      "  15. IBM                                            None\n",
      "\n",
      "re4 \\d+(\\.\\d+)?\n",
      "===============\n",
      "\n",
      "   1. abracadabra                                    None\n",
      "   2. 1billygoat                                     1\n",
      "   3. billygoat1                                     None\n",
      "   4. 43.1789                                        43.1789\n",
      "   5. 43x1789                                        43\n",
      "   6. 43.                                            43\n",
      "   7. 43                                             43\n",
      "   8. road_runner                                    None\n",
      "   9.  road_runner                                   None\n",
      "  10. bathos                                         None\n",
      "  11. The little dog laughed to see such a sight.    None\n",
      "  12. socrates                                       None\n",
      "  13. Socrates                                       None\n",
      "  14. *&%#!?                                         None\n",
      "  15. IBM                                            None\n",
      "\n",
      "re5 ([bcdfghjklmnpqrstvwxyz][aeiou][bcdfghjklmnpqrstvwxyz])*\n",
      "============================================================\n",
      "\n",
      "   1. abracadabra                                    \n",
      "   2. 1billygoat                                     \n",
      "   3. billygoat1                                     bil\n",
      "   4. 43.1789                                        \n",
      "   5. 43x1789                                        \n",
      "   6. 43.                                            \n",
      "   7. 43                                             \n",
      "   8. road_runner                                    \n",
      "   9.  road_runner                                   \n",
      "  10. bathos                                         bathos\n",
      "  11. The little dog laughed to see such a sight.    \n",
      "  12. socrates                                       socrat\n",
      "  13. Socrates                                       \n",
      "  14. *&%#!?                                         \n",
      "  15. IBM                                            \n",
      "\n",
      "re6 \\w+|[^\\w\\s]+\n",
      "================\n",
      "\n",
      "   1. abracadabra                                    abracadabra\n",
      "   2. 1billygoat                                     1billygoat\n",
      "   3. billygoat1                                     billygoat1\n",
      "   4. 43.1789                                        43\n",
      "   5. 43x1789                                        43x1789\n",
      "   6. 43.                                            43\n",
      "   7. 43                                             43\n",
      "   8. road_runner                                    road_runner\n",
      "   9.  road_runner                                   None\n",
      "  10. bathos                                         bathos\n",
      "  11. The little dog laughed to see such a sight.    The\n",
      "  12. socrates                                       socrates\n",
      "  13. Socrates                                       Socrates\n",
      "  14. *&%#!?                                         *&%#!?\n",
      "  15. IBM                                            IBM\n"
     ]
    }
   ],
   "source": [
    "########################################\n",
    "###     Some regular expressions     ###\n",
    "########################################\n",
    "\n",
    "re2 = r'[a-zA-Z]+'   #Any string consisting of ltters of the alphabet, upper or lower case\n",
    "re3 = r'[A-Z][a-z]+'  \n",
    "re4 = r'\\d+(\\.\\d+)?'\n",
    "re5 = r'([bcdfghjklmnpqrstvwxyz][aeiou][bcdfghjklmnpqrstvwxyz])*'\n",
    "re6 = r'\\w+|[^\\w\\s]+'\n",
    "\n",
    "res = [re2,re3,re4,re5,re6]\n",
    "\n",
    "########################################\n",
    "###     Some example strings         ###\n",
    "########################################\n",
    "\n",
    "example1 = 'abracadabra'\n",
    "example2 = '1billygoat'\n",
    "example3 = 'billygoat1'\n",
    "example4 = '43.1789'\n",
    "example4a = '43x1789'\n",
    "example5 = '43.'\n",
    "example6 = '43'\n",
    "example7 = 'road_runner'\n",
    "example8 = ' road_runner'\n",
    "example9 = 'bathos'\n",
    "example10 = \"The little dog laughed to see such a sight.\"\n",
    "example11 = 'socrates'\n",
    "example12 = 'Socrates'\n",
    "example13 = '*&%#!?'\n",
    "example14 = 'IBM'\n",
    "\n",
    "examples = [example1,example2,example3,example4,example4a,example5,example6,\n",
    "            example7,example8,example9,example10,example11,example12,example13,\n",
    "            example14]\n",
    "\n",
    "########################################\n",
    "###     Trying some matches          ###\n",
    "########################################\n",
    "\n",
    "for i,re_pat in enumerate(res):\n",
    "    banner = 're%d %s' % (i+2,re_pat)\n",
    "    print() \n",
    "    print(banner)\n",
    "    print('=' * len(banner))\n",
    "    print()\n",
    "    for (i,ex) in enumerate(examples):\n",
    "        match = re.match(re_pat,ex)\n",
    "        if match:\n",
    "            print('  %2d. %-45s  %s' % (i+1,ex,ex[match.start():match.end()]))\n",
    "        else:\n",
    "            print('  %2d. %-45s  %s' %(i+1,ex,None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure you can answer the following questions about the results of testing these regular expressions on the examples:\n",
    "\n",
    "1. Why does `re2` fail on `example8`?\n",
    "1. Why does `re3` only succeed on `example10` and `example12`?  Be sure to explain why it fails\n",
    "   on `example14`.\n",
    "1. When 're4' matches 'example5', why isn't the decimal point part of the match?\n",
    "1. All of the regular expressions except `re5` report a `None` with at least one\n",
    "   one of the examples.  Why doesn't `re5` report any `None`s?\n",
    "1. Why does `re6` match all the characters in `example13`?\n",
    "1. Why doesnt `re6` match `example8`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An example that requires NLTK to be installed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   To run the code for this example, you will use a **balanced corpus**\n",
    "   of English texts, a corpus collected with the purpose of representing\n",
    "   a balanced variety of English text types: fiction, poetry, speech,\n",
    "   non fiction, and so on.  One relatively well-established, free,\n",
    "   and easy-to-get example of such a corpus is the **Brown Corpus.**\n",
    "   Brown is about 1.2 M words. \n",
    "   \n",
    "   You can import the corpus as follows::\n",
    "\n",
    "     >>> from nltk.corpus import brown\n",
    "\n",
    "\n",
    "   If this does not work, it is because you have nltk installed without the accompanying\n",
    "   corpora. You can download any nltk corpus you need through the `nltk.download` function For example,\n",
    "   to get the Brown corpus, do the following in Python::\n",
    "      \n",
    "      >>> import nltk\n",
    "      >>> nltk.download()\n",
    "\n",
    "   This brings up a window you can interact with.  There are some tabs\n",
    "   at the top.  Choose the tab labeled *Corpora*,\n",
    "   select **Brown**, and click the **download** button\n",
    "   at the bottom of the window.   You will then have\n",
    "   Brown on your machine and you can import the corpus as follows::\n",
    "\n",
    "     >>> from nltk.corpus import brown\n",
    "\n",
    "   The following returns a list of all 1.2 M word tokens in Brown::\n",
    "\n",
    "     >>> brown.words()\n",
    "     ['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...]\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('io', 2787),\n",
       " ('ea', 2249),\n",
       " ('ou', 1855),\n",
       " ('ie', 1799),\n",
       " ('ia', 1400),\n",
       " ('ee', 1289),\n",
       " ('oo', 1174),\n",
       " ('ai', 1145),\n",
       " ('ue', 541),\n",
       " ('au', 540),\n",
       " ('ua', 502),\n",
       " ('ei', 485),\n",
       " ('ui', 483),\n",
       " ('oa', 466),\n",
       " ('oi', 412),\n",
       " ('eo', 250),\n",
       " ('iou', 225),\n",
       " ('eu', 187),\n",
       " ('oe', 181),\n",
       " ('iu', 128),\n",
       " ('ae', 85),\n",
       " ('eau', 54),\n",
       " ('uo', 53),\n",
       " ('eou', 52),\n",
       " ('uou', 37)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# From http://www.nltk.org/book/ch03.html\n",
    "#  Find the most common vowel sequences in English.  Note: be patient.  Evaluating this may take a while.\n",
    "from nltk.corpus import brown\n",
    "from collections import Counter\n",
    "bw = sorted(set(brown.words()))\n",
    "# Find every instance of two or more consecutive vowels, and count tokens of each.\n",
    "ctr = Counter(vs  for word in bw for vs in re.findall('[aeiou]{2,}',word)\n",
    "              )\n",
    "ctr.most_common(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Poker examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose you are writing a poker program where a player’s hand is represented as a 5-character string with each character representing a card, “a” for ace, “k” for king, “q” for queen, “j” for jack, “t” for 10, and “2” through “9” representing the card with that value.\n",
    "\n",
    "To see if a given string is a valid hand, one could run the code in t he following cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "akt5q\t[akt5q]\n",
      "akt5e\tNone\n",
      "akt\tNone\n",
      "727ak\t[727ak]\n",
      "727aka\tNone\n",
      "aaaaa\t[aaaaa]\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "def displaymatch(regex,text):\n",
    "    match = regex.match(text)\n",
    "    if match is None:\n",
    "        matchstring = None\n",
    "    else:\n",
    "        matchstring = f'{text[:match.start()]}[{text[match.start():match.end()]}]{text[match.end():]}'\n",
    "    print(f'{text}\\t{matchstring}')\n",
    "\n",
    "valid = re.compile(r\"^[a2-9tjqk]{5}$\")\n",
    "\n",
    "## Some examples\n",
    "displaymatch(valid, \"akt5q\")  # Valid.\n",
    "displaymatch(valid, \"akt5e\")  # Invalid.\n",
    "displaymatch(valid, \"akt\")    # Invalid.\n",
    "displaymatch(valid, \"727ak\")  # Valid.\n",
    "displaymatch(valid, \"727aka\")  # Invalid.\n",
    "displaymatch(valid, \"aaaaa\")  # Invalid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hand \"727ak\" contains a pair, and we would like to recognize such hands as special, so that we can go all in.  We can do this using regular expression groups and register references.  The match for each parenthesized part of a regular expression is called a **group**.  We can refer back to the particular match  associated with a group with \\integer.  Where integer is any integer from 1 through 9.  \\1 refers to the first group, \\2 to the second, and so on.  So to match poker hands with pairs, we do the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "727ak      [727ak]\n",
      "723ak      None\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'7'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pair = re.compile(r\".*(.).*\\1.*\")\n",
    "displaymatch(pair,\"727ak\")\n",
    "displaymatch(pair,\"723ak\")\n",
    "pair.match(\"727ak\").groups()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a2aak      [a2aak]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'a'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "displaymatch(pair,\"a2aak\")\n",
    "pair.match(\"aa2ak\").groups()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, the regex `pair` does not require the text string to be a Poker hand.  We could revise it to do that and if you think about it a little, it would actually make the regex  **a lot** more complicated.  What we could do instead is first apply `valid` to guarantee we've got a valid poker hand and then apply `pair` to find out if it contains a pair. This makes both regexes simple and easy to understand and still enforce all the constraints we want.  Often a good strategy in applying regexes to enforce some complicated constraints is to divide the constraints up into separate categories and apply them **in succession.**.  \n",
    "\n",
    "A problem with `pair` is that it doesnt tell us  what we've got a pair of.  Actually, the match object contains this information.  It has an attribute called `groups` which contains all portions of the string that matched a group.  We can use a revised version of `displaymatch` to print this, when requested:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pair\n",
      "723ak\tNone\n",
      "7a3ak\t[7a3a]k\t('a',)\n",
      "\n",
      "two pair\n",
      "7a272\t[7a272]\t('7', '2')\n",
      "722a7\tNone\n",
      "7722a\tNone\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "def displaymatch(regex,text, print_groups=False):\n",
    "    match = regex.match(text)\n",
    "    if match is None:\n",
    "        matchstring = None\n",
    "    else:\n",
    "        matchstring = '%s[%s]%s' % (text[:match.start()],text[match.start():match.end()],text[match.end():])\n",
    "    if print_groups and match:\n",
    "        print(f'{text}\\t{matchstring}\\t{match.groups()}')\n",
    "    else:\n",
    "        print(f'{text}\\t{matchstring}')\n",
    "\n",
    "# Re for recognizing pair hands\n",
    "pair = re.compile(r\".*(.).*\\1\")\n",
    "print(\"pair\")\n",
    "displaymatch(pair,\"723ak\",print_groups=True)\n",
    "displaymatch(pair,\"7a3ak\",print_groups=True)\n",
    "print()\n",
    "## Write your regex for recognizing two pair below. Test\n",
    "## This version is not adequate. Look at the examples to see why.\n",
    "print(\"two pair\")\n",
    "two_pair = re.compile(r\".*(.).*(.).*\\1.*\\2.*\")\n",
    "displaymatch(two_pair,\"7a272\",print_groups=True)\n",
    "displaymatch(two_pair,\"722a7\",print_groups=True)  # shd succeed, does not\n",
    "displaymatch(two_pair,\"7722a\",print_groups=True)  # shd succeed, does not\n",
    "#displaymatch(two_pair,\"7a722\",print_groups=True)\n",
    "#displaymatch(two_pair,\"727a2\",print_groups=True)\n",
    "#displaymatch(two_pair,\"aaaa2\",print_groups=True)  # Will succeed on this one, but that's ok"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.  Write regexes that match three-of-a-kind hands,  and four-of-a-kind hands.  Follow the model of `pairs` and dont bother to\n",
    "    guarantee that it's a valid Poker hand.\n",
    "2.  It's quite complex to write a regular expression that checks to see if you've got a straight, but you can try the \n",
    "    following strategy.  First, verify you've got a valid poker hand; then verify you havent got a pair, three-of-kind, or\n",
    "    four-of-a-kind.  So you have a valid poker hand with no repetitions and you dont need the regex that checks for straights\n",
    "    to rule those out.\n",
    "    \n",
    "    Now write a regex that will check to see if a valid poker hand \n",
    "    with no repetitions is a straight  beginning with '2'.  It should succeed on `23456` and `25643` and `32654` and it should fail\n",
    "    `24357`.  To deal with all possible straights in this way, how many cases are there to take care of?  Write a single regular\n",
    "    expression that will identify any straight, given that it is a valid poker hand with no repetitions.  Test it on the \n",
    "    straights above and on straights like `akqjt` and on the non-straight `24357`.\n",
    "3.  Write a regex that matches a two pair hand. This is tricky and the most natural answer will also match four-of-a-kind. \n",
    "    Assume we've eliminated that possibility by failing to match the four-of-kind pattern from 1.  You should \n",
    "    test `722a7`, `7a722` and `727a2`.  You will need a pattern that is a big disjunction using `|`, and you will need to\n",
    "    enclose the disjuncts of this big disjunction in parentheses, but for that purpose you will need parentheses that don't\n",
    "    count as defining a retrievable group.  The notation for that is `(?:` instead of `(` [the same right paren is used \n",
    "    in both cases]. See [Python regex docs.](http://docs.python.org/2/library/re.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to do extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following example is from `The weather underground page for San Diego <http://www.wunderground.com/weather-forecast/US/CA/San_Diego.html>`_.  The temperature is regularly given in a page division (HTML tag `div`) with ID (HTML attribute `divID`) `NowTemp`.  If we can find that division and the temperature inside it, we have what we want.  The pattern needs to be compiled with flags that allow it to match across multiple lines, because the context that identifies the temperature does not occur on the same line as the temperature.  Compiling regular expressions also makes them more efficient when reused.  A key point is that we place the actual temperature we want inside parentheses, the `(\\d{1,3}\\.\\d)` part of the pattern.  Portions of a pattern that occur in parentheses and are matched are placed ins the `groups` attribute of  the match object.  The groups attribute is a tuple of all the matched strings in parentheses in the pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['55.8']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "html_string = \"\"\"\n",
    "<div class=\"br10\" id=\"stationSelect\">\n",
    "\t\t<a class=\"br10\" id=\"stationselector_button\" href=\"javascript:void(0);\" onclick=\"_gaq.push(['_trackEvent', 'Station Select', 'Opened']);\"><span>Station Select</span></a>\n",
    "\t\t</div>\n",
    "\t\t</div>\n",
    "\t\t<div id=\"conds_dashboard\">\n",
    "\t\t<div id=\"hour00\">\n",
    "\t\t<div id=\"nowCond\">\n",
    "\t\t<div class=\"titleSubtle\">Now</div>\n",
    "\t\t<div id=\"curIcon\"><a href=\"\" class=\"iconSwitchBig\"><img src=\"http://icons-ak.wxug.com/i/c/k/nt_partlycloudy.gif\" width=\"44\" height=\"44\" alt=\"Scattered Clouds\" class=\"condIcon\" /></a></div>\n",
    "\t\t<div id=\"curCond\">Scattered Clouds</div>\n",
    "\t\t</div>\n",
    "\t\t<div id=\"nowTemp\">\n",
    "\t\t<div class=\"titleSubtle\">Temperature</div>\n",
    "\t\t<div id=\"tempActual\"><span id=\"rapidtemp\" class=\"pwsrt\" pwsid=\"KCASANDI123\" pwsunit=\"english\" pwsvariable=\"tempf\" english=\"&deg;F\" metric=\"&deg;C\" value=\"55.8\">\n",
    "  <span class=\"nobr\"><span class=\"b\">55.8</span>&nbsp;&deg;F</span>\n",
    "</span></div>\n",
    "\t\t<div id=\"tempFeel\">Feels Like\n",
    "  <span class=\"nobr\"><span class=\"b\">55.1</span>&nbsp;&deg;F</span>\n",
    "</div>\n",
    "\t\t</div>\n",
    "\"\"\"\n",
    "pattern = r'<div\\s+id\\s*=\\s*\\\"tempActual\\\"\\s*>.*?(\\d{1,3}\\.\\d).*?</div>'\n",
    "pattern_re = re.compile(pattern,re.MULTILINE | re.DOTALL)\n",
    "#m = re.search(pattern_re,html_string)\n",
    "#m.groups()\n",
    "pattern_re.findall(html_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pattern in the example above was built up piece by piece.  First we built a regular expression matching the `<div id=\"nowTemp\">` part of the pattern.  That piece looked like this:\n",
    "    \n",
    "     subpattern = r'<div\\s+id\\s*=\\s*\\\"nowTemp\\\"\\s*>\n",
    " \n",
    " The `\\s*` aren't needed for this particular string, but there is considerable variation in how actual HTML is generated, and since\n",
    " white space in the `\\s*` positions wouldn't be meaningful, it is allowed.  Next we tested the core part of the pattern on its own:\n",
    " \n",
    "     corepattern = r'(\\d{1,3}\\.\\d)'\n",
    "  \n",
    "  Finally we tested the last part:\n",
    "  \n",
    "     lastpattern = r`</div>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization  (NLTK assumed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenization is the process of breaking up a text into words.  We have in some cases used `split()` for this purpose, uniformly splitting a text up into words on the spaces, but this doesn't always yield the right results, as the next examples show.\n",
    "\n",
    "There are three tokenizations of `text` string defined in the cell\n",
    "below, `try1`, `try2`, and `try3`; `try1` shows what happens\n",
    "when we just use the Python `split`; `try2` and `try3` use a regular\n",
    "expression that defines different cases of a proper word,\n",
    "such as \n",
    "\n",
    "1. an abbreviation with periods\n",
    "2. an ordinary alphabetic word, with an optional hyphen \n",
    "3. a string of digits, possibly with a decimal, a dollar sign,\n",
    "   or a percent\n",
    " \n",
    "and so on.  We apply this pattern to the example string `text`, \n",
    "using the `re` module function `findall` to find\n",
    "all substrings of `text` that match the pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# From http://www.nltk.org/book/ch03.html\n",
    "import re\n",
    "\n",
    "text = \"\"\"\n",
    "\"That,\" said  Fred, \"is what\n",
    "you ... get in the U.S.A. for $5.29.\"\n",
    "\"\"\"\n",
    "try1 = text.split()\n",
    "\n",
    "# Notice the use of special NONCAPTURING parens (?:...)\n",
    "# All parens in the regexp must be non capturing.\n",
    "pattern = r\"\"\" \n",
    "   (?:[A-Z]\\.)+        # abbreviations, e.g. U.S.A.\n",
    "  |\\w+(?:-\\w+)*        # words with optional internal hyphens\n",
    "  |\\$?\\d+(?:\\.\\d+)?%?  # numbers, money and percents, e.g. 3.14, $12.40, 82%\n",
    "  |\\.\\.\\.            # ellipsis\n",
    "  |[][.,;\"'?():-_`]  # keep punctuation, delimiters as separate word tokens\n",
    "\"\"\"\n",
    "re_flags = re.UNICODE | re.MULTILINE | re.DOTALL | re.X\n",
    "pattern_re = re.compile(pattern,re_flags)\n",
    "try2 = pattern_re.findall(text)\n",
    "# Or equivalently, let nltk do some of the work.\n",
    "import nltk\n",
    "try3 = nltk.regexp_tokenize(text,pattern,flags=re_flags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\"That,\"',\n",
       " 'said',\n",
       " 'Fred,',\n",
       " '\"is',\n",
       " 'what',\n",
       " 'you',\n",
       " '...',\n",
       " 'get',\n",
       " 'in',\n",
       " 'the',\n",
       " 'U.S.A.',\n",
       " 'for',\n",
       " '$5.29.\"']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "try1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `split` tokenized sentence has some very strange words, for example the 7-character strings `\"Fred,'\"` and `\"That,\"`,  and the 3-character string `\"is`. What's being missed here is that certain characters (like comma and quotation-mark) unambiguously mark a word boundary.  Regular expressions are very good at enforcing this sort of generalization, as we can see by comparing the results of tokenizing the same sentence with a regexp that does not allow words to continue past boundary markers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next two tries, we use such a regular expression (defined\n",
    "as `pattern`), compiling it using `re.compile` (for efficiency) and using some compiling flags.  See `re` module docs for a complete description.\n",
    "Here, we'll discuss just the most frequently used one, the `X` flag:\n",
    "\n",
    ">  This flag allows you to write regular expressions \n",
    ">  that look nicer and are more readable by allowing \n",
    ">  you to visually separate logical sections of the pattern\n",
    ">  and add comments. Whitespace within the pattern is ignored, \n",
    ">  except when in a character class, or when preceded by an \n",
    ">  unescaped backslash, or in groups or inisde a few special operators.  \n",
    "\n",
    ">  When a line contains a comment character (#) that is not \n",
    ">  in a character class and is not preceded by an unescaped \n",
    ">  backslash, all characters from the leftmost such # through \n",
    ">  the end of the line are ignored.\n",
    "\n",
    "Next, we call `re.findall(pattern, text)`; `re.findall(pattern, text)` returns a list of all the expressions in `text` that match `pattern`. \n",
    "Since each part (line) of `pattern` is written so as to match\n",
    "a different case of a proper word, `re.findall(pattern, text)`\n",
    "returns a list of the proper words in `text`.\n",
    "Note that all parentheses on `pattern` are what the `re`-module docs call \"non-capturing\".  This means no **groups** are defined by these parens, the\n",
    "matches against expressions in such parens are not put into\n",
    "a register, and they are not returned as separate components\n",
    "in a `findall`.   This is what we want  since the parentheses\n",
    "in `pattern` wrap around parts of words, and we don't want the\n",
    "tokenizer returning word parts, just complete words.\n",
    "\n",
    "The results of using `findall` and the `nltk` tokenizer are equivalent.\n",
    "Basically what the `nltk` tokenizer is compile the regexp using\n",
    "flags and use `findall`.  The `nltk` tokenizer also offers another\n",
    "option, that of writing a tokenizer that matches all word\n",
    "**boundaries** and then using the `re` module method `split`.  That approach\n",
    "has some advantages in some situations, but it is not shown here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\"',\n",
       " 'That',\n",
       " ',',\n",
       " '\"',\n",
       " 'said',\n",
       " 'Fred',\n",
       " ',',\n",
       " '\"',\n",
       " 'is',\n",
       " 'what',\n",
       " 'you',\n",
       " '...',\n",
       " 'get',\n",
       " 'in',\n",
       " 'the',\n",
       " 'U.S.A.',\n",
       " 'for',\n",
       " '$5.29',\n",
       " '.',\n",
       " '\"']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "try3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "try2 == try3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python regular expressions use parentheses for two different things, defining retrievable groups, which as we saw, is useful for extraction, and defining the scope of some regular expression operator (like `*` or `+`). Sometimes these two roles get in each other's way.  This is what happens in `pattern` above: Python `findall` handles groups specially and incorrectly treats the parenthesized elements as groups; so we use the regular expression convention of changing `(` to '(?:'.  The \"(?:' functions unambiguously to scope an operator and does not define a retrievable group.  Rather than make this change by hand, we call the convenient NLTK function `convert_regexp_to_nongrouping`.  We then compile the regular expression using various regular expression compiling flags.  `re.MULTILINE` and `re.DOTALL` allow our regular tokenizing `pattern` to match across lines, while `re.UNICODE` allows our definition of word, which depends on the interpretation of `\\w` to apply to UNICODE characters.  Finally, `re.X` is the most directly relevant to this example.  This allows regular expressions that intersperse comments, which makes them much more readable.  See [Python.org re docs](http://docs.python.org/2/library/re.html) for more details.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"\n",
    "\"That,\" said  Fred, \"is what\n",
    "you ... get in the U.S.A. for $5.29.\"\n",
    "\"\"\"\n",
    "# This is illegal, do you know why?\n",
    "#patx = r'\\b\\B+\\b'\n",
    "patx = r'\\w+'\n",
    "re_flags = re.UNICODE | re.MULTILINE | re.DOTALL | re.X\n",
    "patx_re = re.compile(patx,re_flags)\n",
    "try4 = patx_re.findall(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is what you get.  Is this a good result?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['That',\n",
       " 'said',\n",
       " 'Fred',\n",
       " 'is',\n",
       " 'what',\n",
       " 'you',\n",
       " 'get',\n",
       " 'in',\n",
       " 'the',\n",
       " 'U',\n",
       " 'S',\n",
       " 'A',\n",
       " 'for',\n",
       " '5',\n",
       " '29']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "try4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence boundary detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "text = \"\"\"\n",
    "The king rarely saw Marie \n",
    "on Tuesdays, but\n",
    "he did see her  on Wednesdays.  He liked\n",
    "to take long walks\n",
    "in the garden, gazing longingly at the\n",
    "rhododendrons.  She\n",
    "thought this\n",
    "odd.  Me, too.\n",
    "\"\"\"\n",
    "lines = re.split(r'\\s*[!?.]\\s*', text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\nThe king rarely saw Marie \\non Tuesdays, but\\nhe did see her  on Wednesdays',\n",
       " 'He liked\\nto take long walks\\nin the garden, gazing longingly at the\\nrhododendrons',\n",
       " 'She\\nthought this\\nodd',\n",
       " 'Me, too',\n",
       " '']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's clean this up removing unnecessary line breaks and white space.\n",
    "For each element in `lines`, we split it, then put the pieces back together separated \n",
    "by single spaces.  Finally,we remove empty strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The king rarely saw Marie on Tuesdays, but he did see her on Wednesdays',\n",
       " 'He liked to take long walks in the garden, gazing longingly at the rhododendrons',\n",
       " 'She thought this odd',\n",
       " 'Me, too']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences0 = [' '.join(line.split()) for line in lines]\n",
    "sentences = [exp for exp in sentences0 if exp]\n",
    "sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We wrap it all up in a function, supplying the above pattern\n",
    "as a default if the user doesn't specify one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sent_tokenize (text, pat=r'\\s*[!?.]\\s*'):\n",
    "    lines = re.split(pat, text)\n",
    "    sentences0 = [' '.join(line.split()) for line in lines]\n",
    "    return [exp for exp in sentences0 if exp]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it all  together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next cell we use **negative lookahead**, which allows us \n",
    "to match an instance of one pattern as long as it is not immediately\n",
    "followed by an instance of another.  For example, using `r\"Isaac(?!\\s+Asimov)\"`\n",
    "to define a pattern that matches \"Isaac\" when it is not immediately followed by\n",
    "\" Asimov\", we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Isaac', 'Isaac']\n",
      "['Isaac']\n"
     ]
    }
   ],
   "source": [
    "text = 'Isaac Asimov patted Isaac Stern on the back'\n",
    "print(re.findall(r\"Isaac\",text))\n",
    "print(re.findall(r\"Isaac(?!\\s*Asimov)\",text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We input a raw text string and first\n",
    "tokenize sentences, then words within sentences,\n",
    "returning a list of tokenized sentences.\n",
    "Each tokenized sentence is  a list of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['The',\n",
       "  'king',\n",
       "  'rarely',\n",
       "  'saw',\n",
       "  'Marie',\n",
       "  'on',\n",
       "  'Tuesdays',\n",
       "  ',',\n",
       "  'but',\n",
       "  'he',\n",
       "  'did',\n",
       "  'see',\n",
       "  'her',\n",
       "  'on',\n",
       "  'Wednesdays'],\n",
       " ['He',\n",
       "  'liked',\n",
       "  'to',\n",
       "  'take',\n",
       "  'long',\n",
       "  'walks',\n",
       "  'in',\n",
       "  'the',\n",
       "  'garden',\n",
       "  ',',\n",
       "  'gazing',\n",
       "  'longingly',\n",
       "  'at',\n",
       "  'the',\n",
       "  'rhododendrons'],\n",
       " ['She', 'thought', 'this', 'odd'],\n",
       " ['Me', ',', 'too'],\n",
       " ['\"',\n",
       "  'That',\n",
       "  ',',\n",
       "  '\"',\n",
       "  'said',\n",
       "  'Fred',\n",
       "  ',',\n",
       "  '\"',\n",
       "  'is',\n",
       "  'what',\n",
       "  'you',\n",
       "  '(',\n",
       "  'Texans',\n",
       "  '!',\n",
       "  ')',\n",
       "  'get',\n",
       "  'in',\n",
       "  '1',\n",
       "  '/',\n",
       "  '2',\n",
       "  'the',\n",
       "  'U.S.A.',\n",
       "  'for',\n",
       "  '$5.29',\n",
       "  ',',\n",
       "  '.23%',\n",
       "  'of',\n",
       "  'nothing',\n",
       "  '.',\n",
       "  '\"']]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "import re\n",
    "\n",
    "pattern = r\"\"\" \n",
    "   (?:[A-Z]\\.)+        # abbreviations, e.g. U.S.A.\n",
    "  |\\$?\\d+(?:\\.\\d+)?%?  # numbers, money and percents, e.g. 3.14, $12.40, 82% \n",
    "  |\\$?\\.\\d+%?         # numbers, money and percents, e.g. .14, $.40, '/8%     \n",
    "  |\\w+(?:-\\w+)*        # words with optional internal hyphens. NB \\w includes \\d\n",
    "  |\\.\\.\\.            # ellipsis\n",
    "  |[][./,;\"'!?():-_`]  # keep punctuation, delimiters as separate word tokens\n",
    "\"\"\"\n",
    "\n",
    "re_flags = re.UNICODE | re.MULTILINE | re.DOTALL | re.X\n",
    "# Add in to our sentence boundary pattern\n",
    "# that the next letters following the sentence ender\n",
    "# must NOT be a lower case letter (a-z).\n",
    "back_pat = '\\s*[!?.]\\s+(?![a-z])'\n",
    "def sent_tokenize (text, pat = '\\s*[!?.]\\s+'):\n",
    "    lines = re.split(pat, text)\n",
    "    sentences0 = [' '.join(line.split()) for line in lines]\n",
    "    return [exp for exp in sentences0 if exp]\n",
    "\n",
    "text = \"\"\"\n",
    "The king rarely saw Marie \n",
    "on Tuesdays, but\n",
    "he did see her  on Wednesdays.  He liked\n",
    "to take long walks\n",
    "in the garden, gazing longingly at the\n",
    "rhododendrons.  She\n",
    "thought this\n",
    "odd.  Me, too.\n",
    "\"That,\" said  Fred, \"is what\n",
    "you (Texans!) get in 1/2 the U.S.A. for $5.29, .23% of nothing.\"\n",
    "\"\"\"\n",
    "sents = sent_tokenize(text,pat = back_pat)\n",
    "tokenized_sents = [nltk.regexp_tokenize(sent, pattern, flags=re_flags)\n",
    "                   for sent in sents]\n",
    "tokenized_sents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "226px",
    "width": "252px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}