Notebooks/Python_Tasks/04_NLP_Regular_Expressions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "stuffed-surgeon",
   "metadata": {},
   "source": [
    "# Introduction to Python  \n",
    "\n",
    "## Regular Expressions Tutorial\n",
    "\n",
    "Inspired in these sources: [1](https://github.com/KennyMiyasato/regex_blog_post/blob/master/regex_blog_post.ipynb), [2](https://luca-pessina.medium.com/python-regular-expressions-in-5-minutes-ecc8b6624308), [3](https://developers.google.com/edu/python/regular-expressions)  \n",
    "Tables from: [regexr.com](https://regexr.com/)  \n",
    "Regex [Cheat Sheet](https://www.rexegg.com/regex-quickstart.html)  \n",
    "Test your regexes [here:](https://regex101.com/)  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tribal-oriental",
   "metadata": {},
   "source": [
    "#### Regular expressions are a powerful language for matching text patterns. This page gives a basic introduction to regular expressions themselves sufficient for our Python exercises and shows how regular expressions work in Python. The Python \"re\" module provides regular expression support.\n",
    "\n",
    "#### In Python a regular expression search is typically written as:\n",
    "\n",
    "> match = re.search(pattern, str)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "round-device",
   "metadata": {},
   "source": [
    "### Importing modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ideal-liabilities",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stylish-consumption",
   "metadata": {},
   "source": [
    "### Defining sample strings "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "collaborative-pound",
   "metadata": {},
   "outputs": [],
   "source": [
    "lowercase_alphabet = \"abcdefghijklmnopqrstuvwxyz\"\n",
    "uppercase_alphabet = lowercase_alphabet.upper()\n",
    "numbers = \"1234567890\"\n",
    "sentence = \"The Quick Brown Fox Jumps Over The Lazy Dog\"\n",
    "website = \"https://webmail2016.univie.ac.at/\"\n",
    "phone_numbers = \"\"\"123-456-7890\n",
    "                    987.654.321\n",
    "                    234-567-8901\n",
    "                    654.321.987\n",
    "                    345-678-9012\n",
    "                    321.654.978\n",
    "                    456-789-0123\n",
    "                \"\"\"\n",
    "special_characters = \"[\\^$.|?*+()\"\n",
    "\n",
    "text_with_email = 'xyz alice-b@google.com purple monkey'\n",
    "\n",
    "long_text = '''\n",
    "Nicht alles, was Gold ist, funkelt\n",
    "Nicht alles, was Gold ist, funkelt,\n",
    "Nicht jeder, der wandert, verlorn,\n",
    "Das Alte wird nicht verdunkelt,\n",
    "Noch Wurzeln der Tiefe erfroren.\n",
    "Aus Asche wird Feuer geschlagen,\n",
    "Aus Schatten geht Licht hervor;\n",
    "Heil wird geborstenes Schwert,\n",
    "Und König, der die Krone verlor.\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "neural-moscow",
   "metadata": {},
   "source": [
    "### Match explicit character(s)\n",
    "In order to match characters explicitly, all you need to do is type what you'd like to find."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "behind-nursery",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['abc']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"abc\", lowercase_alphabet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "recovered-plasma",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"abde\", lowercase_alphabet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "honey-kruger",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cht', 'cht', 'cht', 'cht', 'cht']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"cht\", long_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "tight-nylon",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ABC']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"ABC\", uppercase_alphabet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "usual-louis",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"abc\", uppercase_alphabet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "graphic-intention",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['abc']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"ABC\", lowercase_alphabet, flags=re.IGNORECASE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "widespread-sympathy",
   "metadata": {},
   "source": [
    "### Match with special character(s)\n",
    "In order to match any *special characters `[]\\^$.|?*+()`* you have first introduce a backslash `\\` followed by the character you'd like to select."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "public-wales",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['webmail2016.univie.ac.at']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"webmail2016\\.univie\\.ac\\.at\", website)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "great-award",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['webmail2016.univie.ac.at']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"we.ma.l2016.uni.ie..c.at\", website)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "exact-briefing",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['$']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"\\$\", special_characters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "indian-championship",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"$\", special_characters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "golden-perception",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['|']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"\\|\", special_characters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "accompanied-clothing",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['', '', '', '', '', '', '', '', '', '', '', '']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"|\", special_characters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "iraqi-forwarding",
   "metadata": {},
   "source": [
    "### Match by pattern\n",
    "There are a lot of ways we can match a pattern. Regex has its own syntax so we could pick and choose how we want our patterns to look like."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "otherwise-smoke",
   "metadata": {},
   "source": [
    "### Character Classes\n",
    "| Class | Explanation |\n",
    "|---|---|\n",
    "| . | any character except newline |\n",
    "| \\w \\d \\s | word, digit, whitespace |\n",
    "| \\W \\D \\S | not word, not digit, not whitespace |\n",
    "| [abc] | any of a, b, or c |\n",
    "| [^abc] | not a, b, or c |\n",
    "| [a-g] | characters between a & g |\n",
    "\n",
    "[0-9] is not always equivalent to \\d. \n",
    "In python3, [0-9] matches only 0123456789 characters, while \\d matches [0-9] and other digit characters, for example Eastern Arabic numerals ٠١٢٣٤٥٦٧٨٩."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "public-pressure",
   "metadata": {},
   "source": [
    "### Quantifiers & Alternation\n",
    "| Class | Explanation |\n",
    "|---|---|\n",
    "| a* a+ a? | 0 or more, 1 or more, 0 or 1 |\n",
    "| a{5} a{2,} | exactly five, two or more |\n",
    "| a{1,3} | between one & three |\n",
    "| a+? a{2,}? | match as few as possible |\n",
    "| ab\\|cd | match ab or cd |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "popular-scheme",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The', 'Quick', 'Brown', 'Fox', 'Jumps', 'Over', 'The', 'Lazy', 'Dog']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"\\w{1,}\", sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "signal-authorization",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"\\W+\", sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "greek-stomach",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The', 'e', 'The']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"[The]+\", sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "understanding-orbit",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['he Quick ', 'rown Fox Jumps Over ', 'he ', 'azy Dog']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"[^TLB]{1,}\", sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "painful-andrews",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['he', 'Q', 'ick', 'Bro', 'n', 'Fo', 'J', 'mp', 'O', 'er', 'he', 'La', 'Dog']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"[a-rA-R]+\", sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "sustainable-blond",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ll', 'l', 'l', 'll', 'l', 'l', 'l', 'l', 'l', 'l', 'l', 'l', 'l']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"l+\", long_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "european-hindu",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['l', 'l', 'l', 'l', 'l', 'l', 'l', 'l', 'l', 'l', 'l', 'l', 'l', 'l', 'l']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"l+?\", long_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "noble-michael",
   "metadata": {},
   "source": [
    "### Anchors\n",
    "| Class | Explanation |\n",
    "|---|---|\n",
    "| ^abc$ | start / end of the string |\n",
    "| \\b \\B | character between a & g |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "republican-trouble",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['T']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"^[TheQBFJOLD]\", sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "spiritual-battery",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"^[TheQBFJOLD]+\", sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "wound-revelation",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['verlor.']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"[a-z,\\.]{1,}$\", long_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "executive-bench",
   "metadata": {},
   "source": [
    "### Escaped Characters\n",
    "| Class | Explanation |\n",
    "|---|---|\n",
    "| \\\\. \\\\* \\\\\\ | escaped special characters |\n",
    "| \\\\t \\\\n \\\\r | tab, linefeed, carriage return |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "activated-concert",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['123-456-7890', '234-567-8901', '345-678-9012', '456-789-0123']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"\\d{3}\\-\\d{3}\\-\\d{4}\", phone_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "lonely-popularity",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['123-456-7890',\n",
       " '987.654.321',\n",
       " '234-567-8901',\n",
       " '654.321.987',\n",
       " '345-678-9012',\n",
       " '321.654.978',\n",
       " '456-789-0123']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"\\d{3}[\\.\\-]{1}\\d{3}[\\.\\-]{1}\\d{3,4}\", phone_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "mighty-jackson",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Nicht alles, was Gold ist, funkelt\n",
      "Nicht alles, was Gold ist, funkelt,\n",
      "Nicht jeder, der wandert, verlorn,\n",
      "Das Alte wird nicht verdunkelt,\n",
      "Noch Wurzeln der Tiefe erfroren.\n",
      "Aus Asche wird Feuer geschlagen,\n",
      "Aus Schatten geht Licht hervor;\n",
      "Heil wird geborstenes Schwert,\n",
      "Und König, der die Krone verlor.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(long_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "ordinary-uncle",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\nNicht',\n",
       " '\\nNicht',\n",
       " '\\nNicht',\n",
       " '\\nDas A',\n",
       " '\\nNoch ',\n",
       " '\\nAus A',\n",
       " '\\nAus S',\n",
       " '\\nHeil ',\n",
       " '\\nUnd K']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(\"\\n.{5}\", long_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nonprofit-shield",
   "metadata": {},
   "source": [
    "### Groups & Lookaround\n",
    "| Class | Explanation |\n",
    "|---|---|\n",
    "| (abc) | capture group |\n",
    "| \\1 | backreference to group #1 |\n",
    "| (?:abc) | non-capturing group |\n",
    "| (?=abc) | positive lookahead |\n",
    "| (?!abc) | negative lookahead |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "primary-telescope",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "alice-b@google.com\n",
      "alice-b\n",
      "google.com\n"
     ]
    }
   ],
   "source": [
    "match = re.search('([\\w.-]+)@([\\w\\.\\-]+)', text_with_email)\n",
    "if match:\n",
    "    print(match.group())   ## 'alice-b@google.com' (the whole match)\n",
    "    print(match.group(1))  ## 'alice-b' (the username, group 1)\n",
    "    print(match.group(2))  ## 'google.com' (the host, group 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "increasing-shift",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('alice', 'google.com'), ('bob', 'abc.com')]\n",
      "alice\n",
      "google.com\n",
      "bob\n",
      "abc.com\n"
     ]
    }
   ],
   "source": [
    "my_string = 'purple alice@google.com, blah monkey bob@abc.com blah dishwasher'\n",
    "tuples = re.findall(r'([\\w\\.-]+)@([\\w\\.-]+)', my_string)\n",
    "print(tuples)  ## [('alice', 'google.com'), ('bob', 'abc.com')]\n",
    "for group in tuples:\n",
    "    print(group[0])  ## username\n",
    "    print(group[1])  ## host"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unexpected-automation",
   "metadata": {},
   "source": [
    "### Substituting characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "complete-mining",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is an example\n"
     ]
    }
   ],
   "source": [
    "string = 'This%&$is@an#example'\n",
    "string = re.sub('[%&$@#]+', ' ', string)\n",
    "print(string)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bound-nitrogen",
   "metadata": {},
   "source": [
    "### Regular Expressions with Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "usual-billy",
   "metadata": {},
   "outputs": [],
   "source": [
    "fragebogen = pd.read_csv('../../Data/csv/fragebogen.csv', names=[\"id\",\n",
    "                                                                 \"nummer\", \n",
    "                                                                 \"titel\", \n",
    "                                                                 \"schlagwoerter\", \n",
    "                                                                 \"erscheinungsjahr\",\n",
    "                                                                 \"autoren\", \n",
    "                                                                 \"originaldaten\",\n",
    "                                                                 \"anmerkung\",\n",
    "                                                                 \"freigabe\",\n",
    "                                                                 \"checked\",\n",
    "                                                                 \"wordleiste\",\n",
    "                                                                 \"druck\",\n",
    "                                                                 \"online\",\n",
    "                                                                 \"publiziert\",\n",
    "                                                                 \"fragebogen_typ_id\",])\n",
    "fragebogen.drop([\"id\",\"schlagwoerter\",\"erscheinungsjahr\",\"autoren\",\"originaldaten\",\"anmerkung\",\"freigabe\",\n",
    "                 \"checked\",\"wordleiste\",\"druck\",\"online\",\"publiziert\",\"fragebogen_typ_id\",], inplace=True, axis=1)\n",
    "\n",
    "fragebogen.set_index(\"nummer\", drop=True, inplace=True)\n",
    "fragebogen = fragebogen[fragebogen.titel.str.startswith('Fragebogen')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "daily-strain",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 109 entries, 1 to 109\n",
      "Data columns (total 1 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   titel   109 non-null    object\n",
      "dtypes: object(1)\n",
      "memory usage: 1.7+ KB\n"
     ]
    }
   ],
   "source": [
    "fragebogen.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "executive-professor",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>titel</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nummer</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fragebogen 1: Kopf (1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Fragebogen 2: Die Osterwoche (1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Fragebogen 3: Die Osterwoche (2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fragebogen 4: Kopf (2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Fragebogen 5: Zeit zwischen Ostern und Fronlei...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Fragebogen 6: Menschl. Haar und Bart (= Kopf 3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Fragebogen 7: Hochzeit (1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Fragebogen 8: Hochzeit (2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Fragebogen 9: Hochzeit (3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fragebogen 10: Hochzeit (4)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Fragebogen 11: Hochzeit (5)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Fragebogen 12: Nase, Ohr (Kopf 4)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Fragebogen 13: Auge (Kopf 5)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Fragebogen 14: Besiedelung, Flur (1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Fragebogen 15: Besiedelung, Flur (2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Fragebogen 16: Feld, Feldbestellung (1) (Allg.)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Fragebogen 17: Feld, Feldbestellung (2) (Allg.)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Fragebogen 18: Feld, Feldbestellung (3) (Allg.)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Fragebogen 19: Feld, Feldbestellung (4) (Allg.)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Fragebogen 20: Körper (Allgemeines) (1)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    titel\n",
       "nummer                                                   \n",
       "1                                  Fragebogen 1: Kopf (1)\n",
       "2                        Fragebogen 2: Die Osterwoche (1)\n",
       "3                        Fragebogen 3: Die Osterwoche (2)\n",
       "4                                  Fragebogen 4: Kopf (2)\n",
       "5       Fragebogen 5: Zeit zwischen Ostern und Fronlei...\n",
       "6         Fragebogen 6: Menschl. Haar und Bart (= Kopf 3)\n",
       "7                              Fragebogen 7: Hochzeit (1)\n",
       "8                              Fragebogen 8: Hochzeit (2)\n",
       "9                              Fragebogen 9: Hochzeit (3)\n",
       "10                            Fragebogen 10: Hochzeit (4)\n",
       "11                            Fragebogen 11: Hochzeit (5)\n",
       "12                      Fragebogen 12: Nase, Ohr (Kopf 4)\n",
       "13                           Fragebogen 13: Auge (Kopf 5)\n",
       "14                   Fragebogen 14: Besiedelung, Flur (1)\n",
       "15                   Fragebogen 15: Besiedelung, Flur (2)\n",
       "16        Fragebogen 16: Feld, Feldbestellung (1) (Allg.)\n",
       "17        Fragebogen 17: Feld, Feldbestellung (2) (Allg.)\n",
       "18        Fragebogen 18: Feld, Feldbestellung (3) (Allg.)\n",
       "19        Fragebogen 19: Feld, Feldbestellung (4) (Allg.)\n",
       "20                Fragebogen 20: Körper (Allgemeines) (1)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fragebogen.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prescribed-hybrid",
   "metadata": {},
   "source": [
    "#### Spliting the string in groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "announced-action",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nummer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fragebogen</td>\n",
       "      <td>1</td>\n",
       "      <td>Kopf</td>\n",
       "      <td>(1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Fragebogen</td>\n",
       "      <td>2</td>\n",
       "      <td>Die Osterwoche</td>\n",
       "      <td>(1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Fragebogen</td>\n",
       "      <td>3</td>\n",
       "      <td>Die Osterwoche</td>\n",
       "      <td>(2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fragebogen</td>\n",
       "      <td>4</td>\n",
       "      <td>Kopf</td>\n",
       "      <td>(2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Fragebogen</td>\n",
       "      <td>5</td>\n",
       "      <td>Zeit zwischen Ostern und Fronleichnam</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 0  1                                       2    3\n",
       "nummer                                                            \n",
       "1       Fragebogen  1                                   Kopf   (1)\n",
       "2       Fragebogen  2                         Die Osterwoche   (1)\n",
       "3       Fragebogen  3                         Die Osterwoche   (2)\n",
       "4       Fragebogen  4                                   Kopf   (2)\n",
       "5       Fragebogen  5   Zeit zwischen Ostern und Fronleichnam     "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regex1 = r'([Fragebon]+)\\s{1}([0-9]+):{1}([,\\w0-9.\\s]+)[,\\s]*([=\\-\\(\\)\\s\\w0-9]*)'\n",
    "\n",
    "fragebogen.titel.str.extract(regex1).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "eastern-interface",
   "metadata": {},
   "outputs": [],
   "source": [
    "fragebogen.loc[:,'fragebogen_num'] = fragebogen.titel.str.extract(regex1)[1].str.strip()\n",
    "fragebogen.loc[:,'fragebogen_headwords'] = fragebogen.titel.str.extract(regex1)[2].str.strip()\n",
    "fragebogen.loc[:,'fragebogen_series'] = fragebogen.titel.str.extract(regex1)[3].str.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "graduate-danger",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>titel</th>\n",
       "      <th>fragebogen_num</th>\n",
       "      <th>fragebogen_headwords</th>\n",
       "      <th>fragebogen_series</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nummer</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fragebogen 1: Kopf (1)</td>\n",
       "      <td>1</td>\n",
       "      <td>Kopf</td>\n",
       "      <td>(1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Fragebogen 2: Die Osterwoche (1)</td>\n",
       "      <td>2</td>\n",
       "      <td>Die Osterwoche</td>\n",
       "      <td>(1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Fragebogen 3: Die Osterwoche (2)</td>\n",
       "      <td>3</td>\n",
       "      <td>Die Osterwoche</td>\n",
       "      <td>(2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fragebogen 4: Kopf (2)</td>\n",
       "      <td>4</td>\n",
       "      <td>Kopf</td>\n",
       "      <td>(2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Fragebogen 5: Zeit zwischen Ostern und Fronlei...</td>\n",
       "      <td>5</td>\n",
       "      <td>Zeit zwischen Ostern und Fronleichnam</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Fragebogen 6: Menschl. Haar und Bart (= Kopf 3)</td>\n",
       "      <td>6</td>\n",
       "      <td>Menschl. Haar und Bart</td>\n",
       "      <td>(= Kopf 3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Fragebogen 7: Hochzeit (1)</td>\n",
       "      <td>7</td>\n",
       "      <td>Hochzeit</td>\n",
       "      <td>(1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Fragebogen 8: Hochzeit (2)</td>\n",
       "      <td>8</td>\n",
       "      <td>Hochzeit</td>\n",
       "      <td>(2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Fragebogen 9: Hochzeit (3)</td>\n",
       "      <td>9</td>\n",
       "      <td>Hochzeit</td>\n",
       "      <td>(3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fragebogen 10: Hochzeit (4)</td>\n",
       "      <td>10</td>\n",
       "      <td>Hochzeit</td>\n",
       "      <td>(4)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Fragebogen 11: Hochzeit (5)</td>\n",
       "      <td>11</td>\n",
       "      <td>Hochzeit</td>\n",
       "      <td>(5)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Fragebogen 12: Nase, Ohr (Kopf 4)</td>\n",
       "      <td>12</td>\n",
       "      <td>Nase, Ohr</td>\n",
       "      <td>(Kopf 4)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Fragebogen 13: Auge (Kopf 5)</td>\n",
       "      <td>13</td>\n",
       "      <td>Auge</td>\n",
       "      <td>(Kopf 5)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Fragebogen 14: Besiedelung, Flur (1)</td>\n",
       "      <td>14</td>\n",
       "      <td>Besiedelung, Flur</td>\n",
       "      <td>(1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Fragebogen 15: Besiedelung, Flur (2)</td>\n",
       "      <td>15</td>\n",
       "      <td>Besiedelung, Flur</td>\n",
       "      <td>(2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Fragebogen 16: Feld, Feldbestellung (1) (Allg.)</td>\n",
       "      <td>16</td>\n",
       "      <td>Feld, Feldbestellung</td>\n",
       "      <td>(1) (Allg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Fragebogen 17: Feld, Feldbestellung (2) (Allg.)</td>\n",
       "      <td>17</td>\n",
       "      <td>Feld, Feldbestellung</td>\n",
       "      <td>(2) (Allg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Fragebogen 18: Feld, Feldbestellung (3) (Allg.)</td>\n",
       "      <td>18</td>\n",
       "      <td>Feld, Feldbestellung</td>\n",
       "      <td>(3) (Allg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Fragebogen 19: Feld, Feldbestellung (4) (Allg.)</td>\n",
       "      <td>19</td>\n",
       "      <td>Feld, Feldbestellung</td>\n",
       "      <td>(4) (Allg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Fragebogen 20: Körper (Allgemeines) (1)</td>\n",
       "      <td>20</td>\n",
       "      <td>Körper</td>\n",
       "      <td>(Allgemeines) (1)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    titel fragebogen_num  \\\n",
       "nummer                                                                     \n",
       "1                                  Fragebogen 1: Kopf (1)              1   \n",
       "2                        Fragebogen 2: Die Osterwoche (1)              2   \n",
       "3                        Fragebogen 3: Die Osterwoche (2)              3   \n",
       "4                                  Fragebogen 4: Kopf (2)              4   \n",
       "5       Fragebogen 5: Zeit zwischen Ostern und Fronlei...              5   \n",
       "6         Fragebogen 6: Menschl. Haar und Bart (= Kopf 3)              6   \n",
       "7                              Fragebogen 7: Hochzeit (1)              7   \n",
       "8                              Fragebogen 8: Hochzeit (2)              8   \n",
       "9                              Fragebogen 9: Hochzeit (3)              9   \n",
       "10                            Fragebogen 10: Hochzeit (4)             10   \n",
       "11                            Fragebogen 11: Hochzeit (5)             11   \n",
       "12                      Fragebogen 12: Nase, Ohr (Kopf 4)             12   \n",
       "13                           Fragebogen 13: Auge (Kopf 5)             13   \n",
       "14                   Fragebogen 14: Besiedelung, Flur (1)             14   \n",
       "15                   Fragebogen 15: Besiedelung, Flur (2)             15   \n",
       "16        Fragebogen 16: Feld, Feldbestellung (1) (Allg.)             16   \n",
       "17        Fragebogen 17: Feld, Feldbestellung (2) (Allg.)             17   \n",
       "18        Fragebogen 18: Feld, Feldbestellung (3) (Allg.)             18   \n",
       "19        Fragebogen 19: Feld, Feldbestellung (4) (Allg.)             19   \n",
       "20                Fragebogen 20: Körper (Allgemeines) (1)             20   \n",
       "\n",
       "                         fragebogen_headwords  fragebogen_series  \n",
       "nummer                                                            \n",
       "1                                        Kopf                (1)  \n",
       "2                              Die Osterwoche                (1)  \n",
       "3                              Die Osterwoche                (2)  \n",
       "4                                        Kopf                (2)  \n",
       "5       Zeit zwischen Ostern und Fronleichnam                     \n",
       "6                      Menschl. Haar und Bart         (= Kopf 3)  \n",
       "7                                    Hochzeit                (1)  \n",
       "8                                    Hochzeit                (2)  \n",
       "9                                    Hochzeit                (3)  \n",
       "10                                   Hochzeit                (4)  \n",
       "11                                   Hochzeit                (5)  \n",
       "12                                  Nase, Ohr           (Kopf 4)  \n",
       "13                                       Auge           (Kopf 5)  \n",
       "14                          Besiedelung, Flur                (1)  \n",
       "15                          Besiedelung, Flur                (2)  \n",
       "16                       Feld, Feldbestellung          (1) (Allg  \n",
       "17                       Feld, Feldbestellung          (2) (Allg  \n",
       "18                       Feld, Feldbestellung          (3) (Allg  \n",
       "19                       Feld, Feldbestellung          (4) (Allg  \n",
       "20                                     Körper  (Allgemeines) (1)  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fragebogen.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mechanical-witch",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sharp-cassette",
   "metadata": {},
   "source": [
    "### Twitter:\n",
    "    \n",
    "#### In twitter a username is validated if:\n",
    "\n",
    "+ Is a text with a ‘@’ at the beginning so in regex ‘^@’\n",
    "+ Contains only alphanumerical characters (letters A-Z, numbers 0–9). \n",
    "    + In this case we can use either ‘[A-Za-z0–9]’ or ‘[\\w\\d]’. \n",
    "    + We use the brackets because we want to apply another quantifiers to the set.\n",
    "+ It must be between 4 and 15 characters, so {4,15}.\n",
    "\n",
    "You can use the re.match() or re.search() methods to find a match for your string, in the following code is reported the selection throught the regex statement."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "radio-uniform",
   "metadata": {},
   "source": [
    "#### Let's find some twitter databases:\n",
    "\n",
    "[First GOP debate](https://www.kaggle.com/crowdflower/first-gop-debate-twitter-sentiment)  \n",
    "[US Elections 2020](https://www.kaggle.com/manchunhui/us-election-2020-tweets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "american-retirement",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sqlite3\n",
    "conn = sqlite3.connect('../../Data/sqlite/tweet_database.sqlite')\n",
    "cur = conn.cursor()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "limited-occurrence",
   "metadata": {},
   "source": [
    "#### Checking the tables in database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "animal-insurance",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Sentiment',)]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q1 = \"SELECT name FROM sqlite_master WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%'\"\n",
    "cur.execute(q1)\n",
    "cur.fetchall()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dressed-toolbox",
   "metadata": {},
   "source": [
    "#### Checking columns in table \"sentiments\" and querying the first row from the \"sentiments\" table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "decent-provincial",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlite3.Cursor at 0x7fe69ec4ce40>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q2 = \"SELECT * from Sentiment limit 1\"\n",
    "cur.execute(q2)\n",
    "#print([f[0] for f in cur.description])\n",
    "#cur.fetchall()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "included-tradition",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id:\t1\n",
      "\n",
      "candidate:\tNo candidate mentioned\n",
      "\n",
      "candidate_confidence:\t1\n",
      "\n",
      "relevant_yn:\tyes\n",
      "\n",
      "relevant_yn_confidence:\t1\n",
      "\n",
      "sentiment:\tNeutral\n",
      "\n",
      "sentiment_confidence:\t0.6578\n",
      "\n",
      "subject_matter:\tNone of the above\n",
      "\n",
      "subject_matter_confidence:\t1\n",
      "\n",
      "candidate_gold:\t\n",
      "\n",
      "name:\tI_Am_Kenzi\n",
      "\n",
      "relevant_yn_gold:\t\n",
      "\n",
      "retweet_count:\t5\n",
      "\n",
      "sentiment_gold:\t\n",
      "\n",
      "subject_matter_gold:\t\n",
      "\n",
      "text:\tRT @NancyLeeGrahn: How did everyone feel about the Climate Change question last night? Exactly. #GOPDebate\n",
      "\n",
      "tweet_coord:\t\n",
      "\n",
      "tweet_created:\t2015-08-07 09:54:46 -0700\n",
      "\n",
      "tweet_id:\t629697200650592256\n",
      "\n",
      "tweet_location:\t\n",
      "\n",
      "user_timezone:\tQuito\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for idx, row in enumerate(cur.fetchall()[0]):\n",
    "    print(f\"{cur.description[idx][0]}:\\t{row}\")\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interim-dryer",
   "metadata": {},
   "source": [
    "#### Retrieving first 50 rows from field sentiment.text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "stable-pride",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('RT @NancyLeeGrahn: How did everyone feel about the Climate Change question last night? Exactly. #GOPDebate',),\n",
       " (\"RT @ScottWalker: Didn't catch the full #GOPdebate last night. Here are some of Scott's best lines in 90 seconds. #Walker16 http://t.co/ZSfF…\",),\n",
       " ('RT @TJMShow: No mention of Tamir Rice and the #GOPDebate was held in Cleveland? Wow.',)]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q3 = \"SELECT text from Sentiment limit 50\"\n",
    "cur.execute(q3)\n",
    "cur.fetchmany(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "worthy-seeker",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['RT @NancyLeeGrahn: How did everyone feel about the Climate Change question last night? Exactly. #GOPDebate',\n",
       " \"RT @ScottWalker: Didn't catch the full #GOPdebate last night. Here are some of Scott's best lines in 90 seconds. #Walker16 http://t.co/ZSfF…\",\n",
       " 'RT @TJMShow: No mention of Tamir Rice and the #GOPDebate was held in Cleveland? Wow.',\n",
       " \"RT @RobGeorge: That Carly Fiorina is trending -- hours after HER debate -- above any of the men in just-completed #GOPdebate says she's on …\",\n",
       " 'RT @DanScavino: #GOPDebate w/ @realDonaldTrump delivered the highest ratings in the history of presidential debates. #Trump2016 http://t.co…',\n",
       " 'RT @GregAbbott_TX: @TedCruz: \"On my first day I will rescind every illegal executive action taken by Barack Obama.\" #GOPDebate @FoxNews',\n",
       " 'RT @warriorwoman91: I liked her and was happy when I heard she was going to be the moderator. Not anymore. #GOPDebate @megynkelly  https://…',\n",
       " 'Going on #MSNBC Live with @ThomasARoberts around 2 PM ET.  #GOPDebate',\n",
       " 'Deer in the headlights RT @lizzwinstead: Ben Carson, may be the only brain surgeon who has performed a lobotomy on himself. #GOPDebate',\n",
       " \"RT @NancyOsborne180: Last night's debate proved it! #GOPDebate #BATsAsk @BadassTeachersA #TBATs  https://t.co/G2gGjY1bJD\",\n",
       " '@JGreenDC @realDonaldTrump In all fairness #BillClinton owns that phrase.#GOPDebate',\n",
       " 'RT @WayneDupreeShow: Just woke up to tweet this out #GOPDebate \\n\\nBest line of the night via @GovMikeHuckabee http://t.co/6OV5hxHIcV',\n",
       " \"Me reading my family's comments about how great the #GOPDebate was http://t.co/gIaGjPygXZ\",\n",
       " 'RT @ArcticFox2016: RT @AllenWestRepub \"Dear @JebBush #GOPDebate #NotAMistake http://t.co/TtFG7KYcd9\"',\n",
       " \"RT @pattonoswalt: I loved Scott Walker as Mark Harmon's romantic rival in SUMMER SCHOOL. Look it up. #GOPDebate\",\n",
       " \"Hey @ChrisChristie exploiting the tragedy of 9/11 for your own political gain is @rudygiulianiGOP's thing #GOPDebate\",\n",
       " 'RT @CarolCNN: #DonaldTrump under fire for comments about women @PeterBeinart @SL_Schaeffer @IWF @@MyRkiger weigh in on #GOPdebate http://t.…',\n",
       " 'RT @johncardillo: Guess who had most speaking time at the #GOPDebate. @FoxNews moderators with 31.7% of time. http://t.co/2WSUT0c0Lx',\n",
       " \"reason comment is funny 'in case you're ignorant' is the #gop #tcot are the reason the government isn't working for the people #gopdebate\",\n",
       " 'RT @PamelaGeller: Huckabee: Paying for transgender surgery for soldiers, sailors and airmen does not make our country safer #Ha #GOPDebate',\n",
       " 'RT @ChuckNellis: Cruz has class &amp; truth, that gets my vote! #GOPDebate',\n",
       " \"RT @mchamric: RT “@TeaTraitors: #GOPDebate was still Clown Show! I'm glad Head Clown Trump helping destroy GOP. http://t.co/nwGx8G8JWr”\",\n",
       " 'RT @erinmallorylong: No *I* hate Planned Parenthood and women more! NO I HATE PLANNED PARENTHOOD AND WOMEN MORE!!!!! #GOPDebate',\n",
       " 'RT @thekevinryder: #GOPDebate (Vine by @dabulldawg88) https://t.co/XKxVDrADce',\n",
       " 'RT @MrPooni: Fox News trying to convince us young Black Americans are more worried about ISIS than police terrorism #GOPDebate http://t.co/…',\n",
       " '#GOPDebate rankings: worst to be performance - Rubio, Kasich, Christie, Bush, Trump, Cruz, Walker, Paul, Huckabee, Carson.',\n",
       " \"RT @TheBaxterBean: Scott Walker's Abortion Ban Allows Rapist Father To Sue For Emotional Distress http://t.co/rHMvgumuir #GOPDebate @Badger…\",\n",
       " \"RT @feministabulous: It's not a competition, but how have we moved so far on gay rights but women's rights is worse than it's ever been? #G…\",\n",
       " \"RT @mch7576: RT “@TeaTraitors: #GOPDebate was still Clown Show! I'm glad Head Clown Trump helping destroy GOP. http://t.co/pRy2QPCWfu””\",\n",
       " 'RT @HillaryClinton: Watch the #GOPdebate? Bet you feel like donating to a Democrat right about now. http://t.co/pGlQCqQgOP http://t.co/QP1e…',\n",
       " \"@fbhw they're going to need to borrow the train horn for the next #GOPDebate ! The little bell was wimpy! #hornmeansyouredone\",\n",
       " 'RT @LisaVikingstad: Ted Cruz at the #GOPDebate will be like Ted Cruz in bed. He will keep a confusingly sad look on his face and refuse to …',\n",
       " 'The First #GOPDebate: Social Media Reaction and More http://t.co/X6KUVSkltF',\n",
       " '#SocialMedia: The First #GOPDebate: Social Media Reaction and More http://t.co/ogoYfNyiKj',\n",
       " 'RT WinesdayPodcast : An update on FoxNews tech failures for the #GOPDebate http://t.co/FGGlf2J5HU http://t.co/Zw35DkUozi',\n",
       " 'RT @AmyMek: The Torched has been passed -&gt;.@CandyCrowley is no longer the most Famous Political Assassin! Congratulations @megynkelly, U Wi…',\n",
       " 'RT @stockwizards3: Trump/Carson ticket would win in a landslide...\\n@realDonaldTrump @seanhannity @AnnCoulter #Republican #potus #Trump2016 …',\n",
       " 'RT @kvxrdashian: Jeb Bush: \"Obama is at fault, not my brother, because Obama didn\\'t clean up the mess my brother made.\" #GOPDebate http://t…',\n",
       " 'RT @ali: After the Fox News #GOPDebate, we primary voters are none the wiser on the policy plans these folks are putting forth. Journalism …',\n",
       " \"RT @itsashlyperez: listening to fetty wap while packing because i'm a true american #GOPDebate #doesthishashtagexpire?\",\n",
       " 'RT @mgell: Trump is a cross between the shrug emoji and President Business from the Lego movie. #GOPDebate',\n",
       " 'RT @JamelleMyBelle: Meanwhile, in the White House...  #GOPDebate http://t.co/nouUUt5hKq',\n",
       " 'RT @RowdyGentleman: If you’re planning out a #GOPDebate drinking game right now, you’re a Rowdy Gentleman.',\n",
       " 'RT @brownblaze: PLEASE RT. #KKKorGOP #GOPDebate http://t.co/YaqYAC3uV3',\n",
       " 'Trump thinks criticism of his misogynistic remarks is a \"political correctness\" problem. Nope, it\\'s a pervasive #sexism problem. #GOPDebate',\n",
       " 'RT @LeKarmaSucre: How the #GOPDebate handled #BlackLivesMatter \\n\\nhttp://t.co/8ZGdY6lPCT',\n",
       " 'RT @palmaceiahome1: The goal of #FoxNews last night was to take out Trump and pave the way for a RINO like Bush or Kasich. #GOPDebate',\n",
       " 'RT @philstockworld: From our Live Chat Room: #GOPDebate #Trump #Futures $SPY #NonFarmPayrolls #Jobs #Netflix -- http://t.co/0K06Sf81rq http…',\n",
       " '“@DanScavino: #GOPDebate w/ @realDonaldTrump delivered the highest ratings in the history of presidential debates. #Trump2016 Fox say thanks',\n",
       " 'RT @gov: Most-Tweeted @GovMikeHuckabee #GOPDebate moment: saying purpose of military \"to kill people/break things\" re trans servicemembers']"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = cur.execute(q3).fetchall()\n",
    "tweets = [t[0] for t in tweets]\n",
    "tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "southwest-cradle",
   "metadata": {},
   "source": [
    "### Or you may choose using Pandas for querying"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "generous-laugh",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>candidate</th>\n",
       "      <th>candidate_confidence</th>\n",
       "      <th>relevant_yn</th>\n",
       "      <th>relevant_yn_confidence</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>sentiment_confidence</th>\n",
       "      <th>subject_matter</th>\n",
       "      <th>subject_matter_confidence</th>\n",
       "      <th>candidate_gold</th>\n",
       "      <th>...</th>\n",
       "      <th>relevant_yn_gold</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>sentiment_gold</th>\n",
       "      <th>subject_matter_gold</th>\n",
       "      <th>text</th>\n",
       "      <th>tweet_coord</th>\n",
       "      <th>tweet_created</th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>tweet_location</th>\n",
       "      <th>user_timezone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>No candidate mentioned</td>\n",
       "      <td>1.0</td>\n",
       "      <td>yes</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>0.6578</td>\n",
       "      <td>None of the above</td>\n",
       "      <td>1.0000</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>5</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>RT @NancyLeeGrahn: How did everyone feel about...</td>\n",
       "      <td></td>\n",
       "      <td>2015-08-07 09:54:46 -0700</td>\n",
       "      <td>629697200650592256</td>\n",
       "      <td></td>\n",
       "      <td>Quito</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Scott Walker</td>\n",
       "      <td>1.0</td>\n",
       "      <td>yes</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Positive</td>\n",
       "      <td>0.6333</td>\n",
       "      <td>None of the above</td>\n",
       "      <td>1.0000</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>26</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>RT @ScottWalker: Didn't catch the full #GOPdeb...</td>\n",
       "      <td></td>\n",
       "      <td>2015-08-07 09:54:46 -0700</td>\n",
       "      <td>629697199560069120</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>No candidate mentioned</td>\n",
       "      <td>1.0</td>\n",
       "      <td>yes</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>0.6629</td>\n",
       "      <td>None of the above</td>\n",
       "      <td>0.6629</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>27</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>RT @TJMShow: No mention of Tamir Rice and the ...</td>\n",
       "      <td></td>\n",
       "      <td>2015-08-07 09:54:46 -0700</td>\n",
       "      <td>629697199312482304</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>No candidate mentioned</td>\n",
       "      <td>1.0</td>\n",
       "      <td>yes</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Positive</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>None of the above</td>\n",
       "      <td>0.7039</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>138</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>RT @RobGeorge: That Carly Fiorina is trending ...</td>\n",
       "      <td></td>\n",
       "      <td>2015-08-07 09:54:45 -0700</td>\n",
       "      <td>629697197118861312</td>\n",
       "      <td>Texas</td>\n",
       "      <td>Central Time (US &amp; Canada)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Donald Trump</td>\n",
       "      <td>1.0</td>\n",
       "      <td>yes</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Positive</td>\n",
       "      <td>0.7045</td>\n",
       "      <td>None of the above</td>\n",
       "      <td>1.0000</td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>156</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>RT @DanScavino: #GOPDebate w/ @realDonaldTrump...</td>\n",
       "      <td></td>\n",
       "      <td>2015-08-07 09:54:45 -0700</td>\n",
       "      <td>629697196967903232</td>\n",
       "      <td></td>\n",
       "      <td>Arizona</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id               candidate  candidate_confidence relevant_yn  \\\n",
       "0   1  No candidate mentioned                   1.0         yes   \n",
       "1   2            Scott Walker                   1.0         yes   \n",
       "2   3  No candidate mentioned                   1.0         yes   \n",
       "3   4  No candidate mentioned                   1.0         yes   \n",
       "4   5            Donald Trump                   1.0         yes   \n",
       "\n",
       "   relevant_yn_confidence sentiment  sentiment_confidence     subject_matter  \\\n",
       "0                     1.0   Neutral                0.6578  None of the above   \n",
       "1                     1.0  Positive                0.6333  None of the above   \n",
       "2                     1.0   Neutral                0.6629  None of the above   \n",
       "3                     1.0  Positive                1.0000  None of the above   \n",
       "4                     1.0  Positive                0.7045  None of the above   \n",
       "\n",
       "   subject_matter_confidence candidate_gold  ... relevant_yn_gold  \\\n",
       "0                     1.0000                 ...                    \n",
       "1                     1.0000                 ...                    \n",
       "2                     0.6629                 ...                    \n",
       "3                     0.7039                 ...                    \n",
       "4                     1.0000                 ...                    \n",
       "\n",
       "  retweet_count  sentiment_gold subject_matter_gold  \\\n",
       "0             5                                       \n",
       "1            26                                       \n",
       "2            27                                       \n",
       "3           138                                       \n",
       "4           156                                       \n",
       "\n",
       "                                                text tweet_coord  \\\n",
       "0  RT @NancyLeeGrahn: How did everyone feel about...               \n",
       "1  RT @ScottWalker: Didn't catch the full #GOPdeb...               \n",
       "2  RT @TJMShow: No mention of Tamir Rice and the ...               \n",
       "3  RT @RobGeorge: That Carly Fiorina is trending ...               \n",
       "4  RT @DanScavino: #GOPDebate w/ @realDonaldTrump...               \n",
       "\n",
       "               tweet_created            tweet_id  tweet_location  \\\n",
       "0  2015-08-07 09:54:46 -0700  629697200650592256                   \n",
       "1  2015-08-07 09:54:46 -0700  629697199560069120                   \n",
       "2  2015-08-07 09:54:46 -0700  629697199312482304                   \n",
       "3  2015-08-07 09:54:45 -0700  629697197118861312           Texas   \n",
       "4  2015-08-07 09:54:45 -0700  629697196967903232                   \n",
       "\n",
       "                user_timezone  \n",
       "0                       Quito  \n",
       "1                              \n",
       "2                              \n",
       "3  Central Time (US & Canada)  \n",
       "4                     Arizona  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q4 = \"SELECT * from Sentiment limit 50\"\n",
    "df = pd.read_sql_query(q4, conn)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "associate-means",
   "metadata": {},
   "source": [
    "### Now it is you time!\n",
    "\n",
    "#### Extract from tweets:\n",
    "+ Mentions\n",
    "+ URLs\n",
    "+ Hashtags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "wireless-tender",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['@NancyLeeGrahn']\n",
      "['@ScottWalker']\n",
      "['@TJMShow']\n",
      "['@RobGeorge']\n",
      "['@DanScavino', '@realDonaldTrump']\n",
      "['@GregAbbott_TX', '@TedCruz', '@FoxNews']\n",
      "['@warriorwoman91', '@megynkelly']\n",
      "['@ThomasARoberts']\n",
      "['@lizzwinstead']\n",
      "['@NancyOsborne180', '@BadassTeachersA']\n",
      "['@JGreenDC', '@realDonaldTrump']\n",
      "['@WayneDupreeShow', '@GovMikeHuckabee']\n",
      "['@ArcticFox2016', '@AllenWestRepub', '@JebBush']\n",
      "['@pattonoswalt']\n",
      "['@ChrisChristie', '@rudygiulianiGOP']\n",
      "['@CarolCNN', '@PeterBeinart', '@SL_Schaeffer', '@IWF', '@MyRkiger']\n",
      "['@johncardillo', '@FoxNews']\n",
      "['@PamelaGeller']\n",
      "['@ChuckNellis']\n",
      "['@mchamric', '@TeaTraitors']\n",
      "['@erinmallorylong']\n",
      "['@thekevinryder', '@dabulldawg88']\n",
      "['@MrPooni']\n",
      "['@TheBaxterBean', '@Badger']\n",
      "['@feministabulous']\n",
      "['@mch7576', '@TeaTraitors']\n",
      "['@HillaryClinton']\n",
      "['@fbhw']\n",
      "['@LisaVikingstad']\n",
      "['@AmyMek', '@CandyCrowley', '@megynkelly']\n",
      "['@stockwizards3', '@realDonaldTrump', '@seanhannity', '@AnnCoulter']\n",
      "['@kvxrdashian']\n",
      "['@ali']\n",
      "['@itsashlyperez']\n",
      "['@mgell']\n",
      "['@JamelleMyBelle']\n",
      "['@RowdyGentleman']\n",
      "['@brownblaze']\n",
      "['@LeKarmaSucre']\n",
      "['@palmaceiahome1']\n",
      "['@philstockworld']\n",
      "['@DanScavino', '@realDonaldTrump']\n",
      "['@gov', '@GovMikeHuckabee']\n"
     ]
    }
   ],
   "source": [
    "for tweet in tweets:\n",
    "    match = re.findall('@\\w+', tweet)\n",
    "    if match:\n",
    "        print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "demographic-pride",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['http://t.co/ZSfF']\n",
      "['https://t.co/G2gGjY1bJD']\n",
      "['http://t.co/6OV5hxHIcV']\n",
      "['http://t.co/gIaGjPygXZ']\n",
      "['http://t.co/TtFG7KYcd9']\n",
      "['http://t.co/2WSUT0c0Lx']\n",
      "['http://t.co/nwGx8G8JWr']\n",
      "['https://t.co/XKxVDrADce']\n",
      "['http://t.co/rHMvgumuir']\n",
      "['http://t.co/pRy2QPCWfu']\n",
      "['http://t.co/pGlQCqQgOP', 'http://t.co/QP1e']\n",
      "['http://t.co/X6KUVSkltF']\n",
      "['http://t.co/ogoYfNyiKj']\n",
      "['http://t.co/FGGlf2J5HU', 'http://t.co/Zw35DkUozi']\n",
      "['http://t.co/nouUUt5hKq']\n",
      "['http://t.co/YaqYAC3uV3']\n",
      "['http://t.co/8ZGdY6lPCT']\n",
      "['http://t.co/0K06Sf81rq']\n"
     ]
    }
   ],
   "source": [
    "for tweet in tweets:\n",
    "    match = re.findall('http+s?:/{2}\\w{1,}\\.\\w{1,}/.\\w{1,}', tweet)\n",
    "    if match:\n",
    "        print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "ignored-wedding",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['#GOPDebate']\n",
      "['#GOPdebate', '#Walker16']\n",
      "['#GOPDebate']\n",
      "['#GOPdebate']\n",
      "['#GOPDebate', '#Trump2016']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#MSNBC', '#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate', '#BATsAsk', '#TBATs']\n",
      "['#BillClinton', '#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate', '#NotAMistake']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#DonaldTrump', '#GOPdebate']\n",
      "['#GOPDebate']\n",
      "['#gop', '#tcot', '#gopdebate']\n",
      "['#Ha', '#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#G']\n",
      "['#GOPDebate']\n",
      "['#GOPdebate']\n",
      "['#GOPDebate', '#hornmeansyouredone']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#SocialMedia', '#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#Republican', '#potus', '#Trump2016']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate', '#doesthishashtagexpire']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#GOPDebate']\n",
      "['#KKKorGOP', '#GOPDebate']\n",
      "['#sexism', '#GOPDebate']\n",
      "['#GOPDebate', '#BlackLivesMatter']\n",
      "['#FoxNews', '#GOPDebate']\n",
      "['#GOPDebate', '#Trump', '#Futures', '#NonFarmPayrolls', '#Jobs', '#Netflix']\n",
      "['#GOPDebate', '#Trump2016']\n",
      "['#GOPDebate']\n"
     ]
    }
   ],
   "source": [
    "for tweet in tweets:\n",
    "    match = re.findall('\\#\\w+', tweet)\n",
    "    if match:\n",
    "        print(match)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "seeing-literature",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}