Exercises/Exercise_6_String_Manipulation_Solutions.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_7l_9ZgxdMUp"
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import re\n",
    "from bs4 import BeautifulSoup\n",
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CMwe5lqUdMUu"
   },
   "outputs": [],
   "source": [
    "page = requests.get(\"http://www.vatican.va/archive/bible/genesis/documents/bible_genesis_en.html\")\n",
    "soup = BeautifulSoup(page.content, 'html.parser')\n",
    "\n",
    "for script in soup([\"script\", \"style\"]):\n",
    "    script.decompose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uVNnr4S6tkvU"
   },
   "outputs": [],
   "source": [
    "genesis = soup.get_text()\n",
    "genesis = re.sub(r'[\\n]+','\\n', genesis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8QYlso3VpUke"
   },
   "outputs": [],
   "source": [
    "# Removing undesirable characters\n",
    "for symbol in string.punctuation:\n",
    "    genesis = genesis.replace(symbol,' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Questions:\n",
    "\n",
    "\n",
    "#### 1. How many times the word 'God' appear (as an isolated word)?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GmsIFpYJssJN"
   },
   "source": [
    "**for** loops and **if** statements are required to answer the questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "gLoRChE4fmS6",
    "outputId": "c440a959-66ef-4d20-f5c1-78c0bd54cc8d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "227"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "genesis.count('God')   # wrong way! We are not counting 'God' as an isolated word here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "iDNhKUJ_eQYj",
    "outputId": "ae0f4e9f-5ac8-49ef-fe67-c0554145e7b5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "227"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "genesis.split().count('God')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ahVinp2GsHqt"
   },
   "source": [
    "Coincidentally, there are no words that contain 'God'. Ex: Goddess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. What are the 5 most common words?    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lL4mqhHAuJ_q"
   },
   "source": [
    "To accomplish this task we need to convert all uppercase characters into lowercase characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DywmtTL8uROS"
   },
   "outputs": [],
   "source": [
    "genesis = genesis.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GfZ6FdqS0k0m"
   },
   "source": [
    "**word_list**: list that contains every word of the string (no duplicates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vAvw-MbC0MbS"
   },
   "outputs": [],
   "source": [
    "word_list = [] \n",
    "# using the data structure 'set' to remove duplicates\n",
    "for isolated_string in list(set(genesis.split())):\n",
    "# ignoring numeric strings  \n",
    "    if isolated_string.isalpha():   \n",
    "        word_list.append(isolated_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r8lg6Ev709dL"
   },
   "source": [
    "**count_list**: list that contains the frequency of every element of **word_list**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "No_hmhR60NNt"
   },
   "outputs": [],
   "source": [
    "count_list = []\n",
    "for word in word_list:\n",
    "    count_list.append(genesis.split().count(word))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ugOQM9x8voI3"
   },
   "source": [
    "For the same index, we have the word and the associated frequency. The natural data structure here would be dictionaries or dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "ZmbxJn3ruvxb",
    "outputId": "3469e69e-6a3b-4715-e541-8506c6fbd03c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(count_list) == len(word_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f_JiuJIYvNwC"
   },
   "source": [
    "Making a copy in order to create an ordered version of **count_list**. Bearing in mind that `list.sort` is an inplace method, i.e., our **count_list**'s order would be modified and we want to preserve it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5juPkI_PBjWp"
   },
   "outputs": [],
   "source": [
    "sorted_count_list = count_list.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SMzdCM224xh9"
   },
   "outputs": [],
   "source": [
    "sorted_count_list.sort(reverse = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "y9Ro0Hoiv0Dc"
   },
   "outputs": [],
   "source": [
    "list_5_freq = sorted_count_list[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "FUemI2eKwPe1",
    "outputId": "7827aef7-96d2-4e89-b442-e5d53d9121c3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2475, 2018, 1271, 1078, 650]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_5_freq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 101
    },
    "colab_type": "code",
    "id": "7Hdf28hf1N8k",
    "outputId": "17033b55-c6bc-4443-d729-8401b6fc1155"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the\n",
      "and\n",
      "of\n",
      "to\n",
      "you\n"
     ]
    }
   ],
   "source": [
    "#Let's find the index in count_list to find the index in word_list\n",
    "\n",
    "for i in range(5):\n",
    "    word_index = count_list.index(list_5_freq[i])\n",
    "    print(word_list[word_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using dictionaries, filter and lambda functions in questions 2 and 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_dict = dict() \n",
    "# using the data structure 'set' to remove duplicates\n",
    "for isolated_string in list(set(genesis.split())):\n",
    "# ignoring numeric strings  \n",
    "    if isolated_string.isalpha():   \n",
    "        word_dict[isolated_string] = genesis.split().count(isolated_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sorting the dictionary by using an lambda function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 2475), ('and', 2018), ('of', 1271), ('to', 1078), ('you', 650)]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(word_dict.items(), key = lambda x:x[1], reverse=True)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. What are the words that appear only once ([hapaxes](https://en.wikipedia.org/wiki/Hapax_legomenon))?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q8Sqw_z5z-QI"
   },
   "source": [
    "Let's create **list_one_time** to receive the **hapaxes**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uq2ASnXuy0z-"
   },
   "outputs": [],
   "source": [
    "list_one_time = []\n",
    "for freq,word in zip(count_list,word_list):\n",
    "    if freq == 1:\n",
    "        list_one_time.append(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Xp7KFmDg0iTC"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['vindicated',\n",
       " 'fourteenth',\n",
       " 'sustained',\n",
       " 'displeasing',\n",
       " 'chariots',\n",
       " 'confused',\n",
       " 'belongs',\n",
       " 'stands',\n",
       " 'buz',\n",
       " 'banks']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_one_time[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Oza1oOGw1TpC",
    "outputId": "0e710064-aa09-4b2f-c2a3-6576f103d0c8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1106"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(list_one_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UZYuVi1FC5HO"
   },
   "source": [
    "Double check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GIhd5Eq_CS1-"
   },
   "outputs": [],
   "source": [
    "for word in list_one_time:\n",
    "    counting = genesis.split().count(word)\n",
    "    if counting > 1:\n",
    "        print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the built-in function **filter**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_one_time = dict(filter(lambda x:x[1]==1, word_dict.items())).keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1106"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dict_one_time)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "1_Exercise_Pedro.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}