Assignments/EN/Assignment_5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://communications.univie.ac.at/fileadmin/_processed_/csm_Uni_Logo_2016_2f47aacf37.jpg\" \n",
    "     alt=\"Logo Universität Wien\" \n",
    "     width=\"200\"/>\n",
    "\n",
    "# Practical Machine Learning for Natural Language Processing - 2023 SS  \n",
    "\n",
    "### Assigment 1 - Python for Poets  \n",
    "\n",
    "This assigment is an adaptation for Python of the original exercise [\"Unix for Poets\"](https://www.cs.upc.edu/~padro/Unixforpoets.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KBR said Friday the global economic downturn so far has\n",
      "had\n",
      "little effect on its business but warned some projects on its books\n",
      "could be in jeopardy if the headwinds persist into next year.\n",
      "\n",
      "\"With the economic outlook remaining uncertain, it is possible\n",
      "that\n",
      "customers may cancel or delay projects that are under way,\" said\n",
      "William\n",
      "Utt, chief executive of the Houston-based engineering and\n",
      "construction\n",
      "giant and government contractor.\n",
      "\n",
      "He did not predict how much of the company's $15.3billion in\n",
      "fu\n"
     ]
    }
   ],
   "source": [
    "with open(\"../../Data/txt/nyt_200811.txt\", \"r\") as f:\n",
    "    text = f.read()\n",
    "\n",
    "print(text[0:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You will solve the following exercises using **Pure Python**  \n",
    "### (only packages \"string\" and \"re\" are allowed).  \n",
    "\n",
    "1. Count words in a text  \n",
    "2. Sort a list of words in various ways  \n",
    "   • ascii order   \n",
    "   • \"rhyming\" order   \n",
    "3. Extract useful info for a dictionary  \n",
    "4. Compute ngram statistics  \n",
    "5. Make a Concordance  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Count words in a text\n",
    "\n",
    "a. Output a list of words in the file along with their frequency counts (ignoring case).   \n",
    "a. Count how many unique words there are (ignoring case).    \n",
    "c. Check how common are all different sequences of vowels (e.g. the sequences \"ieu\" or just \"e\" in \"lieutenant\")?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Sorting and reversing lines of text\n",
    "\n",
    "a. Sort each line alphabetically (ignoring case).  \n",
    "b. Sort in numeric ([ascii](https://python-reference.readthedocs.io/en/latest/docs/str/ASCII.html)) order.  \n",
    "c. Alphabetically reverse sort (ignoring case).  \n",
    "d. Sort in reverse numeric ([ascii](https://python-reference.readthedocs.io/en/latest/docs/str/ASCII.html)) order.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Computing basic statistics\n",
    "\n",
    "a. Find the 50 most common words  \n",
    "b. Find the words in the NYT that end in \"zz\"  \n",
    "c. Count the lines, the words, and the characters  \n",
    "d. How many all uppercase words are there in this NYT file?  \n",
    "e, How many 4-letter words?  \n",
    "f. How many different words are there with no vowels?  \n",
    "g. **tricky:** How many “1 syllable” words are there?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Compute ngrams  \n",
    "\n",
    "a. Find the 10 most common bigrams  \n",
    "b. Find the 10 most common trigrams  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Make a Concordance\n",
    "\n",
    "a. Create a concordance display for an arbitrary word. See the example below  \n",
    "\n",
    "![](../../Data/figs/Sample-concordance-lines-of-actually.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Extra Credit – Secret Message\n",
    "+ The answers to the extra credit exercises will reveal a secret message.  \n",
    "+ We will be working with the following text file for these exercises:  \n",
    "[Link to Text](https://web.stanford.edu/class/cs124/lec/secret_ec.txt)  \n",
    "(No starter code in the Extra Credit)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Extra Credit Exercise 1\n",
    "• Find the 2 most common words in secret_ec.txt containing the letter e.  \n",
    "• Your answer will correspond to the first two words of the secret message.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Extra Credit Exercise 2\n",
    "• Find the 2 most common bigrams in secret_ec.txt where the second word in the bigram ends with a consonant.  \n",
    "• Your answer will correspond to the next four words of the secret message.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Extra Credit Exercise 3\n",
    "• Find all 5-letter-long words that only appear once in secret_ec.txt.   \n",
    "• Concatenate your result. This will be the final word of the secret message.  \n",
    "\n",
    "What is the secret message?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}