project_tv_script_generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TV Script Generation\n",
    "\n",
    "In this project, you'll generate your own [Seinfeld](https://en.wikipedia.org/wiki/Seinfeld) TV scripts using RNNs.  You'll be using part of the [Seinfeld dataset](https://www.kaggle.com/thec03u5/seinfeld-chronicles#scripts.csv) of scripts from 9 seasons.  The Neural Network you'll build will generate a new ,\"fake\" TV script, based on patterns it recognizes in this training data.\n",
    "\n",
    "## Get the Data\n",
    "\n",
    "The data is already provided for you in `./data/Seinfeld_Scripts.txt` and you're encouraged to open that file and look at the text. \n",
    ">* As a first step, we'll load in this data and look at some samples. \n",
    "* Then, you'll be tasked with defining and training an RNN to generate a new script!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "# load in data\n",
    "import helper\n",
    "data_dir = './data/Seinfeld_Scripts.txt'\n",
    "text = helper.load_data(data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Data\n",
    "Play around with `view_line_range` to view different parts of the data. This will give you a sense of the data you'll be working with. You can see, for example, that it is all lowercase text, and each new line of dialogue is separated by a newline character `\\n`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset Stats\n",
      "Roughly the number of unique words: 46367\n",
      "Number of lines: 109233\n",
      "Average number of words in each line: 5.544240293684143\n",
      "\n",
      "The lines 0 to 10:\n",
      "jerry: do you know what this is all about? do you know, why were here? to be out, this is out...and out is one of the single most enjoyable experiences of life. people...did you ever hear people talking about we should go out? this is what theyre talking about...this whole thing, were all out now, no one is home. not one person here is home, were all out! there are people trying to find us, they dont know where we are. (on an imaginary phone) did you ring?, i cant find him. where did he go? he didnt tell me where he was going. he must have gone out. you wanna go out you get ready, you pick out the clothes, right? you take the shower, you get all ready, get the cash, get your friends, the car, the spot, the reservation...then youre standing around, what do you do? you go we gotta be getting back. once youre out, you wanna get back! you wanna go to sleep, you wanna get up, you wanna go out again tomorrow, right? where ever you are in life, its my feeling, youve gotta go. \n",
      "\n",
      "jerry: (pointing at georges shirt) see, to me, that button is in the worst possible spot. the second button literally makes or breaks the shirt, look at it. its too high! its in no-mans-land. you look like you live with your mother. \n",
      "\n",
      "george: are you through? \n",
      "\n",
      "jerry: you do of course try on, when you buy? \n",
      "\n",
      "george: yes, it was purple, i liked it, i dont actually recall considering the buttons. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "view_line_range = (0, 10)\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "import numpy as np\n",
    "\n",
    "print('Dataset Stats')\n",
    "print('Roughly the number of unique words: {}'.format(len({word: None for word in text.split()})))\n",
    "\n",
    "lines = text.split('\\n')\n",
    "print('Number of lines: {}'.format(len(lines)))\n",
    "word_count_line = [len(line.split()) for line in lines]\n",
    "print('Average number of words in each line: {}'.format(np.average(word_count_line)))\n",
    "\n",
    "print()\n",
    "print('The lines {} to {}:'.format(*view_line_range))\n",
    "print('\\n'.join(text.split('\\n')[view_line_range[0]:view_line_range[1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Implement Pre-processing Functions\n",
    "The first thing to do to any dataset is pre-processing.  Implement the following pre-processing functions below:\n",
    "- Lookup Table\n",
    "- Tokenize Punctuation\n",
    "\n",
    "### Lookup Table\n",
    "To create a word embedding, you first need to transform the words to ids.  In this function, create two dictionaries:\n",
    "- Dictionary to go from the words to an id, we'll call `vocab_to_int`\n",
    "- Dictionary to go from the id to word, we'll call `int_to_vocab`\n",
    "\n",
    "Return these dictionaries in the following **tuple** `(vocab_to_int, int_to_vocab)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "import problem_unittests as tests\n",
    "from string import punctuation\n",
    "from collections import Counter\n",
    "\n",
    "def create_lookup_tables(text, vocab_threshold = 1):\n",
    "    \"\"\"\n",
    "    Create lookup tables for vocabulary\n",
    "    :param text: The text of tv scripts split into words\n",
    "    :return: A tuple of dicts (vocab_to_int, int_to_vocab)\n",
    "    \"\"\"\n",
    "    \n",
    "    counter = Counter(text)\n",
    "    words = [word for word, cnt in counter.items() if cnt >= vocab_threshold]\n",
    "    \n",
    "    for key, value in helper.SPECIAL_WORDS.items():\n",
    "        words = [value] + words\n",
    "    \n",
    "    # build a dictionary that maps words to integers\n",
    "    vocab_to_int = {word:i for i, word in enumerate(words)}\n",
    "    #print(vocab_to_int['<PAD>'])\n",
    "    # build a dictionary that maps integers to words\n",
    "    int_to_vocab = {i:word for i, word in enumerate(words)}\n",
    "    # return tuple\n",
    "    return (vocab_to_int, int_to_vocab)\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_create_lookup_tables(create_lookup_tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenize Punctuation\n",
    "We'll be splitting the script into a word array using spaces as delimiters.  However, punctuations like periods and exclamation marks can create multiple ids for the same word. For example, \"bye\" and \"bye!\" would generate two different word ids.\n",
    "\n",
    "Implement the function `token_lookup` to return a dict that will be used to tokenize symbols like \"!\" into \"||Exclamation_Mark||\".  Create a dictionary for the following symbols where the symbol is the key and value is the token:\n",
    "- Period ( **.** )\n",
    "- Comma ( **,** )\n",
    "- Quotation Mark ( **\"** )\n",
    "- Semicolon ( **;** )\n",
    "- Exclamation mark ( **!** )\n",
    "- Question mark ( **?** )\n",
    "- Left Parentheses ( **(** )\n",
    "- Right Parentheses ( **)** )\n",
    "- Dash ( **-** )\n",
    "- Return ( **\\n** )\n",
    "\n",
    "This dictionary will be used to tokenize the symbols and add the delimiter (space) around it.  This separates each symbols as its own word, making it easier for the neural network to predict the next word. Make sure you don't use a value that could be confused as a word; for example, instead of using the value \"dash\", try using something like \"||dash||\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def token_lookup():\n",
    "    \"\"\"\n",
    "    Generate a dict to turn punctuation into a token.\n",
    "    :return: Tokenized dictionary where the key is the punctuation and the value is the token\n",
    "    \"\"\"\n",
    "    punctuation_to_token = {\n",
    "        '.':'<PERIOD>',\n",
    "        ',':'<COMMA>',\n",
    "        '\"':'<QUOTATION_MARK>',\n",
    "        ';':'<SEMICOLON>',\n",
    "        '!':'<EXCLAMATION_MARK>',\n",
    "        '?':'<QUESTION_MARK>',\n",
    "        '(':'<LEFT_PAREN>',\n",
    "        ')':'<RIGHT_PAREN>',\n",
    "        '-': '<DASH>',\n",
    "        '?':'<QUESTION_MARK>',\n",
    "        '\\n':'<NEW_LINE>',\n",
    "        #':':'<COLON>',\n",
    "        #'--': '<HYPHENS>',\n",
    "    }\n",
    "    \n",
    "    return punctuation_to_token\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_tokenize(token_lookup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-process all the data and save it\n",
    "\n",
    "Running the code cell below will pre-process all the data and save it to file. You're encouraged to lok at the code for `preprocess_and_save_data` in the `helpers.py` file to see what it's doing in detail, but you do not need to change this code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "vocab_threshold = 5\n",
    "# pre-process training data\n",
    "helper.preprocess_and_save_data(data_dir, token_lookup, create_lookup_tables, vocab_threshold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check Point\n",
    "This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6180\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "int_text, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()\n",
    "\n",
    "print(len(vocab_to_int))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the Neural Network\n",
    "In this section, you'll build the components necessary to build an RNN by implementing the RNN Module and forward and backpropagation functions.\n",
    "\n",
    "### Check Access to GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import torch\n",
    "\n",
    "# Check for a GPU\n",
    "train_on_gpu = torch.cuda.is_available()\n",
    "if not train_on_gpu:\n",
    "    print('No GPU found. Please use a GPU to train your neural network.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input\n",
    "Let's start with the preprocessed input data. We'll use [TensorDataset](http://pytorch.org/docs/master/data.html#torch.utils.data.TensorDataset) to provide a known format to our dataset; in combination with [DataLoader](http://pytorch.org/docs/master/data.html#torch.utils.data.DataLoader), it will handle batching, shuffling, and other dataset iteration functions.\n",
    "\n",
    "You can create data with TensorDataset by passing in feature and target tensors. Then create a DataLoader as usual.\n",
    "```\n",
    "data = TensorDataset(feature_tensors, target_tensors)\n",
    "data_loader = torch.utils.data.DataLoader(data, \n",
    "                                          batch_size=batch_size)\n",
    "```\n",
    "\n",
    "### Batching\n",
    "Implement the `batch_data` function to batch `words` data into chunks of size `batch_size` using the `TensorDataset` and `DataLoader` classes.\n",
    "\n",
    ">You can batch words using the DataLoader, but it will be up to you to create `feature_tensors` and `target_tensors` of the correct size and content for a given `sequence_length`.\n",
    "\n",
    "For example, say we have these as input:\n",
    "```\n",
    "words = [1, 2, 3, 4, 5, 6, 7]\n",
    "sequence_length = 4\n",
    "```\n",
    "\n",
    "Your first `feature_tensor` should contain the values:\n",
    "```\n",
    "[1, 2, 3, 4]\n",
    "```\n",
    "And the corresponding `target_tensor` should just be the next \"word\"/tokenized word value:\n",
    "```\n",
    "5\n",
    "```\n",
    "This should continue with the second `feature_tensor`, `target_tensor` being:\n",
    "```\n",
    "[2, 3, 4, 5]  # features\n",
    "6             # target\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test_text:\n",
      "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]\n",
      "\n",
      "test_text_pad:\n",
      "[0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]\n",
      "\n",
      "feature_tensors:\n",
      "[[ 0  0  0  0  1]\n",
      " [ 0  0  0  1  2]\n",
      " [ 0  0  1  2  3]\n",
      " [ 0  1  2  3  4]\n",
      " [ 1  2  3  4  5]\n",
      " [ 2  3  4  5  6]\n",
      " [ 3  4  5  6  7]\n",
      " [ 4  5  6  7  8]\n",
      " [ 5  6  7  8  9]\n",
      " [ 6  7  8  9 10]\n",
      " [ 7  8  9 10 11]\n",
      " [ 8  9 10 11 12]\n",
      " [ 9 10 11 12 13]\n",
      " [10 11 12 13 14]\n",
      " [11 12 13 14 15]\n",
      " [12 13 14 15 16]\n",
      " [13 14 15 16 17]\n",
      " [14 15 16 17 18]\n",
      " [15 16 17 18 19]\n",
      " [16 17 18 19 20]\n",
      " [17 18 19 20 21]\n",
      " [18 19 20 21 22]\n",
      " [19 20 21 22 23]\n",
      " [20 21 22 23 24]\n",
      " [21 22 23 24 25]\n",
      " [22 23 24 25 26]\n",
      " [23 24 25 26 27]\n",
      " [24 25 26 27 28]\n",
      " [25 26 27 28 29]\n",
      " [26 27 28 29 30]\n",
      " [27 28 29 30 31]\n",
      " [28 29 30 31 32]\n",
      " [29 30 31 32 33]\n",
      " [30 31 32 33 34]\n",
      " [31 32 33 34 35]\n",
      " [32 33 34 35 36]\n",
      " [33 34 35 36 37]\n",
      " [34 35 36 37 38]\n",
      " [35 36 37 38 39]\n",
      " [36 37 38 39 40]\n",
      " [37 38 39 40 41]\n",
      " [38 39 40 41 42]\n",
      " [39 40 41 42 43]\n",
      " [40 41 42 43 44]\n",
      " [41 42 43 44 45]\n",
      " [42 43 44 45 46]\n",
      " [43 44 45 46 47]\n",
      " [44 45 46 47 48]]\n",
      "\n",
      "target_tensors:\n",
      "[ 2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26\n",
      " 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49]\n",
      "\n",
      "\t\t\tFeature Shapes:\n",
      "Train set: \t\t(43, 5) \n",
      "Validation set: \t(5, 5)\n"
     ]
    }
   ],
   "source": [
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "def batch_data(words, sequence_length, batch_size, verbose=False):\n",
    "    \"\"\"\n",
    "    Batch the neural network data using DataLoader\n",
    "    :param words: The word ids of the TV scripts\n",
    "    :param sequence_length: The sequence length of each batch\n",
    "    :param batch_size: The size of each batch; the number of sequences in a batch\n",
    "    :return: DataLoader with batched data\n",
    "    \"\"\"\n",
    "    \n",
    "    words_len = len(words)\n",
    "    pad_word = helper.SPECIAL_WORDS['PADDING']\n",
    "    pad_value = vocab_to_int[pad_word]\n",
    "    \n",
    "    # add padding\n",
    "    for _ in range(sequence_length):\n",
    "        words = [pad_value] + words\n",
    "    if verbose:\n",
    "        print(f'test_text_pad:\\n{words }\\n')\n",
    "    \n",
    "    feature_tensors = np.array([words[i:sequence_length+i] for i in range(1,words_len)])\n",
    "    target_tensors = np.array([words[sequence_length+i] for i in range(1,words_len)])\n",
    "    \n",
    "    if verbose:\n",
    "        print(f'feature_tensors:\\n{feature_tensors}\\n')\n",
    "        print(f'target_tensors:\\n{target_tensors}\\n')\n",
    "    \n",
    "    \n",
    "    split_frac = 0.9\n",
    "\n",
    "    ## split data into training and validation (features and targets, x and y)\n",
    "\n",
    "    split_idx = int(len(feature_tensors)*split_frac)\n",
    "    train_x, val_x = feature_tensors[:split_idx], feature_tensors[split_idx:]\n",
    "    train_y, val_y = target_tensors[:split_idx], target_tensors[split_idx:]\n",
    "    \n",
    "    if verbose:\n",
    "        ## print out the shapes of your resultant feature data\n",
    "        print(\"\\t\\t\\tFeature Shapes:\")\n",
    "        print(\"Train set: \\t\\t{}\".format(train_x.shape), \n",
    "              \"\\nValidation set: \\t{}\".format(val_x.shape))\n",
    "    \n",
    "    # create Tensor datasets\n",
    "    train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))\n",
    "    valid_data = TensorDataset(torch.from_numpy(val_x), torch.from_numpy(val_y))\n",
    "        \n",
    "    # make sure the SHUFFLE your training data\n",
    "    train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)\n",
    "    valid_loader = DataLoader(valid_data, shuffle=True, batch_size=batch_size)\n",
    "    \n",
    "    # return a dataloaders\n",
    "    dataloaders = {'train':train_loader ,'valid':valid_loader}\n",
    "    return dataloaders\n",
    "test_text = list(range(1,50))\n",
    "print(f'test_text:\\n{test_text}\\n')\n",
    "t_loader = batch_data(test_text, sequence_length=5, batch_size=10, verbose=True)\n",
    "# there is no test for this function, but you are encouraged to create\n",
    "# print statements and tests of your own"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test your dataloader \n",
    "\n",
    "You'll have to modify this code to test a batching function, but it should look fairly similar.\n",
    "\n",
    "Below, we're generating some test text data and defining a dataloader using the function you defined, above. Then, we are getting some sample batch of inputs `sample_x` and targets `sample_y` from our dataloader.\n",
    "\n",
    "Your code should return something like the following (likely in a different order, if you shuffled your data):\n",
    "\n",
    "```\n",
    "torch.Size([10, 5])\n",
    "tensor([[ 28,  29,  30,  31,  32],\n",
    "        [ 21,  22,  23,  24,  25],\n",
    "        [ 17,  18,  19,  20,  21],\n",
    "        [ 34,  35,  36,  37,  38],\n",
    "        [ 11,  12,  13,  14,  15],\n",
    "        [ 23,  24,  25,  26,  27],\n",
    "        [  6,   7,   8,   9,  10],\n",
    "        [ 38,  39,  40,  41,  42],\n",
    "        [ 25,  26,  27,  28,  29],\n",
    "        [  7,   8,   9,  10,  11]])\n",
    "\n",
    "torch.Size([10])\n",
    "tensor([ 33,  26,  22,  39,  16,  28,  11,  43,  30,  12])\n",
    "```\n",
    "\n",
    "### Sizes\n",
    "Your sample_x should be of size `(batch_size, sequence_length)` or (10, 5) in this case and sample_y should just have one dimension: batch_size (10). \n",
    "\n",
    "### Values\n",
    "\n",
    "You should also notice that the targets, sample_y, are the *next* value in the ordered test_text data. So, for an input sequence `[ 28,  29,  30,  31,  32]` that ends with the value `32`, the corresponding output should be `33`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train\n",
      "torch.Size([10, 5])\n",
      "tensor([[ 237,  238,  239,  240,  241],\n",
      "        [  72,   73,   74,   75,   76],\n",
      "        [ 230,  231,  232,  233,  234],\n",
      "        [  71,   72,   73,   74,   75],\n",
      "        [ 284,  285,  286,  287,  288],\n",
      "        [  52,   53,   54,   55,   56],\n",
      "        [  17,   18,   19,   20,   21],\n",
      "        [ 195,  196,  197,  198,  199],\n",
      "        [ 221,  222,  223,  224,  225],\n",
      "        [ 150,  151,  152,  153,  154]])\n",
      "\n",
      "torch.Size([10])\n",
      "tensor([ 242,   77,  235,   76,  289,   57,   22,  200,  226,  155])\n",
      "\n",
      "valid\n",
      "torch.Size([10, 5])\n",
      "tensor([[ 477,  478,  479,  480,  481],\n",
      "        [ 463,  464,  465,  466,  467],\n",
      "        [ 460,  461,  462,  463,  464],\n",
      "        [ 490,  491,  492,  493,  494],\n",
      "        [ 481,  482,  483,  484,  485],\n",
      "        [ 485,  486,  487,  488,  489],\n",
      "        [ 484,  485,  486,  487,  488],\n",
      "        [ 451,  452,  453,  454,  455],\n",
      "        [ 465,  466,  467,  468,  469],\n",
      "        [ 447,  448,  449,  450,  451]])\n",
      "\n",
      "torch.Size([10])\n",
      "tensor([ 482,  468,  465,  495,  486,  490,  489,  456,  470,  452])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# test dataloader\n",
    "\n",
    "test_text = list(range(1,500))\n",
    "t_loader = batch_data(test_text, sequence_length=5, batch_size=10)\n",
    "\n",
    "for key,val in t_loader.items():\n",
    "    data_iter = iter(t_loader[key])\n",
    "    sample_x, sample_y = data_iter.next()\n",
    "    \n",
    "    print(key)\n",
    "    print(sample_x.shape)\n",
    "    print(sample_x)\n",
    "    print()\n",
    "    print(sample_y.shape)\n",
    "    print(sample_y)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Build the Neural Network\n",
    "Implement an RNN using PyTorch's [Module class](http://pytorch.org/docs/master/nn.html#torch.nn.Module). You may choose to use a GRU or an LSTM. To complete the RNN, you'll have to implement the following functions for the class:\n",
    " - `__init__` - The initialize function. \n",
    " - `init_hidden` - The initialization function for an LSTM/GRU hidden state\n",
    " - `forward` - Forward propagation function.\n",
    " \n",
    "The initialize function should create the layers of the neural network and save them to the class. The forward propagation function will use these layers to run forward propagation and generate an output and a hidden state.\n",
    "\n",
    "**The output of this model should be the *last* batch of word scores** after a complete sequence has been processed. That is, for each input sequence of words, we only want to output the word scores for a single, most likely, next word.\n",
    "\n",
    "### Hints\n",
    "\n",
    "1. Make sure to stack the outputs of the lstm to pass to your fully-connected layer, you can do this with `lstm_output = lstm_output.contiguous().view(-1, self.hidden_dim)`\n",
    "2. You can get the last batch of word scores by shaping the output of the final, fully-connected layer like so:\n",
    "\n",
    "```\n",
    "# reshape into (batch_size, seq_length, output_size)\n",
    "output = output.view(batch_size, -1, self.output_size)\n",
    "# get last batch\n",
    "out = output[:, -1]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, dropout=0.5):\n",
    "        \"\"\"\n",
    "        Initialize the PyTorch RNN Module\n",
    "        :param vocab_size: The number of input dimensions of the neural network (the size of the vocabulary)\n",
    "        :param output_size: The number of output dimensions of the neural network\n",
    "        :param embedding_dim: The size of embeddings, should you choose to use them        \n",
    "        :param hidden_dim: The size of the hidden layer outputs\n",
    "        :param dropout: dropout to add in between LSTM/GRU layers\n",
    "        \"\"\"\n",
    "        super(RNN, self).__init__()\n",
    "        \n",
    "        # set class variables\n",
    "        \n",
    "        self.output_size = output_size\n",
    "        self.n_layers = n_layers\n",
    "        self.hidden_dim = hidden_dim\n",
    "        \n",
    "        # define model layers\n",
    "        self.dropout = nn.Dropout(0.5)\n",
    "        self.fc = nn.Linear(hidden_dim, output_size)\n",
    "\n",
    "        # embedding and LSTM layers\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, \n",
    "                            dropout=dropout, batch_first=True)\n",
    "        \n",
    "        # initialize the weights\n",
    "        self.init_weights()\n",
    "        \n",
    "    def forward(self, nn_input, hidden):\n",
    "        \"\"\"\n",
    "        Forward propagation of the neural network\n",
    "        :param nn_input: The input to the neural network\n",
    "        :param hidden: The hidden state        \n",
    "        :return: Two Tensors, the output of the neural network and the latest hidden state\n",
    "        \"\"\"\n",
    "        batch_size = nn_input.shape[0]\n",
    "        \n",
    "        # embeddings and lstm_out\n",
    "        embeds = self.embedding(nn_input)\n",
    "        lstm_out, hidden = self.lstm(embeds, hidden)\n",
    "        \n",
    "        # stack up lstm outputs\n",
    "        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)\n",
    "        \n",
    "        # dropout and fully-connected layer\n",
    "        out = self.dropout(lstm_out)\n",
    "        out = self.fc(out)\n",
    "        \n",
    "        # reshape into (batch_size, seq_length, output_size)\n",
    "        out = out.view(batch_size, -1, self.output_size)\n",
    "        out = out[:, -1] # get last batch of labels\n",
    "                \n",
    "        # return one batch of output word scores and the hidden state\n",
    "        return out, hidden\n",
    "         \n",
    "    def init_hidden(self, batch_size):\n",
    "        '''\n",
    "        Initialize the hidden state of an LSTM/GRU\n",
    "        :param batch_size: The batch_size of the hidden state\n",
    "        :return: hidden state of dims (n_layers, batch_size, hidden_dim)\n",
    "        '''\n",
    "        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,\n",
    "        # initialized to zero, for hidden state and cell state of LSTM\n",
    "        weight = next(self.parameters()).data\n",
    "        \n",
    "        if (train_on_gpu):\n",
    "            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),\n",
    "                  weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())\n",
    "        else:\n",
    "            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),\n",
    "                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())\n",
    "        \n",
    "        return hidden\n",
    "    \n",
    "    def init_weights(self):\n",
    "        ''' Initialize weights for fully connected layer '''\n",
    "        initrange = 0.1\n",
    "        \n",
    "        # Set bias tensor to all zeros\n",
    "        self.fc.bias.data.fill_(0)\n",
    "        # FC weights as random uniform\n",
    "        self.fc.weight.data.uniform_(-1, 1)\n",
    "        \n",
    "        # init forget gate bias to 1\n",
    "        for names in self.lstm._all_weights:\n",
    "            for name in filter(lambda n: \"bias\" in n,  names):\n",
    "                bias = getattr(self.lstm, name)\n",
    "                n = bias.size(0)\n",
    "                start, end = n//4, n//2\n",
    "                bias.data[start:end].fill_(1.)\n",
    "        # \"Importantly, adding a bias of size 1 significantly improved \n",
    "        # the performance of the LSTM on tasks where it fell behind the \n",
    "        # GRU and MUT1. Thus we recommend adding a bias of 1 to the forget \n",
    "        # gate of every LSTM in every application; it is easy to do often \n",
    "        # results in better performance on our tasks. This adjustment is \n",
    "        # the simple improvement over the LSTM that we set out to discover.\"\n",
    "        # http://proceedings.mlr.press/v37/jozefowicz15.pdf\n",
    "        # https://discuss.pytorch.org/t/set-forget-gate-bias-of-lstm/1745/4\n",
    "        \n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_rnn(RNN, train_on_gpu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define forward and backpropagation\n",
    "\n",
    "Use the RNN class you implemented to apply forward and back propagation. This function will be called, iteratively, in the training loop as follows:\n",
    "```\n",
    "loss = forward_back_prop(decoder, decoder_optimizer, criterion, inp, target)\n",
    "```\n",
    "\n",
    "And it should return the average loss over a batch and the hidden state returned by a call to `RNN(inp, hidden)`. Recall that you can get this loss by computing it, as usual, and calling `loss.item()`.\n",
    "\n",
    "**If a GPU is available, you should move your data to that GPU device, here.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def forward_back_prop(rnn, optimizer, criterion, inp, target, hidden):\n",
    "    \"\"\"\n",
    "    Forward and backward propagation on the neural network\n",
    "    :param rnn: The PyTorch Module that holds the neural network\n",
    "    :param optimizer: The PyTorch optimizer for the neural network\n",
    "    :param criterion: The PyTorch loss function\n",
    "    :param inp: A batch of input to the neural network\n",
    "    :param target: The target output for the batch of input\n",
    "    :return: The loss and the latest hidden state Tensor\n",
    "    \"\"\"\n",
    "    clip=5 # gradient clipping\n",
    "    \n",
    "    # Creating new variables for the hidden state, otherwise\n",
    "    # we'd backprop through the entire training history\n",
    "    hidden = tuple([each.data for each in hidden])\n",
    "\n",
    "    # move data to GPU, if available\n",
    "    if(train_on_gpu):\n",
    "        inp, target = inp.cuda(), target.cuda()\n",
    "        \n",
    "    # zero accumulated gradients\n",
    "    rnn.zero_grad()\n",
    "    \n",
    "    # perform backpropagation and optimization\n",
    "    output, hidden = rnn(inp, hidden)\n",
    "        \n",
    "    loss = criterion(output.squeeze(), target)\n",
    "    loss.backward()\n",
    "    # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.\n",
    "    nn.utils.clip_grad_norm_(rnn.parameters(), clip)\n",
    "    optimizer.step()\n",
    "    \n",
    "    # return the loss over a batch and the hidden state produced by our model\n",
    "    return loss.item(), hidden\n",
    "\n",
    "# Note that these tests aren't completely extensive.\n",
    "# they are here to act as general checks on the expected outputs of your functions\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_forward_back_prop(RNN, forward_back_prop, train_on_gpu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Early Stopping\n",
    "Early Stopping is a form of regularization used to avoid overfitting on the training dataset. A simple implementation of Early Stopping keeps track of the validation loss, if the loss stops decreasing several epochs in a row the training stops. My EarlyStopping class is inspired by the [ignite EarlyStopping class](https://github.com/pytorch/ignite/blob/master/ignite/handlers/early_stopping.py), but this is a bare minimum version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EarlyStopping:\n",
    "    \"\"\"Early stops the training if validation loss dosen't improve after a given patience.\"\"\"\n",
    "    def __init__(self,patience=3):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            patience (int): How long to wait after last time validation loss improved. \n",
    "        \"\"\"\n",
    "        self.patience = patience\n",
    "        self.counter = 0\n",
    "        self.best_score = None\n",
    "        self.early_stop = False\n",
    "        self.val_loss_min = np.Inf\n",
    "    \n",
    "    def __call__(self, val_loss, model):\n",
    "        \n",
    "        score = -val_loss\n",
    "        \n",
    "        if self.best_score is None:\n",
    "            self.best_score = score\n",
    "        elif score < self.best_score:\n",
    "            self.counter += 1\n",
    "            print(f'EarlyStopping counter: {self.counter} out of {self.patience}')\n",
    "            if self.counter >= self.patience:\n",
    "                self.early_stop = True\n",
    "        else:\n",
    "            self.best_score = score\n",
    "            self.save_checkpoint(val_loss, model)\n",
    "            self.counter = 0\n",
    "    \n",
    "    def save_checkpoint(self, val_loss, model):\n",
    "        '''Saves model when validation loss decrease.'''\n",
    "        print(f'Validation loss decreased ({self.val_loss_min:.6f} --> {val_loss:.6f}).  Saving model ...')\n",
    "        torch.save(model.state_dict(), 'saved_models/checkpoint.pt')\n",
    "        self.val_loss_min = val_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network Training\n",
    "\n",
    "With the structure of the network complete and data ready to be fed in the neural network, it's time to train it.\n",
    "\n",
    "### Train Loop\n",
    "\n",
    "The training loop is implemented for you in the `train_decoder` function. This function will train the network over all the batches for the number of epochs given. The model progress will be shown every number of batches. This number is set with the `show_every_n_batches` parameter. You'll set this parameter along with other parameters in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "\n",
    "def train_rnn(rnn, batch_size, optimizer, criterion, n_epochs, show_every_n_batches=100):\n",
    "    batch_losses = []\n",
    "    \n",
    "    rnn.train()\n",
    "\n",
    "    print(\"Training for %d epoch(s)...\" % n_epochs)\n",
    "    for epoch_i in range(1, n_epochs + 1):\n",
    "        \n",
    "        # initialize hidden state\n",
    "        hidden = rnn.init_hidden(batch_size)\n",
    "        \n",
    "        for batch_i, (inputs, labels) in enumerate(train_loader, 1):\n",
    "            \n",
    "            # make sure you iterate over completely full batches, only\n",
    "            n_batches = len(train_loader.dataset)//batch_size\n",
    "            if(batch_i > n_batches):\n",
    "                break\n",
    "            \n",
    "            # forward, back prop\n",
    "            loss, hidden = forward_back_prop(rnn, optimizer, criterion, inputs, labels, hidden)          \n",
    "            # record loss\n",
    "            \n",
    "            batch_losses.append(loss)\n",
    "\n",
    "            # printing loss stats\n",
    "            if batch_i % show_every_n_batches == 0:\n",
    "                print('Epoch: {:>4}/{:<4}  Loss: {}\\n'.format(\n",
    "                    epoch_i, n_epochs, np.average(batch_losses)))\n",
    "                batch_losses = []\n",
    "\n",
    "    # returns a trained rnn\n",
    "    return rnn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_rnn_2(rnn, batch_size, optimizer, criterion, n_epochs, scheduler, show_every_n_batches=100):\n",
    "    \n",
    "    train_losses = []\n",
    "    valid_losses = []\n",
    "    \n",
    "    early_stopping = EarlyStopping()\n",
    "    \n",
    "    rnn.train()\n",
    "\n",
    "    print(\"Training for %d epoch(s)...\" % n_epochs)\n",
    "    for epoch_i in range(1, n_epochs + 1):\n",
    "        \n",
    "        # initialize hidden state\n",
    "        hidden = rnn.init_hidden(batch_size)\n",
    "        \n",
    "        for batch_i, (inputs, labels) in enumerate(train_loader, 1):\n",
    "            \n",
    "            # make sure you iterate over completely full batches, only\n",
    "            n_batches = len(train_loader.dataset)//batch_size\n",
    "            if(batch_i > n_batches):\n",
    "                break\n",
    "            \n",
    "            # forward, back prop\n",
    "            tarin_loss, hidden = forward_back_prop(rnn, optimizer, criterion, inputs, labels, hidden)          \n",
    "            # record loss\n",
    "            train_losses.append(tarin_loss)\n",
    "\n",
    "            # printing loss stats\n",
    "            if batch_i % show_every_n_batches == 0 or batch_i == len(train_loader)-1:\n",
    "                \n",
    "                # Get validation loss\n",
    "                val_h = rnn.init_hidden(batch_size)\n",
    "                rnn.eval()\n",
    "                for val_batch_i, (inputs, labels) in enumerate(valid_loader, 1):\n",
    "                    \n",
    "                    # make sure you iterate over completely full batches, only\n",
    "                    n_batches = len(valid_loader.dataset)//batch_size\n",
    "                    if(val_batch_i > n_batches):\n",
    "                        break\n",
    "                    \n",
    "                    val_h = tuple([each.data for each in val_h])\n",
    "                    if(train_on_gpu):\n",
    "                        inputs, labels = inputs.cuda(), labels.cuda()\n",
    "                    output, val_h = rnn(inputs, val_h)\n",
    "                    valid_loss = criterion(output.squeeze(), labels)\n",
    "                    valid_losses.append(valid_loss.item())\n",
    "                    \n",
    "                rnn.train()\n",
    "                \n",
    "                epoch_len = len(str(n_epochs))\n",
    "                batches = len(train_loader)\n",
    "                batches_len = len(str(batches))\n",
    "                avg_val_loss = np.average(valid_losses)\n",
    "                \n",
    "                print(f'[{epoch_i:>{epoch_len}}/{n_epochs:>{epoch_len}}][{batch_i:>{batches_len}}/{batches:>{batches_len}}] train_loss: {np.average(train_losses):.5f} valid_loss: {avg_val_loss:.5f}')\n",
    "        \n",
    "        train_losses = []\n",
    "        valid_losses = []\n",
    "\n",
    "        # reduce learning rate when avg_val_loss has stopped improving\n",
    "        scheduler.step(avg_val_loss)\n",
    "        \n",
    "        early_stopping(avg_val_loss, rnn)\n",
    "        \n",
    "        if early_stopping.early_stop:\n",
    "            print(\"Early stopping\")\n",
    "            break\n",
    "    \n",
    "    # load last checkpoint with the best model\n",
    "    rnn.load_state_dict(torch.load('saved_models/checkpoint.pt'))\n",
    "\n",
    "    # returns a trained rnn\n",
    "    return rnn, early_stopping.val_loss_min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameters\n",
    "\n",
    "Set and train the neural network with the following parameters:\n",
    "- Set `sequence_length` to the length of a sequence.\n",
    "- Set `batch_size` to the batch size.\n",
    "- Set `num_epochs` to the number of epochs to train for.\n",
    "- Set `learning_rate` to the learning rate for an Adam optimizer.\n",
    "- Set `vocab_size` to the number of uniqe tokens in our vocabulary.\n",
    "- Set `output_size` to the desired size of the output.\n",
    "- Set `embedding_dim` to the embedding dimension; smaller than the vocab_size.\n",
    "- Set `hidden_dim` to the hidden dimension of your RNN.\n",
    "- Set `n_layers` to the number of layers/cells in your RNN.\n",
    "- Set `show_every_n_batches` to the number of batches at which the neural network should print progress.\n",
    "\n",
    "If the network isn't getting the desired results, tweak these parameters and/or the layers in the `RNN` class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train\n",
    "In the next cell, you'll train the neural network on the pre-processed data.  If you have a hard time getting a good loss, you may consider changing your hyperparameters. In general, you may get better results with larger hidden and n_layer dimensions, but larger models take a longer time to train. \n",
    "> **You should aim for a loss less than 3.5.** \n",
    "\n",
    "You should also experiment with different sequence lengths, which determine the size of the long range dependencies that a model can learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Data params\n",
    "# Sequence Length\n",
    "sequence_length =  16 # of words in a sequence\n",
    "# Batch Size\n",
    "batch_size = 512\n",
    "\n",
    "# data loaders\n",
    "dataloaders = batch_data(int_text, sequence_length, batch_size)\n",
    "train_loader = dataloaders['train']\n",
    "valid_loader = dataloaders['valid']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training parameters\n",
    "# Number of Epochs\n",
    "num_epochs = 200\n",
    "# Learning Rate\n",
    "learning_rate = 0.001\n",
    "\n",
    "# Model parameters\n",
    "# Vocab size\n",
    "vocab_size = len(vocab_to_int)\n",
    "# Output size\n",
    "output_size = vocab_size\n",
    "# Embedding Dimension\n",
    "embedding_dim = 400\n",
    "# Hidden Dimension\n",
    "hidden_dim = 512\n",
    "# Number of RNN Layers\n",
    "n_layers = 2\n",
    "\n",
    "# Show stats for every n number of batches\n",
    "show_every_n_batches = 300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training for 200 epoch(s)...\n",
      "[  1/200][ 300/1569] train_loss: 5.54476 valid_loss: 4.48504\n",
      "[  1/200][ 600/1569] train_loss: 5.06631 valid_loss: 4.38971\n",
      "[  1/200][ 900/1569] train_loss: 4.83897 valid_loss: 4.32295\n",
      "[  1/200][1200/1569] train_loss: 4.69539 valid_loss: 4.27037\n",
      "[  1/200][1500/1569] train_loss: 4.59219 valid_loss: 4.22650\n",
      "[  1/200][1568/1569] train_loss: 4.57349 valid_loss: 4.19645\n",
      "[  2/200][ 300/1569] train_loss: 4.06533 valid_loss: 3.99651\n",
      "[  2/200][ 600/1569] train_loss: 4.04192 valid_loss: 3.98431\n",
      "[  2/200][ 900/1569] train_loss: 4.01955 valid_loss: 3.97094\n",
      "[  2/200][1200/1569] train_loss: 4.00139 valid_loss: 3.95689\n",
      "[  2/200][1500/1569] train_loss: 3.98300 valid_loss: 3.94538\n",
      "[  2/200][1568/1569] train_loss: 3.97855 valid_loss: 3.93634\n",
      "Validation loss decreased (inf --> 3.936337).  Saving model ...\n",
      "[  3/200][ 300/1569] train_loss: 3.82666 valid_loss: 3.87983\n",
      "[  3/200][ 600/1569] train_loss: 3.82819 valid_loss: 3.87481\n",
      "[  3/200][ 900/1569] train_loss: 3.82268 valid_loss: 3.86863\n",
      "[  3/200][1200/1569] train_loss: 3.81352 valid_loss: 3.86125\n",
      "[  3/200][1500/1569] train_loss: 3.80759 valid_loss: 3.85526\n",
      "[  3/200][1568/1569] train_loss: 3.80481 valid_loss: 3.85120\n",
      "Validation loss decreased (3.936337 --> 3.851200).  Saving model ...\n",
      "[  4/200][ 300/1569] train_loss: 3.70579 valid_loss: 3.82643\n",
      "[  4/200][ 600/1569] train_loss: 3.70633 valid_loss: 3.82411\n",
      "[  4/200][ 900/1569] train_loss: 3.70400 valid_loss: 3.82179\n",
      "[  4/200][1200/1569] train_loss: 3.69995 valid_loss: 3.81832\n",
      "[  4/200][1500/1569] train_loss: 3.69699 valid_loss: 3.81459\n",
      "[  4/200][1568/1569] train_loss: 3.69599 valid_loss: 3.81187\n",
      "Validation loss decreased (3.851200 --> 3.811866).  Saving model ...\n",
      "[  5/200][ 300/1569] train_loss: 3.60465 valid_loss: 3.79926\n",
      "[  5/200][ 600/1569] train_loss: 3.61010 valid_loss: 3.79513\n",
      "[  5/200][ 900/1569] train_loss: 3.61749 valid_loss: 3.79288\n",
      "[  5/200][1200/1569] train_loss: 3.61441 valid_loss: 3.79128\n",
      "[  5/200][1500/1569] train_loss: 3.61522 valid_loss: 3.78900\n",
      "[  5/200][1568/1569] train_loss: 3.61415 valid_loss: 3.78775\n",
      "Validation loss decreased (3.811866 --> 3.787746).  Saving model ...\n",
      "[  6/200][ 300/1569] train_loss: 3.54053 valid_loss: 3.77850\n",
      "[  6/200][ 600/1569] train_loss: 3.54464 valid_loss: 3.77760\n",
      "[  6/200][ 900/1569] train_loss: 3.54561 valid_loss: 3.77796\n",
      "[  6/200][1200/1569] train_loss: 3.54603 valid_loss: 3.77804\n",
      "[  6/200][1500/1569] train_loss: 3.54739 valid_loss: 3.77826\n",
      "[  6/200][1568/1569] train_loss: 3.54897 valid_loss: 3.77642\n",
      "Validation loss decreased (3.787746 --> 3.776417).  Saving model ...\n",
      "[  7/200][ 300/1569] train_loss: 3.45743 valid_loss: 3.77792\n",
      "[  7/200][ 600/1569] train_loss: 3.47913 valid_loss: 3.77350\n",
      "[  7/200][ 900/1569] train_loss: 3.48239 valid_loss: 3.77161\n",
      "[  7/200][1200/1569] train_loss: 3.49088 valid_loss: 3.76931\n",
      "[  7/200][1500/1569] train_loss: 3.49203 valid_loss: 3.76836\n",
      "[  7/200][1568/1569] train_loss: 3.49316 valid_loss: 3.76786\n",
      "Validation loss decreased (3.776417 --> 3.767856).  Saving model ...\n",
      "[  8/200][ 300/1569] train_loss: 3.42618 valid_loss: 3.77368\n",
      "[  8/200][ 600/1569] train_loss: 3.43374 valid_loss: 3.77216\n",
      "[  8/200][ 900/1569] train_loss: 3.43489 valid_loss: 3.77001\n",
      "[  8/200][1200/1569] train_loss: 3.44187 valid_loss: 3.77039\n",
      "[  8/200][1500/1569] train_loss: 3.44504 valid_loss: 3.76948\n",
      "[  8/200][1568/1569] train_loss: 3.44453 valid_loss: 3.76900\n",
      "EarlyStopping counter: 1 out of 3\n",
      "[  9/200][ 300/1569] train_loss: 3.37211 valid_loss: 3.77287\n",
      "[  9/200][ 600/1569] train_loss: 3.38436 valid_loss: 3.77318\n",
      "[  9/200][ 900/1569] train_loss: 3.39109 valid_loss: 3.77236\n",
      "[  9/200][1200/1569] train_loss: 3.39576 valid_loss: 3.77120\n",
      "[  9/200][1500/1569] train_loss: 3.39930 valid_loss: 3.77094\n",
      "[  9/200][1568/1569] train_loss: 3.39999 valid_loss: 3.77029\n",
      "Epoch     8: reducing learning rate of group 0 to 1.0000e-04.\n",
      "EarlyStopping counter: 2 out of 3\n",
      "[ 10/200][ 300/1569] train_loss: 3.31109 valid_loss: 3.76735\n",
      "[ 10/200][ 600/1569] train_loss: 3.30678 valid_loss: 3.76691\n",
      "[ 10/200][ 900/1569] train_loss: 3.30594 valid_loss: 3.76692\n",
      "[ 10/200][1200/1569] train_loss: 3.30109 valid_loss: 3.76697\n",
      "[ 10/200][1500/1569] train_loss: 3.30127 valid_loss: 3.76666\n",
      "[ 10/200][1568/1569] train_loss: 3.30148 valid_loss: 3.76653\n",
      "Validation loss decreased (3.767856 --> 3.766528).  Saving model ...\n",
      "[ 11/200][ 300/1569] train_loss: 3.27929 valid_loss: 3.76784\n",
      "[ 11/200][ 600/1569] train_loss: 3.27017 valid_loss: 3.76787\n",
      "[ 11/200][ 900/1569] train_loss: 3.27287 valid_loss: 3.76810\n",
      "[ 11/200][1200/1569] train_loss: 3.27050 valid_loss: 3.76780\n",
      "[ 11/200][1500/1569] train_loss: 3.27107 valid_loss: 3.76765\n",
      "[ 11/200][1568/1569] train_loss: 3.27070 valid_loss: 3.76755\n",
      "EarlyStopping counter: 1 out of 3\n",
      "[ 12/200][ 300/1569] train_loss: 3.25071 valid_loss: 3.76916\n",
      "[ 12/200][ 600/1569] train_loss: 3.25454 valid_loss: 3.76932\n",
      "[ 12/200][ 900/1569] train_loss: 3.25525 valid_loss: 3.76962\n",
      "[ 12/200][1200/1569] train_loss: 3.25577 valid_loss: 3.76961\n",
      "[ 12/200][1500/1569] train_loss: 3.25511 valid_loss: 3.76996\n",
      "[ 12/200][1568/1569] train_loss: 3.25461 valid_loss: 3.77017\n",
      "Epoch    11: reducing learning rate of group 0 to 1.0000e-05.\n",
      "EarlyStopping counter: 2 out of 3\n",
      "[ 13/200][ 300/1569] train_loss: 3.23349 valid_loss: 3.77136\n",
      "[ 13/200][ 600/1569] train_loss: 3.23475 valid_loss: 3.77131\n",
      "[ 13/200][ 900/1569] train_loss: 3.23440 valid_loss: 3.77131\n",
      "[ 13/200][1200/1569] train_loss: 3.23143 valid_loss: 3.77155\n",
      "[ 13/200][1500/1569] train_loss: 3.23429 valid_loss: 3.77156\n",
      "[ 13/200][1568/1569] train_loss: 3.23515 valid_loss: 3.77156\n",
      "EarlyStopping counter: 3 out of 3\n",
      "Early stopping\n",
      "Model Trained and Saved\n"
     ]
    }
   ],
   "source": [
    "from torch.optim import lr_scheduler\n",
    "\n",
    "# create model and move to gpu if available\n",
    "rnn = RNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers, dropout=0.5)\n",
    "if train_on_gpu:\n",
    "    rnn.cuda()\n",
    "\n",
    "# defining loss and optimization functions for training\n",
    "optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# reduce learning rate when a validation loss has stopped improving\n",
    "plateau_lr_scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, 'min',  patience=1, verbose=True)\n",
    "\n",
    "# training the model\n",
    "trained_rnn, val_loss = train_rnn_2(rnn, batch_size, optimizer, criterion, num_epochs,\n",
    "                          plateau_lr_scheduler, show_every_n_batches)\n",
    "\n",
    "# saving the trained model\n",
    "torch.save(rnn.state_dict(), f'saved_models/trained_rnn_seql_{sequence_length}_vloss_{val_loss:.5f}.pt')\n",
    "print('Model Trained and Saved')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question: How did you decide on your model hyperparameters? \n",
    "For example, did you try different sequence_lengths and find that one size made the model converge faster? What about your hidden_dim and n_layers; how did you decide on those?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:**\n",
    "\n",
    "```sequence_length``` I tried many different sequence_lengths, and found the longer the sequence the lower the validation loss would go, but the training would take longer. I found a good length too be 16 after trying many different lengths in the range 1-32. \n",
    "\n",
    "```hidden_dim``` When increasing the hidden_dim the validation loss would decrease, the largest hidden dim I could have and still train fast was 512, this is due to memory limitations in the GPU. \n",
    "\n",
    "```learning_rate``` I have tried different learning rates in the range 0.1 to 0.0005 and found that the default value of 0.001 gives the best results. \n",
    "\n",
    "```num_epochs``` I have implemented early stopping, so the training will stop when the validation loss stops decreasing. \n",
    "\n",
    "```n_layers``` It is recommended to use between 1-3 layers, I have had good results with 2 layers in the past, so that was a natural choice. I also tried with 3 layers without seeing any improvements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Checkpoint\n",
    "\n",
    "After running the above training cell, your model will be saved by name, `trained_rnn`, and if you save your notebook progress, **you can pause here and come back to this code at another time**. You can resume your progress by running the next cell, which will load in our word:id dictionaries _and_ load in your saved model by name!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import torch\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "_, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()\n",
    "#trained_rnn = helper.load_model('./save/saved_models/trained_rnn_vloss_3.77889')\n",
    "rnn.load_state_dict(torch.load('saved_models/trained_rnn_seql_16_vloss_3.76653.pt'))\n",
    "trained_rnn = rnn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate TV Script\n",
    "With the network trained and saved, you'll use it to generate a new, \"fake\" Seinfeld TV script in this section.\n",
    "\n",
    "### Generate Text\n",
    "To generate the text, the network needs to start with a single word and repeat its predictions until it reaches a set length. You'll be using the `generate` function to do this. It takes a word id to start with, `prime_id`, and generates a set length of text, `predict_len`. Also note that it uses topk sampling to introduce some randomness in choosing the most likely next word, given an output set of word scores!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "import torch.nn.functional as F\n",
    "\n",
    "def generate(rnn, prime_id, int_to_vocab, token_dict, pad_value, predict_len=100):\n",
    "    \"\"\"\n",
    "    Generate text using the neural network\n",
    "    :param decoder: The PyTorch Module that holds the trained neural network\n",
    "    :param prime_id: The word id to start the first prediction\n",
    "    :param int_to_vocab: Dict of word id keys to word values\n",
    "    :param token_dict: Dict of puncuation tokens keys to puncuation values\n",
    "    :param pad_value: The value used to pad a sequence\n",
    "    :param predict_len: The length of text to generate\n",
    "    :return: The generated text\n",
    "    \"\"\"\n",
    "    rnn.eval()\n",
    "    \n",
    "    # create a sequence (batch_size=1) with the prime_id\n",
    "    current_seq = np.full((1, sequence_length), pad_value)\n",
    "    current_seq[-1][-1] = prime_id\n",
    "    predicted = [int_to_vocab[prime_id]]\n",
    "    \n",
    "    for _ in range(predict_len):\n",
    "        if train_on_gpu:\n",
    "            current_seq = torch.LongTensor(current_seq).cuda()\n",
    "        else:\n",
    "            current_seq = torch.LongTensor(current_seq)\n",
    "        \n",
    "        # initialize the hidden state\n",
    "        hidden = rnn.init_hidden(current_seq.size(0))\n",
    "        \n",
    "        # get the output of the rnn\n",
    "        output, _ = rnn(current_seq, hidden)\n",
    "        \n",
    "        word = '<UNK>'\n",
    "        top_k = 10\n",
    "        \n",
    "        while word == '<UNK>':\n",
    "        \n",
    "            # get the next word probabilities\n",
    "            p = F.softmax(output, dim=1).data\n",
    "            if(train_on_gpu):\n",
    "                p = p.cpu() # move to cpu\n",
    "        \n",
    "            # use top_k sampling to get the index of the next word\n",
    "            \n",
    "            p, top_i = p.topk(top_k)\n",
    "            top_i = top_i.numpy().squeeze()\n",
    "        \n",
    "            # select the likely next word index with some element of randomness\n",
    "            p = p.numpy().squeeze()\n",
    "            word_i = np.random.choice(top_i, p=p/p.sum())\n",
    "        \n",
    "            # retrieve that word from the dictionary\n",
    "            word = int_to_vocab[word_i]\n",
    "        \n",
    "        predicted.append(word)     \n",
    "        \n",
    "        # the generated word becomes the next \"current sequence\" and the cycle can continue\n",
    "        current_seq = np.roll(current_seq, -1, 1)\n",
    "        current_seq[-1][-1] = word_i\n",
    "    \n",
    "    gen_sentences = ' '.join(predicted)\n",
    "    \n",
    "    # Replace punctuation tokens\n",
    "    for key, token in token_dict.items():\n",
    "        ending = ' ' if key in ['\\n', '(', '\"'] else ''\n",
    "        gen_sentences = gen_sentences.replace(' ' + token.lower(), key)\n",
    "    gen_sentences = gen_sentences.replace('\\n ', '\\n')\n",
    "    gen_sentences = gen_sentences.replace('( ', '(')\n",
    "    \n",
    "    # return all the sentences\n",
    "    return gen_sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a New Script\n",
    "It's time to generate the text. Set `gen_length` to the length of TV script you want to generate and set `prime_word` to one of the following to start the prediction:\n",
    "- \"jerry\"\n",
    "- \"elaine\"\n",
    "- \"george\"\n",
    "- \"kramer\"\n",
    "\n",
    "You can set the prime word to _any word_ in our dictionary, but it's best to start with a name for generating a TV script. (You can also start with any other names you find in the original text file!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kramer:(to jerry) oh god. i think i was wondering. i was just saying that. he didn't get it all over the night.\n",
      "\n",
      "jerry: yeah.\n",
      "\n",
      "george:(pause) what about the movie?\n",
      "\n",
      "elaine: well... i got a date! i'm not in this house. i'm gonna get a new car, and then, uh...(george hangs up, and starts to be a man with jerry)\n",
      "\n",
      "jerry: you have to go?\n",
      "\n",
      "elaine:(to himself) i don't want to see it.\n",
      "\n",
      "jerry: so you know what i want.\n",
      "\n",
      "george:(pause) yeah, well, i'm going back for this.\n",
      "\n",
      "jerry: yeah, well, i'm sorry, we were going to have to do a little bit.\n",
      "\n",
      "george: no, no, it's not the way.\n",
      "\n",
      "jerry: well.\n",
      "\n",
      "kramer: oh, i'm not a man of a new new york party.\n",
      "\n",
      "kramer: well, i don't know, but, i think i'm in the mood for that.\n",
      "\n",
      "george: oh, that's right, that's enough!\n",
      "\n",
      "jerry:(pause) i know, you know, i'm in a lot of people.(laughs)\n",
      "\n",
      "[setting: jerry's apartment]\n",
      "\n",
      "jerry:(to george, he turns back into the bathroom) alright, i'll take care of this!(kramer enters.)\n",
      "\n",
      "[setting: george's apartment building]\n",
      "\n",
      "estelle: oh, hi kramer.\n",
      "\n",
      "kramer:(to mickey) hey, you know, we should have been in a big mistake.\n",
      "\n",
      "jerry:(laughing) you know, i don't think so.\n",
      "\n",
      "george: what?\n",
      "\n",
      "kramer: oh, you know, i was just wondering if you want to have to go over there.\n",
      "\n",
      "kramer:(to jerry) you know, you don't have to have that, and you think i have an idea where we could do?\n",
      "\n",
      "george: i got it all.\n",
      "\n",
      "elaine:(to george) hey, george\n"
     ]
    }
   ],
   "source": [
    "# run the cell multiple times to get different results!\n",
    "gen_length = 400 # modify the length to your preference\n",
    "prime_word = 'kramer' # name for starting the script\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "pad_word = helper.SPECIAL_WORDS['PADDING']\n",
    "generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)\n",
    "print(generated_script)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "elaine: jerry, you want to talk to him about a little while, huh?\n",
      "\n",
      "elaine:(laughing) i'm a very good man. this isn't a great idea. i have to have the same time, the other way, but it's not that i can do it.\n",
      "\n",
      "susan: well, i don't care.\n",
      "\n",
      "elaine: oh, you think it's a big problem?\n",
      "\n",
      "kramer: yeah.\n",
      "\n",
      "george: oh yeah, right.\n",
      "\n",
      "jerry: i don't understand why you're not going to be able to get out of my apartment.\n",
      "\n",
      "jerry: i can't believe i don't want any money.\n",
      "\n",
      "kramer: you know.\n",
      "\n",
      "george:(pause) i didn't know what that was. you know i didn't even know it was in the house, and you have to go out to work to the movies.\n",
      "\n",
      "estelle: what is that?\n",
      "\n",
      "george:(laughing) i'm not going to take my car, so i can tell you how about the whole thing for you.\n",
      "\n",
      "jerry: oh, well, i'm just not going to go to the airport.(jerry nods and looks at george)\n",
      "\n",
      "kramer:(looking at jerry) hey, jerry.\n",
      "\n",
      "jerry: i know, i'm sorry, i can't, you know, we have to get the new card.\n",
      "\n",
      "elaine: what is it?!\n",
      "\n",
      "kramer: i think it's the guy in the hospital. i can't go in there.\n",
      "\n",
      "elaine: so, did you get that?\n",
      "\n",
      "jerry: i don't know. i don't know.\n",
      "\n",
      "george:(to george) what does she want to talk to you about?\n",
      "\n",
      "george: what is that?\n",
      "\n",
      "jerry: it's not a problem, jerry.\n",
      "\n",
      "george: well, that's not what i know. i know what i'm doing.\n",
      "\n",
      "kramer: i'm not going to be going.\n",
      "\n",
      "jerry: hey, jerry!\n",
      "\n",
      "jerry: hi, elaine!\n",
      "\n",
      "elaine: hi, mr. steinbrenner, i don't see how you know.(jerry and george)\n"
     ]
    }
   ],
   "source": [
    "gen_length = 400 # modify the length to your preference\n",
    "prime_word = \"elaine\" # name for starting the script\n",
    "pad_word = helper.SPECIAL_WORDS['PADDING']\n",
    "generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)\n",
    "print(generated_script)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "george: what am i gonna do now? you don't wanna have a big deal with your mother.\n",
      "\n",
      "elaine: oh, come on.\n",
      "\n",
      "jerry: no, don't you?\n",
      "\n",
      "kramer: oh, yeah.\n",
      "\n",
      "jerry:(to jerry) hey, you know, i gotta be honest with you.\n",
      "\n",
      "jerry: yeah yeah, you don't have to be the best, you wanna know, you know how they could have.\n",
      "\n",
      "morty: i thought you might never be going with me.\n",
      "\n",
      "elaine:(to jerry) i think you're going to make a few people in the middle of my building.\n",
      "\n",
      "jerry: i thought you were going to be here with her in an accident!\n",
      "\n",
      "elaine: i mean, you know what i was going on here?\n",
      "\n",
      "jerry: i'm going to have to go in the hospital.\n",
      "\n",
      "kramer: well, you're a big salad.\n",
      "\n",
      "frank: what?\n",
      "\n",
      "george: i don't know, you don't think i can...\n",
      "\n",
      "jerry: i don't know. you know, i have to do this.\n",
      "\n",
      "susan: hey, i gotta talk to elaine about it, and i have a little more thing. i can't believe i can't get a big deal.\n",
      "\n",
      "george: what do you mean?\n",
      "\n",
      "jerry: you know, i'm just going to be a little nervous about you.\n",
      "\n",
      "george:(smiling, to jerry) so what are you talking about? i'm a big salad. it's not a very good guy.\n",
      "\n",
      "helen: what was you going to do?\n",
      "\n",
      "jerry: oh my god.\n",
      "\n",
      "george: yeah, the guy was gonna be here?\n",
      "\n",
      "jerry: i think he's a little nervous.(elaine is looking in his apartment.) hey, hey. what about the guy in the city?\n",
      "\n",
      "elaine: i was thinking, i'm sorry.\n",
      "\n",
      "jerry: oh, i'm a comedian...\n",
      "\n",
      "george: no, no. it wasn't good.\n",
      "\n",
      "kramer: you know, if you had to go back\n"
     ]
    }
   ],
   "source": [
    "gen_length = 400 # modify the length to your preference\n",
    "prime_word = 'george' # name for starting the script\n",
    "pad_word = helper.SPECIAL_WORDS['PADDING']\n",
    "generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)\n",
    "print(generated_script)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jerry: what about the car.\n",
      "\n",
      "george: i'm not taking this. i'm gonna get it to the office. i don't need that job.\n",
      "\n",
      "jerry: what?\n",
      "\n",
      "newman: well it's not the way that i am.\n",
      "\n",
      "jerry: what did you think?\n",
      "\n",
      "george:(to george) hey, i'm sorry, jerry! you got my name! what kind of a big deal are you? i'm not gonna be able to get it down at the office.\n",
      "\n",
      "jerry:(to jerry) hey.\n",
      "\n",
      "george: you know what i want to do?\n",
      "\n",
      "kramer: well i'm not going to do this, i just wanted to tell you that i had it for some time.\n",
      "\n",
      "jerry: what? how are we gonna be in the car? i don't know what they can do.\n",
      "\n",
      "kramer: well, you want to talk to him?\n",
      "\n",
      "jerry: no, i just thought it would be nice.\n",
      "\n",
      "george: what are you gonna do now?\n",
      "\n",
      "jerry: i'm not really happy.\n",
      "\n",
      "george: what?\n",
      "\n",
      "elaine:(still worried) well, we have to do it!\n",
      "\n",
      "kramer: yeah. well, i don't want to go.\n",
      "\n",
      "jerry: what?\n",
      "\n",
      "george: i told you that was the only way to go in a long time.\n",
      "\n",
      "jerry: oh, no.\n",
      "\n",
      "elaine: i don't know, what if we are gonna be a kid?\n",
      "\n",
      "george: what?\n",
      "\n",
      "jerry: no.\n",
      "\n",
      "george: i think she might be a little bit more than you. i mean i don't know about it. what is this?\n",
      "\n",
      "kramer:(to george) yeah, it's the only way it could do. so, if you don't have any time, you're gonna tell the guy. it's not a little bit of a very strange person to get to get to work.\n",
      "\n",
      "george: what are you doing?\n",
      "\n",
      "george: what? why?\n",
      "\n",
      "jerry: why don't you just go?\n",
      "\n",
      "elaine:(smiling) what is\n"
     ]
    }
   ],
   "source": [
    "gen_length = 400 # modify the length to your preference\n",
    "prime_word = 'jerry' # name for starting the script\n",
    "pad_word = helper.SPECIAL_WORDS['PADDING']\n",
    "generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)\n",
    "print(generated_script)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save your favorite scripts\n",
    "\n",
    "Once you have a script that you like (or find interesting), save it to a text file!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run the cell multiple times to get different results!\n",
    "gen_length = 400 # modify the length to your preference\n",
    "prime_word = 'kramer' # name for starting the script\n",
    "pad_word = helper.SPECIAL_WORDS['PADDING']\n",
    "\n",
    "for i in range(100):\n",
    "    generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)\n",
    "    # save script to a text file\n",
    "    f =  open(f'generated_scripts/generated_script_{i:03}.txt',\"w\")\n",
    "    f.write(generated_script)\n",
    "    f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The TV Script is Not Perfect\n",
    "It's ok if the TV script doesn't make perfect sense. It should look like alternating lines of dialogue, here is one such example of a few generated lines.\n",
    "\n",
    "### Example generated script\n",
    "\n",
    ">jerry: what about me?\n",
    ">\n",
    ">jerry: i don't have to wait.\n",
    ">\n",
    ">kramer:(to the sales table)\n",
    ">\n",
    ">elaine:(to jerry) hey, look at this, i'm a good doctor.\n",
    ">\n",
    ">newman:(to elaine) you think i have no idea of this...\n",
    ">\n",
    ">elaine: oh, you better take the phone, and he was a little nervous.\n",
    ">\n",
    ">kramer:(to the phone) hey, hey, jerry, i don't want to be a little bit.(to kramer and jerry) you can't.\n",
    ">\n",
    ">jerry: oh, yeah. i don't even know, i know.\n",
    ">\n",
    ">jerry:(to the phone) oh, i know.\n",
    ">\n",
    ">kramer:(laughing) you know...(to jerry) you don't know.\n",
    "\n",
    "You can see that there are multiple characters that say (somewhat) complete sentences, but it doesn't have to be perfect! It takes quite a while to get good results, and often, you'll have to use a smaller vocabulary (and discard uncommon words), or get more data.  The Seinfeld dataset is about 3.4 MB, which is big enough for our purposes; for script generation you'll want more than 1 MB of text, generally. \n",
    "\n",
    "# Submitting This Project\n",
    "When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as \"dlnd_tv_script_generation.ipynb\" and save another copy as an HTML file by clicking \"File\" -> \"Download as..\"->\"html\". Include the \"helper.py\" and \"problem_unittests.py\" files in your submission. Once you download these files, compress them into one zip file for submission."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}