Notebooks/6_NLP/Textual_Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Data Science\n",
    "\n",
    "### Predictive Analysis - Textual data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Text Vectorization Techniques\n",
    "#### Bag Of Words, Topic Modeling and Word2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.kaggle.com/c/word2vec-nlp-tutorial  \n",
    "https://github.com/wendykan/DeepLearningMovies  \n",
    "http://fastml.com/classifying-text-with-bag-of-words-a-tutorial/  \n",
    "\n",
    "\n",
    "In this tutorial competition, we dig a little \"deeper\" into sentiment analysis. Google's Word2Vec is a deep-learning inspired method that focuses on the meaning of words. Word2Vec attempts to understand meaning and semantic relationships among words. It works in a way that is similar to deep approaches, such as recurrent neural nets or deep neural nets, but is computationally more efficient. This tutorial focuses on Word2Vec for sentiment analysis.\n",
    "\n",
    "Sentiment analysis is a challenging subject in machine learning. People express their emotions in language that is often obscured by sarcasm, ambiguity, and plays on words, all of which could be very misleading for both humans and computers. There's another Kaggle competition for movie review sentiment analysis. In this tutorial we explore how Word2Vec can be applied to a similar problem.\n",
    "\n",
    "Deep learning has been in the news a lot over the past few years, even making it to the front page of the New York Times. These machine learning techniques, inspired by the architecture of the human brain and made possible by recent advances in computing power, have been making waves via breakthrough results in image recognition, speech processing, and natural language tasks. Recently, deep learning approaches won several Kaggle competitions, including a drug discovery task, and cat and dog image recognition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's charge the batteries for our analysis..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import time\n",
    "import pickle\n",
    "from zipfile import ZipFile\n",
    "\n",
    "import string\n",
    "import re\n",
    "\n",
    "import logging\n",
    "import warnings\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import pylab\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.font_manager import FontProperties\n",
    "\n",
    "import nltk\n",
    "import nltk.data\n",
    "from nltk.corpus import stopwords\n",
    "from nltk import word_tokenize\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "from sklearn import model_selection\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer \n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.metrics import roc_auc_score as AUC\n",
    "\n",
    "import gensim\n",
    "from gensim import corpora\n",
    "from gensim import models\n",
    "from gensim import similarities\n",
    "from gensim.models import Word2Vec\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "#logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data Set  \n",
    "--\n",
    "\n",
    "The labeled data set consists of 50,000 IMDB movie reviews, specially selected for sentiment analysis. The sentiment of reviews is binary, meaning the IMDB rating < 5 results in a sentiment score of 0, and rating >=7 have a sentiment score of 1. No individual movie has more than 30 reviews. The 25,000 review labeled training set does not include any of the same movies as the 25,000 review test set. In addition, there are another 50,000 IMDB reviews provided without any rating labels.\n",
    "\n",
    "File descriptions\n",
    "\n",
    "labeledTrainData - The labeled training set. The file is tab-delimited and has a header row followed by 25,000 rows containing an id, sentiment, and text for each review.  \n",
    "\n",
    "testData - The test set. The tab-delimited file has a header row followed by 25,000 rows containing an id and text for each review. Your task is to predict the sentiment for each one. \n",
    "\n",
    "unlabeledTrainData - An extra training set with no labels. The tab-delimited file has a header row followed by 50,000 rows containing an id and text for each review. \n",
    "\n",
    "sampleSubmission - A comma-delimited sample submission file in the correct format.\n",
    "Data fields\n",
    "\n",
    "id - Unique ID of each review  \n",
    "sentiment - Sentiment of the review; 1 for positive reviews and 0 for negative reviews  \n",
    "review - Text of the review  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the dataset:  \n",
    "--"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "datapath = \"../datasets/Kaggle/BOWPopcorn/\"\n",
    "outputs = \"../outputs/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "with ZipFile(os.path.join(datapath, 'BOW_labeledTrainData.zip'), 'r') as myzip:\n",
    "    with myzip.open('BOW_labeledTrainData.tsv') as myfile:\n",
    "        train = pd.read_csv(myfile, header=0, delimiter=\"\\t\", quoting=3)\n",
    "        \n",
    "with ZipFile(os.path.join(datapath, 'BOW_testData.zip'), 'r') as myzip:\n",
    "    with myzip.open('BOW_testData.tsv') as myfile:\n",
    "        test = pd.read_csv(myfile, header=0, delimiter=\"\\t\", quoting=3)\n",
    "        \n",
    "with ZipFile(os.path.join(datapath, 'BOW_unlabeledTrainData.zip'), 'r') as myzip:\n",
    "    with myzip.open('BOW_unlabeledTrainData.tsv') as myfile:\n",
    "        unlabeled_train = pd.read_csv(myfile, header=0, delimiter=\"\\t\", quoting=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read 25000 labeled train reviews, 25000 labeled test reviews, and 50000 unlabeled reviews\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Read {} labeled train reviews, \\\n",
    "{} labeled test reviews, and \\\n",
    "{} unlabeled reviews\\n\".format(train[\"review\"].size,\n",
    "                               test[\"review\"].size,\n",
    "                               unlabeled_train[\"review\"].size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"5814_8\"</td>\n",
       "      <td>1</td>\n",
       "      <td>\"With all this stuff going down at the moment ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"2381_9\"</td>\n",
       "      <td>1</td>\n",
       "      <td>\"\\\"The Classic War of the Worlds\\\" by Timothy ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"7759_3\"</td>\n",
       "      <td>0</td>\n",
       "      <td>\"The film starts with a manager (Nicholas Bell...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"3630_4\"</td>\n",
       "      <td>0</td>\n",
       "      <td>\"It must be assumed that those who praised thi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"9495_8\"</td>\n",
       "      <td>1</td>\n",
       "      <td>\"Superbly trashy and wondrously unpretentious ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id  sentiment                                             review\n",
       "0  \"5814_8\"          1  \"With all this stuff going down at the moment ...\n",
       "1  \"2381_9\"          1  \"\\\"The Classic War of the Worlds\\\" by Timothy ...\n",
       "2  \"7759_3\"          0  \"The film starts with a manager (Nicholas Bell...\n",
       "3  \"3630_4\"          0  \"It must be assumed that those who praised thi...\n",
       "4  \"9495_8\"          1  \"Superbly trashy and wondrously unpretentious ..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 25000 entries, 0 to 24999\n",
      "Data columns (total 3 columns):\n",
      "id           25000 non-null object\n",
      "sentiment    25000 non-null int64\n",
      "review       25000 non-null object\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 586.0+ KB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>25000.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.50000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.50001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.50000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         sentiment\n",
       "count  25000.00000\n",
       "mean       0.50000\n",
       "std        0.50001\n",
       "min        0.00000\n",
       "25%        0.00000\n",
       "50%        0.50000\n",
       "75%        1.00000\n",
       "max        1.00000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part 1: For Beginners - Bag of Words\n",
    "--\n",
    "https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-1-for-beginners-bag-of-words  \n",
    "\n",
    "What is NLP?\n",
    "\n",
    "NLP (Natural Language Processing) is a set of techniques for approaching text problems. This page will help you get started with loading and cleaning the IMDB movie reviews, then applying a simple Bag of Words model to get surprisingly accurate predictions of whether a review is thumbs-up or thumbs-down."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Text_Cleaning_Utilities(object):\n",
    "    \"\"\"Tools for processing text into segments for further learning\"\"\"\n",
    "\n",
    "    @staticmethod\n",
    "    def text_to_wordlist(text, \n",
    "                         remove_stopwords=False, \n",
    "                         remove_html=False, \n",
    "                         remove_non_letters=False, \n",
    "                         steeming=False):\n",
    "        '''Split a text into a list of words'''\n",
    "        #text = text.replace('-\\n','')\n",
    "        text = text.lower()\n",
    "        if remove_html:\n",
    "            text = BeautifulSoup(text, \"html5lib\").get_text()\n",
    "        if remove_non_letters:\n",
    "            text = re.sub(\"[^-A-Za-z0-9_]\", \" \", text)\n",
    "        list_words = word_tokenize(text)\n",
    "        list_words = [w.strip(string.punctuation) for w in list_words if w not in string.punctuation]\n",
    "        list_words = [w for w in list_words if len(w) > 1]\n",
    "        if remove_stopwords:\n",
    "            stops = set(stopwords.words(\"english\"))\n",
    "            list_words = [w for w in list_words if w not in stops]\n",
    "        if steeming:\n",
    "            stemmer = PorterStemmer()\n",
    "            list_words = [stemmer.stem(item) for item in list_words]\n",
    "        return list_words\n",
    "    \n",
    "    @staticmethod\n",
    "    def df_to_list_of_texts(dataframe, column, \n",
    "                            remove_stopwords=False, \n",
    "                            remove_html=False, \n",
    "                            remove_non_letters=False, \n",
    "                            steeming=False):\n",
    "        clean_texts = []\n",
    "        for txt_id in range(len(dataframe[column])):\n",
    "            clean_texts.append(' '.join(Text_Cleaning_Utilities.text_to_wordlist(dataframe[column][txt_id],\n",
    "                                                                                 remove_stopwords=remove_stopwords,\n",
    "                                                                                 remove_html=remove_html,\n",
    "                                                                                 remove_non_letters=remove_non_letters,\n",
    "                                                                                 steeming=steeming)))\n",
    "            \n",
    "        return clean_texts\n",
    "\n",
    "    @staticmethod\n",
    "    def df_to_list_of_tokens(dataframe, column, \n",
    "                             remove_stopwords=False, \n",
    "                             remove_html=False, \n",
    "                             remove_non_letters=False, \n",
    "                             steeming=False):\n",
    "        clean_texts = []\n",
    "        for txt_id in range(len(dataframe[column])):\n",
    "            clean_texts.append(Text_Cleaning_Utilities.text_to_wordlist(dataframe[column][txt_id],\n",
    "                                                                        remove_stopwords=remove_stopwords,\n",
    "                                                                        remove_html=remove_html,\n",
    "                                                                        remove_non_letters=remove_non_letters,\n",
    "                                                                        steeming=steeming))\n",
    "            \n",
    "        return clean_texts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cleaning all the datasets and getting word lists\n",
    "--\n",
    "first set is without stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_train_reviews = Text_Cleaning_Utilities.df_to_list_of_texts(train, \n",
    "                                                                  'review', \n",
    "                                                                  remove_stopwords=True,\n",
    "                                                                  remove_html=True,)\n",
    "clean_test_reviews = Text_Cleaning_Utilities.df_to_list_of_texts(test, \n",
    "                                                                 'review', \n",
    "                                                                 remove_stopwords=True,\n",
    "                                                                 remove_html=True,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'stuff going moment mj started listening music watching odd documentary watched w'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_train_reviews[0][0:80]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'naturally film main themes mortality nostalgia loss innocence perhaps surprising'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_test_reviews[0][0:80]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second set mantains stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_train_reviews_sw = Text_Cleaning_Utilities.df_to_list_of_texts(train, \n",
    "                                                                     'review',\n",
    "                                                                     remove_html=True,)\n",
    "\n",
    "clean_test_reviews_sw = Text_Cleaning_Utilities.df_to_list_of_texts(test, \n",
    "                                                                    'review',\n",
    "                                                                    remove_html=True,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'with all this stuff going down at the moment with mj ve started listening to his'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_train_reviews_sw[0][0:80]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'naturally in film who main themes are of mortality nostalgia and loss of innocen'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_test_reviews_sw[0][0:80]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(outputs, 'clean_reviews.pkl'),'wb') as f:\n",
    "    pickle.dump((clean_train_reviews, \n",
    "                 clean_test_reviews,\n",
    "                 clean_train_reviews_sw, \n",
    "                 clean_test_reviews_sw),f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading Pickle  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(outputs, 'clean_reviews.pkl'),'rb') as f:\n",
    "    (clean_train_reviews, \n",
    "     clean_test_reviews,\n",
    "     clean_train_reviews_sw,\n",
    "     clean_test_reviews_sw) = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating Features from a Bag of Words (Using scikit-learn)\n",
    "--\n",
    "\n",
    "Now that we have our training reviews tidied up, how do we convert them to some kind of numeric representation for machine learning? One common approach is called a Bag of Words. The Bag of Words model learns a vocabulary from all of the documents, then models each document by counting the number of times each word appears. For example, consider the following two sentences:\n",
    "\n",
    "Sentence 1: \"The cat sat on the hat\"  \n",
    "Sentence 2: \"The dog ate the cat and the hat\"  \n",
    "\n",
    "From these two sentences, our vocabulary is as follows:\n",
    "\n",
    "{ the, cat, sat, on, hat, dog, ate, and }\n",
    "\n",
    "To get our bags of words, we count the number of times each word occurs in each sentence. In Sentence 1, \"the\" appears twice, and \"cat\", \"sat\", \"on\", and \"hat\" each appear once, so the feature vector for Sentence 1 is:\n",
    "\n",
    "{ the, cat, sat, on, hat, dog, ate, and }\n",
    "\n",
    "Sentence 1: [ 2, 1, 1, 1, 1, 0, 0, 0 ]\n",
    "\n",
    "Similarly, the features for Sentence 2 are: [ 3, 1, 0, 0, 1, 1, 1, 1]\n",
    "\n",
    "In the IMDB data, we have a very large number of reviews, which will give us a large vocabulary. To limit the size of the feature vectors, we should choose some maximum vocabulary size. Below, we use the 5000 most frequent words (remembering that stop words have already been removed).\n",
    "\n",
    "We'll be using the feature_extraction module from scikit-learn to create bag-of-words features.  \n",
    "We will test two strategies: CountVectorizer (term frequecies - TF) and TFIDF Vectorizer:  \n",
    "First we'll start with plain word counts (TF):  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the \"CountVectorizer\" object, which is scikit-learn's bag of words tool.\n",
    "#http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html\n",
    "vectorizer_tf = CountVectorizer(input='content', \n",
    "                               encoding='utf-8', \n",
    "                               decode_error='strict', \n",
    "                               strip_accents=None, \n",
    "                               lowercase=True, \n",
    "                               preprocessor=None, \n",
    "                               tokenizer=None, \n",
    "                               stop_words=None, \n",
    "                               #token_pattern='(?u)\\b\\w\\w+\\b',\n",
    "                               ngram_range=(1, 2),\n",
    "                               analyzer='word', \n",
    "                               max_df=1.0, \n",
    "                               min_df=1, \n",
    "                               max_features=5000, \n",
    "                               vocabulary=None, \n",
    "                               binary=False, \n",
    "                               dtype=np.int64,\n",
    "                              )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit_transform() does two functions: First, it fits the model and learns the vocabulary; \n",
    "second, it transforms our training data into feature vectors. \n",
    "The input to fit_transform should be a list of strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 5000)\n"
     ]
    }
   ],
   "source": [
    "train_data_features_tf = vectorizer_tf.fit_transform(clean_train_reviews)\n",
    "train_data_features_tf = train_data_features_tf.toarray() # Numpy arrays are easy to work with\n",
    "print(train_data_features_tf.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 5000)\n"
     ]
    }
   ],
   "source": [
    "test_data_features_tf = vectorizer_tf.fit_transform(clean_test_reviews)\n",
    "test_data_features_tf = test_data_features_tf.toarray() # Numpy arrays are easy to work with\n",
    "print(test_data_features_tf.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to use TfIDf vectors and the train/test cleaned reviews with stopwords:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html\n",
    "#Another approach using TfIDf vectorizer and using the texts with stopwords in:\n",
    "#https://github.com/zygmuntz/classifying-text/blob/master/bow_predict.py \n",
    "vectorizer_tfidf = TfidfVectorizer(input='content',\n",
    "                                  #encoding='utf-8',\n",
    "                                  decode_error='strict',\n",
    "                                  strip_accents=None,\n",
    "                                  lowercase=True,\n",
    "                                  preprocessor=None,\n",
    "                                  tokenizer=None,\n",
    "                                  analyzer='word',\n",
    "                                  stop_words=None,\n",
    "                                  #token_pattern='(?u)\\b\\w\\w+\\b',\n",
    "                                  ngram_range=(1, 2),\n",
    "                                  max_df=1.0,\n",
    "                                  min_df=1,\n",
    "                                  max_features=5000,\n",
    "                                  vocabulary=None, \n",
    "                                  binary=False, \n",
    "                                  dtype=np.int64,\n",
    "                                  norm='l2',\n",
    "                                  use_idf=True,\n",
    "                                  smooth_idf=True,\n",
    "                                  sublinear_tf=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 5000)\n"
     ]
    }
   ],
   "source": [
    "train_data_features_tfidf = vectorizer_tfidf.fit_transform(clean_train_reviews_sw)\n",
    "train_data_features_tfidf = train_data_features_tfidf.toarray() # Numpy arrays are easy to work with\n",
    "print(train_data_features_tfidf.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 5000)\n"
     ]
    }
   ],
   "source": [
    "test_data_features_tfidf = vectorizer_tfidf.fit_transform(clean_test_reviews_sw)\n",
    "test_data_features_tfidf = test_data_features_tfidf.toarray() # Numpy arrays are easy to work with\n",
    "print(test_data_features_tfidf.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(outputs, 'train_test_data_features.pkl'),'wb') as f:\n",
    "    pickle.dump((train_data_features_tf, \n",
    "                 test_data_features_tf,\n",
    "                 train_data_features_tfidf,\n",
    "                 test_data_features_tfidf),f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(outputs, 'train_test_data_features.pkl'),'rb') as f:\n",
    "    (train_data_features_tf, \n",
    "    test_data_features_tf,\n",
    "    train_data_features_tfidf,\n",
    "    test_data_features_tfidf) = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dividing Train set for Cross Validation  \n",
    "--"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://github.com/zygmuntz/classifying-text/blob/master/bow_validate.py  \n",
    "Alternatively, we can use the indexes to divide the train samples  \n",
    "\n",
    "train_i, test_i = train_test_split(np.arange(len(train)), train_size = 0.8, random_state = 44)  \n",
    "\n",
    "After generating indexes, we can divide ou datasets:  \n",
    "traincv = train_data_features1[train_i]  \n",
    "testcv = train_data_features1[test_i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Plain Word Counts\n",
    "X_traincv_tf, X_testcv_tf, y_traincv_tf, y_testcv_tf = model_selection.train_test_split(train_data_features_tf,\n",
    "                                                                                        train[\"sentiment\"],\n",
    "                                                                                        test_size=0.2,\n",
    "                                                                                        random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TfIdf\n",
    "(X_traincv_tfidf, \n",
    " X_testcv_tfidf, \n",
    " y_traincv_tfidf, \n",
    " y_testcv_tfidf) = model_selection.train_test_split(train_data_features_tfidf,\n",
    "                                                    train[\"sentiment\"],\n",
    "                                                    test_size=0.2,\n",
    "                                                    random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training some Classifiers  \n",
    "--"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we have numeric training features from the Bag of Words and the original sentiment labels for each feature vector, so let's do some supervised learning! Here, we'll use some classifiers implementations included in  the scikit-learn package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a Random Forest classifier with 300 trees\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "clf_RF_tf = RandomForestClassifier(n_estimators=300, \n",
    "                                   criterion='gini', \n",
    "                                   max_depth=None, \n",
    "                                   min_samples_split=2, \n",
    "                                   min_samples_leaf=1, \n",
    "                                   min_weight_fraction_leaf=0.0, \n",
    "                                   max_features='auto', \n",
    "                                   max_leaf_nodes=None, \n",
    "                                   bootstrap=False, \n",
    "                                   oob_score=False, \n",
    "                                   n_jobs=-1, \n",
    "                                   random_state=0, \n",
    "                                   verbose=0, \n",
    "                                   warm_start=False, \n",
    "                                   class_weight=None).fit(X_traincv_tf, y_traincv_tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8608\n"
     ]
    }
   ],
   "source": [
    "eval_RF_tf_tts = clf_RF_tf.score(X_testcv_tf, y_testcv_tf)\n",
    "print(eval_RF_tf_tts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.45666667, 0.54333333],\n",
       "       [0.37333333, 0.62666667],\n",
       "       [0.35      , 0.65      ],\n",
       "       ...,\n",
       "       [0.32      , 0.68      ],\n",
       "       [0.81666667, 0.18333333],\n",
       "       [0.22666667, 0.77333333]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_RF_tf.predict_proba(X_testcv_tf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are going to train on the TfIdf samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a Random Forest classifier with 300 trees\n",
    "clf_RF_tfidf = RandomForestClassifier(n_estimators=300, \n",
    "                                      criterion='gini', \n",
    "                                      max_depth=None, \n",
    "                                      min_samples_split=2, \n",
    "                                      min_samples_leaf=1, \n",
    "                                      min_weight_fraction_leaf=0.0, \n",
    "                                      max_features='auto', \n",
    "                                      max_leaf_nodes=None, \n",
    "                                      bootstrap=False, \n",
    "                                      oob_score=False, \n",
    "                                      n_jobs=-1, \n",
    "                                      random_state=0, \n",
    "                                      verbose=0, \n",
    "                                      warm_start=False, \n",
    "                                      class_weight=None).fit(X_traincv_tfidf, y_traincv_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8536\n"
     ]
    }
   ],
   "source": [
    "eval_RF_tfidf_tts = clf_RF_tfidf.score(X_testcv_tfidf, y_testcv_tfidf)\n",
    "print(eval_RF_tfidf_tts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.42333333, 0.57666667],\n",
       "       [0.36666667, 0.63333333],\n",
       "       [0.33      , 0.67      ],\n",
       "       ...,\n",
       "       [0.25666667, 0.74333333],\n",
       "       [0.75333333, 0.24666667],\n",
       "       [0.33      , 0.67      ]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_RF_tfidf.predict_proba(X_testcv_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logistic Regression  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\n",
    "from sklearn.linear_model import LogisticRegression as LR\n",
    "\n",
    "clf_LR_tf = LR(penalty='l2',\n",
    "               dual=False,\n",
    "               tol=0.0001,\n",
    "               C=1.0,\n",
    "               fit_intercept=True,\n",
    "               intercept_scaling=1,\n",
    "               class_weight=None,\n",
    "               random_state=0,\n",
    "               solver='liblinear',\n",
    "               max_iter=100,\n",
    "               multi_class='ovr',\n",
    "               verbose=0).fit(X_traincv_tf, y_traincv_tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8518\n"
     ]
    }
   ],
   "source": [
    "eval_LR_tf_tts = clf_LR_tf.score(X_testcv_tf, y_testcv_tf)\n",
    "print(eval_LR_tf_tts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_LR_tfidf = LR(penalty='l2',\n",
    "                  dual=False,\n",
    "                  tol=0.0001,\n",
    "                  C=1.0,\n",
    "                  fit_intercept=True,\n",
    "                  intercept_scaling=1,\n",
    "                  class_weight=None,\n",
    "                  random_state=0,\n",
    "                  solver='liblinear',\n",
    "                  max_iter=100,\n",
    "                  multi_class='ovr',\n",
    "                  verbose=0).fit(X_traincv_tfidf, y_traincv_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8926\n"
     ]
    }
   ],
   "source": [
    "eval_LR_tfidf_tts = clf_LR_tfidf.score(X_testcv_tfidf, y_testcv_tfidf)\n",
    "print(eval_LR_tfidf_tts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regularized Greedy Forests "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rgf.sklearn import RGFClassifier, FastRGFClassifier\n",
    "\n",
    "clf_RGF_tf = RGFClassifier(max_leaf=240,\n",
    "                           algorithm=\"RGF_Sib\",\n",
    "                           test_interval=100,\n",
    "                           verbose=False,).fit(X_traincv_tf, y_traincv_tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.814\n"
     ]
    }
   ],
   "source": [
    "eval_RGF_tf_tts = clf_RGF_tf.score(X_testcv_tf, y_testcv_tf)\n",
    "print(eval_RGF_tf_tts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_RGF_tfidf = RGFClassifier(max_leaf=240,\n",
    "                              algorithm=\"RGF_Sib\",\n",
    "                              test_interval=100,\n",
    "                              verbose=False,).fit(X_traincv_tfidf, y_traincv_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8112\n"
     ]
    }
   ],
   "source": [
    "eval_RGF_tfidf_tts = clf_RGF_tfidf.score(X_testcv_tfidf, y_testcv_tfidf)\n",
    "print(eval_RGF_tfidf_tts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's do some voting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "\n",
    "clf_vot_tf = VotingClassifier(estimators=[('rf', clf_RF_tf),\n",
    "                                          ('lr', clf_LR_tf),\n",
    "                                          ('rgf', clf_RGF_tf)], voting='soft').fit(X_traincv_tf, y_traincv_tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8666\n"
     ]
    }
   ],
   "source": [
    "eval_vot_tf_tts = clf_vot_tf.score(X_testcv_tf, y_testcv_tf)\n",
    "print(eval_vot_tf_tts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_vot_tfidf = VotingClassifier(estimators=[('rf', clf_RF_tfidf),\n",
    "                                             ('lr', clf_LR_tfidf),\n",
    "                                             ('rgf', clf_RGF_tfidf)], voting='soft').fit(X_traincv_tfidf, y_traincv_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8724\n"
     ]
    }
   ],
   "source": [
    "eval_vot_tfidf_tts = clf_vot_tfidf.score(X_testcv_tfidf, y_testcv_tfidf)\n",
    "print(eval_vot_tfidf_tts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving the trained classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(outputs, 'classifiers.pkl'),'wb') as f:\n",
    "    pickle.dump((clf_RF_tf, eval_RF_tf_tts,\n",
    "                 clf_RF_tfidf, eval_RF_tfidf_tts,\n",
    "                 clf_LR_tf, eval_LR_tf_tts,\n",
    "                 clf_LR_tfidf, eval_LR_tfidf_tts,                \n",
    "                 clf_RGF_tf, eval_RGF_tf_tts,\n",
    "                 clf_RGF_tfidf, eval_RGF_tfidf_tts,\n",
    "                 clf_vot_tf, eval_vot_tf_tts,\n",
    "                 clf_vot_tfidf, eval_vot_tfidf_tts),f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the classifiers from Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(outputs, 'classifiers.pkl'),'rb') as f:\n",
    "    (clf_RF_tf, eval_RF_tf_tts,\n",
    "     clf_RF_tfidf, eval_RF_tfidf_tts,\n",
    "     clf_LR_tf, eval_LR_tf_tts,\n",
    "     clf_LR_tfidf, eval_LR_tfidf_tts,                \n",
    "     clf_RGF_tf, eval_RGF_tf_tts,\n",
    "     clf_RGF_tfidf, eval_RGF_tfidf_tts,\n",
    "     clf_vot_tf, eval_vot_tf_tts,\n",
    "     clf_vot_tfidf, eval_vot_tfidf_tts) = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtoAAAIqCAYAAAAafO0qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3X+cV2Wd///HS35IZP5I0ZUZSAlEh+SHDgrpVsTq6PBxNH/MgsXKUlqGmatmtBmxrn4iDakPWq1Ei2lfWNraoBXxB4mZqwIK+QOSMSGY0VXQhJSQZri+f7zfTDOAMNpcjAOP++3GzXOuc53zfp33dOv2nGuuc51IKSFJkiSpde3X1gVIkiRJeyODtiRJkpSBQVuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQl7bUiYmJE3Jnx+s9ExMeK2xER/x4Rf4iIRRHxtxHxbK7Pbo8i4vqIWB8R/9vWtUjSnmDQltSuRcSFEbEkIl6PiBcj4u6IOHVPfHZKqV9KaWFx91TgNKA0pXRSSumhlFLfv/YzImJ1RPzdX3udt7j2xIiYmOPaO/msnsBVQFlK6W/2xGdKUlszaEtqtyLiSuDbwP8FjgB6At8Fzm6Dcj4ArE4pvdEGn71TEdGxrWuAxjp6Aq+klF5+h+dLUrtj0JbULkXEQcB1wLiU0s9SSm+klP6cUvpFSulLb3HOTyLifyNiQ0T8KiL6NTlWGRHLI+KPEVEXEVcX2w+LiP+OiNci4tWIeCgi9iseWx0RfxcRnwZ+AAwtjqz/S0R8LCJqm1y/R0T8LCLWRcQrEXFLsf2DEfHLYtv6iPhxRBxcPHYHhYD6i+J1rym2VxWnrbwWEQsj4rgmn7M6Ir4cEU8Cb0REx+J+XfHeno2I4S34fsdExMMRcUvx+/pt0/Mi4qCImF78K0JdcVpIh+3OnRIRrwALgfuA7sX7mPEO72N1RHwpIp6MiDeKn39E8a8Yf4yI+yPikBb+vGdExK0RcVfx3Mci4oNNjveLiPuKP/OXIuKfi+37RcT4iPhd8Wc2OyLev7vvU9K+yaAtqb0aCnQB/uttnHM30Ac4HHgC+HGTY9OBz6aU3gd8CPhlsf0qoBboRmHU/J+B1PSiKaXpwOeAR1JKB6SUvt70eDGA/jfwe+AooASYte0w8A2gO3Ac0AOYWLzuaGANcFbxujdGxDHATOCKYk3zKATxzk0+chQwAjgY+CBwGTC4eG8VwOri9SemlCbu4vs6GfgdcBjwdeBnTULlDKAe6A0MAk4HPrPduc8Xv7PTgDOBF4r3Mebt3kdKqb7Ydl7xescAZ1H4mf5z8Rr7AZc3OX9XP2+AkcC/AIcAzwE3AETE+4D7gfkUfi69gQXFc74AnAN8tHjsD8Ctb/kNStqnGbQltVeHAuubBLDdSin9MKX0x5TSmxTC7IDiyDjAn4GyiDgwpfSHlNITTdqPBD5QHDF/KKWUdrz6Lp1EIZR9qTjyvjml9OtiTc+llO5LKb2ZUloH3EwhxL2VvwfuKp7zZ+BbwHuADzfp8/9SSmtTSn8CGoD9i/fWKaW0OqX0uxbW/TLw7eJ9/wfwLDAiIo4AKoErivfzMjCFQnDd5oWU0tSUUn2xjr/2PraZmlJ6KaVUBzwEPJZSWppS2kzhl65B2zru5ucN8F8ppUXF/w39GBhYbP8/wP+mlCYXf1Z/TCk9Vjz2OeCrKaXaJtc93+ktknbGoC2pvXoFOKylASciOkTEpOKf/DdSHNWlMFoLhZHSSuD3EfFgRAwttt9EYbTz3oh4PiLGv4NaewC/39kvBcWpD7OK0y82Anc2qWlnulMYGQcgpbQVWEthlHybtU2OP0dh1Hgi8HLxs7q3sO667X6p+H3x8z8AdAJeLE77eA34NwojxzvU0Br30cRLTbb/tJP9A6BFP2+ApqufbNp2LoWf11v9MvIB4L+a3PcKCr/MHPEW/SXtwwzaktqrR4A3KfwZvyUupPCQ5N8BB1GYwgGFqRuklBanlM6mEBZ/Dswutv8xpXRVSqkXUAVc2ZI5zttZC/R8i18K/i+FqSjHp5QOBD61raai7UfPX6AQ9grFRwSFYFj3VueklP6/lNKpxfMS8M0W1l1SvP42PYufv5bCd39YSung4r8DU0r9mvTd3aj/276Pt2mXP+/dWAv02sWxM5vc98EppS7FEXZJasagLaldSiltACYAt0bEORHRNSI6RcSZEXHjTk55H4Vw+ArQlULABSAiOkfEJyPioOI0ho3A1uKx/xMRvYtBcAOF0cutb7PcRcCLwKSIeG9EdImIU5rU9TqwISJKgO0f5HyJ5qFvNoXpG8MjohOFOeRvAv+zsw+OiL4R8fGI2B/YTGHUt6X1Hw5cXvxeL6Awh3xeSulF4F5gckQcWHxA8IMRsaspL9t7W/fxDrzlz7sF/hs4MiKuiIj9I+J9EXFy8dj3gRsi4gMAEdEtItpilRtJ7YBBW1K7lVKaDFwJXAusozDaeBmFEent/YjCVIU6YDnw6HbHRwOri9MMPgd8stjeh8KDca9TGEX/bkrpgbdZZwOFB/d6U3i4sZbCHGUoPIx3AoUQfxfws+1O/wZwbXGqwtUppWcpjHpPBdYXr3tWSmnLW3z8/sCkYt//pRCev9LC0h+jcP/rKTwoeH5K6ZXisX8AOlP4Lv8A/CeFuewt8g7u4+3a3c97V7X9kcIDl2dR+M5qgGHFw98B5lKYSvTH4nVP3tl1JCne/jM9kqS9XUSMAT5TnHIiSXoHHNGWJEmSMsgatCPijCi8HOG5nT2pHxEfiIgFxZcPLIyI0mL7wIh4JAovMngyIv5+x6tLkiRJ717Zpo4UX9CwksI8t1pgMTAqpbS8SZ+fAP+dUro9Ij4O/GNKaXTxRQYppVRTXIbqceC4lNJrWYqVJEmSWlnOEe2TgOdSSs8XH26ZRWGppabK+Mvb1x7YdjyltDKlVFPcfoHCSxO6ZaxVkiRJalU5g3YJzV82UEvzFxEA/AY4t7j9CeB9EXFo0w4RcRKFJ9tb+iYzSZIkqc219StjrwZuKT7d/isKyzA1bDsYEUcCdwAXFd8a1kxEXAJcAvDe9773xGOPPXZP1CxJkqR92OOPP74+pbTb2RY5g3Ydhbd8bVNK8zd+bZsWci5ARBwAnLdtHnZEHEhhTdmvppR2uv5pSuk24DaA8vLytGTJkta+B0mSJKmZiPh9S/rlnDqyGOgTEUdHRGdgJIVF/htFxGERsa2GrwA/LLZ3Bv4L+FFK6T8z1ihJkiRlkS1op5TqKbyh7R5gBTA7pfRMRFwXEVXFbh8Dno2IlcARFN48BlANfAQYExHLiv8G5qpVkiRJam17zZshnToiSZKkPSEiHk8ple+un2+GlCRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpA4O2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMjBoS5IkSRkYtCVJkqQMDNqSJElSBgZtSZIkKQODtiRJkpSBQVuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpA4O2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMjBoS5IkSRkYtCVJkqQMDNqSJElSBgZtSZIkKQODtiRJkpSBQVuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCmDrEE7Is6IiGcj4rmIGL+T4x+IiAUR8WRELIyI0ibHLoqImuK/i3LWKUmSJLW2bEE7IjoAtwJnAmXAqIgo267bt4AfpZT6A9cB3yie+37g68DJwEnA1yPikFy1SpIkSa0t54j2ScBzKaXnU0pbgFnA2dv1KQN+Wdx+oMnxCuC+lNKrKaU/APcBZ2SsVZIkSWpVOYN2CbC2yX5tsa2p3wDnFrc/AbwvIg5t4bmSJElqI/Pnz6dv37707t2bSZMm7XB8zZo1DBs2jEGDBtG/f3/mzZsHwJYtW/jHf/xHjj/+eAYMGMDChQsB2LRpEyNGjODYY4+lX79+jB/ffNbx7NmzKSsro1+/flx44YXZ7681dGzjz78auCUixgC/AuqAhpaeHBGXAJcA9OzZM0d9kiRJ2k5DQwPjxo3jvvvuo7S0lMGDB1NVVUVZ2V9mCV9//fVUV1dz6aWXsnz5ciorK1m9ejXTpk0D4KmnnuLll1/mzDPPZPHixQBcffXVDBs2jC1btjB8+HDuvvtuzjzzTGpqavjGN77Bww8/zCGHHMLLL7/cJvf9duUc0a4DejTZLy22NUopvZBSOjelNAj4arHttZacW+x7W0qpPKVU3q1bt9auX5IkSTuxaNEievfuTa9evejcuTMjR45kzpw5zfpEBBs3bgRgw4YNdO/eHYDly5fz8Y9/HIDDDz+cgw8+mCVLltC1a1eGDRsGQOfOnTnhhBOora0FYNq0aYwbN45DDjmk8bz2IGfQXgz0iYijI6IzMBKY27RDRBwWEdtq+Arww+L2PcDpEXFI8SHI04ttkiRJamN1dXX06PGXMdHS0lLq6pqPiU6cOJE777yT0tJSKisrmTp1KgADBgxg7ty51NfXs2rVKh5//HHWrl3b7NzXXnuNX/ziFwwfPhyAlStXsnLlSk455RSGDBnC/PnzM99h68gWtFNK9cBlFALyCmB2SumZiLguIqqK3T4GPBsRK4EjgBuK574K/CuFsL4YuK7YJkmSpHZg5syZjBkzhtraWubNm8fo0aPZunUrY8eOpbS0lPLycq644go+/OEP06FDh8bz6uvrGTVqFJdffjm9evVqbKupqWHhwoXMnDmTiy++mNdee62tbq3Fss7RTinNA+Zt1zahyfZ/Av/5Fuf+kL+McEuSJOldoqSkpNkodG1tLSUlzdetmD59euPI89ChQ9m8eTPr16/n8MMPZ8qUKY39PvzhD3PMMcc07l9yySX06dOHK664orGttLSUk08+mU6dOnH00UdzzDHHUFNTw+DBg3PdYqvwzZCSJEl6WwYPHkxNTQ2rVq1iy5YtzJo1i6qqqmZ9evbsyYIFCwBYsWIFmzdvplu3bmzatIk33ngDgPvuu4+OHTs2PkR57bXXsmHDBr797W83u9Y555zTuDrJ+vXrWblyZeNo97tZW686IkmSpHamY8eO3HLLLVRUVNDQ0MDYsWPp168fEyZMoLy8nKqqKiZPnszFF1/MlClTiAhmzJhBRPDyyy9TUVHBfvvtR0lJCXfccQdQGBW/4YYbOPbYYznhhBMAuOyyy/jMZz5DRUUF9957L2VlZXTo0IGbbrqJQw89tC2/ghaJlFJb19AqysvL05IlS9q6DEmSJO3lIuLxlFL57vo5dUSSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpg45tXYAkSZLan6PG39XWJbB60oi2LmGXHNGWJEmSMjBoS5IkSRkYtCVJkqQMDNqSJElSBgZtSZIkKQODtiRJkpSBQVuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpA4O2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMjBoS5IkSRkYtCVJkqQMDNqSJElSBgZtSZIkKQODtiRJkpSBQVuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZZA3aEXFGRDwbEc9FxPidHO8ZEQ9ExNKIeDIiKovtnSLi9oh4KiJWRMRXctYpSZIktbZsQTsiOgC3AmcCZcCoiCjbrtu1wOyU0iBgJPDdYvsFwP4ppeOBE4HPRsRRuWqVJEmSWlvOEe2TgOdSSs+nlLYAs4Czt+uTgAOL2wcBLzRpf29EdATeA2wBNmasVZIkSWpVOYN2CbC2yX5tsa2picCnIqIWmAd8odj+n8AbwIvAGuBbKaVXM9YqSZIktaq2fhhyFDAjpVQKVAJ3RMR+FEbDG4DuwNHAVRHRa/uTI+KSiFgSEUvWrVu3J+uWJEmSdiln0K4DejTZLy22NfVpYDZASukRoAtwGHAhMD+l9OeU0svAw0D59h+QUrotpVSeUirv1q1bhluQJEl7m/nz59O3b1969+7NpEmTdji+Zs0ahg0bxqBBg+jfvz/z5s0D4Mc//jEDBw5s/LfffvuxbNkyNm3axIgRIzj22GPp168f48fvsP4DP/3pT4kIlixZkv3+9O6RM2gvBvpExNER0ZnCw45zt+uzBhgOEBHHUQja64rtHy+2vxcYAvw2Y62SJGkf0NDQwLhx47j77rtZvnw5M2fOZPny5c36XH/99VRXV7N06VJmzZrF5z//eQA++clPsmzZMpYtW8Ydd9zB0UcfzcCBAwG4+uqr+e1vf8vSpUt5+OGHufvuuxuv98c//pHvfOc7nHzyyXvuRvWukC1op5TqgcuAe4AVFFYXeSYirouIqmK3q4CLI+I3wExgTEopUVit5ICIeIZCYP/3lNKTuWqVJEn7hkWLFtG7d2969epF586dGTlyJHPmzGnWJyLYuLGwBsOGDRvo3r37DteZOXMmI0eOBKBr164MGzYMgM6dO3PCCSdQW1vb2PdrX/saX/7yl+nSpUuu29K7VMecF08pzaPwkGPTtglNtpcDp+zkvNcpLPEnSZLUaurq6ujR4y8zW0tLS3nsscea9Zk4cSKnn346U6dO5Y033uD+++/f4Tr/8R//sUNAB3jttdf4xS9+wRe/+EUAnnjiCdauXcuIESO46aabWvlu9G7X1g9DSpIkvavMnDmTMWPGUFtby7x58xg9ejRbt25tPP7YY4/RtWtXPvShDzU7r76+nlGjRnH55ZfTq1cvtm7dypVXXsnkyZP39C3oXcKgLUmS9hklJSWsXfuX1Ydra2spKWm++vD06dOprq4GYOjQoWzevJn169c3Hp81axajRo3a4dqXXHIJffr04YorrgAKc7OffvppPvaxj3HUUUfx6KOPUlVV5QOR+xCDtiRJ2mcMHjyYmpoaVq1axZYtW5g1axZVVVXN+vTs2ZMFCxYAsGLFCjZv3sy21c22bt3K7NmzG+dnb3PttdeyYcMGvv3tbze2HXTQQaxfv57Vq1ezevVqhgwZwty5cykv32EhNe2lDNqSJGmf0bFjR2655RYqKio47rjjqK6upl+/fkyYMIG5cwuLo02ePJlp06YxYMAARo0axYwZM4gIAH71q1/Ro0cPevX6y+s9amtrueGGG1i+fDknnHACAwcO5Ac/+EGb3J/eXaKwyEf7V15envxTjCRJ0p5x1Pi72roEVk8a0SafGxGPp5R2+6cJR7QlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIktRPz58+nb9++9O7dm0mTJu1wfM2aNQwbNoxBgwbRv39/5s2b13jsySefZOjQofTr14/jjz+ezZs3A7BlyxYuueQSjjnmGI499lh++tOfNp4ze/ZsysrK6NevHxdeeGH+G5T2Mh3bugBJkrR7DQ0NjBs3jvvuu4/S0lIGDx5MVVUVZWVljX2uv/56qqurufTSS1m+fDmVlZWsXr2a+vp6PvWpT3HHHXcwYMAAXnnlFTp16gTADTfcwOGHH87KlSvZunUrr776KgA1NTV84xvf4OGHH+aQQw7h5ZdfbpP7ltozg7YkSe3AokWL6N27N7169QJg5MiRzJkzp1nQjgg2btwIwIYNG+jevTsA9957L/3792fAgAEAHHrooY3n/PCHP+S3v/0tAPvttx+HHXYYANOmTWPcuHEccsghABx++OGZ71Da+zh1RJKkdqCuro4ePXo07peWllJXV9esz8SJE7nzzjspLS2lsrKSqVOnArBy5UoigoqKCk444QRuvPFGAF577TUAvva1r3HCCSdwwQUX8NJLLzWes3LlSk455RSGDBnC/Pnz98RtSnsVR7QlSdpLzJw5kzFjxnDVVVfxyCOPMHr0aJ5++mnq6+v59a9/zeLFi+natSvDhw/nxBNPZMCAAdTW1vLhD3+Ym2++mZtvvpmrr76aO+64g/r6empqali4cCG1tbV85CMf4amnnuLggw9u69v8qxw1/q62LoHVk0a0dQnaQxzRliSpHSgpKWHt2rWN+7W1tZSUlDTrM336dKqrqwEYOnQomzdvZv369ZSWlvKRj3yEww47jK5du1JZWckTTzzBoYceSteuXTn33HMBuOCCC3jiiSeAwoh5VVUVnTp14uijj+aYY46hpqZmD92ttHcwaEuS1A4MHjyYmpoaVq1axZYtW5g1axZVVVXN+vTs2ZMFCxYAsGLFCjZv3ky3bt2oqKjgqaeeYtOmTdTX1/Pggw9SVlZGRHDWWWexcOFCABYsWNA45/ucc85pbF+/fj0rV65snB8uqWWcOiJJUjvQsWNHbrnlFioqKmhoaGDs2LH069ePCRMmUF5eTlVVFZMnT+biiy9mypQpRAQzZswgIjjkkEO48sorGTx4MBFBZWUlI0YUpi9885vfZPTo0VxxxRV069aNf//3fwegoqKCe++9l7KyMjp06MBNN93U7CFKSbsXKaW2rqFVlJeXpyVLlrR1GZIk6V3MOdqtZ1/+LiPi8ZRS+e76OXVEkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpA4O2JEmSlIFBW5IkScrAoC1Jym7+/Pn07duX3r17M2nSpB2Or1mzhmHDhjFo0CD69+/PvHnzGo89+eSTDB06lH79+nH88cezefNmAB5//HGOP/54evfuzeWXX862Nx0vW7aMIUOGMHDgQMrLy1m0aNGeuUlJ2o5BW5KUVUNDA+PGjePuu+9m+fLlzJw5k+XLlzfrc/3111NdXc3SpUuZNWsWn//85wGor6/nU5/6FN///vd55plnWLhwIZ06dQLg0ksvZdq0adTU1FBTU8P8+fMBuOaaa/j617/OsmXLuO6667jmmmv27A1LUpFBW5KU1aJFi+jduze9evWic+fOjBw5kjlz5jTrExFs3LgRgA0bNtC9e3cA7r33Xvr378+AAQMAOPTQQ+nQoQMvvvgiGzduZMiQIUQE//AP/8DPf/7zXV5Lkva0jm1dgCRp71ZXV0ePHj0a90tLS3nsscea9Zk4cSKnn346U6dO5Y033uD+++8HYOXKlUQEFRUVrFu3jpEjR3LNNddQV1dHaWlps2vW1dUB8O1vf5uKigquvvpqtm7dyv/8z//sgbuUpB0ZtCVJbW7mzJmMGTOGq666ikceeYTRo0fz9NNPU19fz69//WsWL15M165dGT58OCeeeCIHHXTQW17re9/7HlOmTOG8885j9uzZfPrTn24M7u3ZUePvausSWD1pRFuXILUrTh2RJGVVUlLC2rVrG/dra2spKSlp1mf69OlUV1cDMHToUDZv3sz69espLS3lIx/5CIcddhhdu3alsrKSJ554gpKSEmpra3d6zdtvv51zzz0XgAsuuMCHISW1GYO2JCmrwYMHU1NTw6pVq9iyZQuzZs2iqqqqWZ+ePXuyYMECAFasWMHmzZvp1q0bFRUVPPXUU2zatIn6+noefPBBysrKOPLIIznwwAN59NFHSSnxox/9iLPPPhuA7t278+CDDwLwy1/+kj59+uzZG5akIqeOSJKy6tixI7fccgsVFRU0NDQwduxY+vXrx4QJEygvL6eqqorJkydz8cUXM2XKFCKCGTNmEBEccsghXHnllQwePJiIoLKykhEjCtMXvvvd7zJmzBj+9Kc/ceaZZ3LmmWcCMG3aNL74xS9SX19Ply5duO2229ry9iXtw2LbuqPtXXl5eVqyZElblyFJUhbO0W4dfo+tZ1/+LiPi8ZRS+e76OXVEkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpA4O2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMjBoS5IkSRkYtCVJkqQMOua8eEScAXwH6AD8IKU0abvjPYHbgYOLfcanlOYVj/UH/g04ENgKDE4pbc5ZryQ1NX/+fL74xS/S0NDAZz7zGcaPH9/s+Jo1a7jooot47bXXaGhoYNKkSVRWVrJ69WqOO+44+vbtC8CQIUP4/ve/D8AZZ5zBiy++SH19PX/7t3/LrbfeSocOHQCYOnVq4/6IESO48cYb9+wNZ3LU+LvaugRWTxrR1iVI2gdlC9oR0QG4FTgNqAUWR8TclNLyJt2uBWanlL4XEWXAPOCoiOgI3AmMTin9JiIOBf6cq1ZJ2l5DQwPjxo3jvvvuo7S0lMGDB1NVVUVZWVljn+uvv57q6mouvfRSli9f3hiyAT74wQ+ybNmyHa47e/ZsDjzwQFJKnH/++fzkJz9h5MiRPPDAA8yZM4ff/OY37L///rz88st76lYlSZnknDpyEvBcSun5lNIWYBZw9nZ9EoURa4CDgBeK26cDT6aUfgOQUnolpdSQsVZJambRokX07t2bXr160blzZ0aOHMmcOXOa9YkINm7cCMCGDRvo3r37bq974IGF/8urr69ny5YtRAQA3/ve9xg/fjz7778/AIcffnhr3o4kqQ3kDNolwNom+7XFtqYmAp+KiFoKo9lfKLYfA6SIuCcinoiIa3b2ARFxSUQsiYgl69ata93qJe3T6urq6NGjR+N+aWkpdXV1zfpMnDiRO++8k9LSUiorK5k6dWrjsVWrVjFo0CA++tGP8tBDDzU7r6KigsMPP5z3ve99nH/++QCsXLmShx56iJNPPpmPfvSjLF68OOPdSZL2hLYsVRwmAAAgAElEQVR+GHIUMCOlVApUAndExH4UprScCnyy+N9PRMTw7U9OKd2WUipPKZV369ZtT9YtScycOZMxY8ZQW1vLvHnzGD16NFu3buXII49kzZo1LF26lJtvvpkLL7ywceQb4J577uHFF1/kzTff5Je//CVQGOF+9dVXefTRR7npppuorq4mpdRWtyZJagU5g3Yd0KPJfmmxralPA7MBUkqPAF2AwyiMfv8qpbQ+pbSJwmj3CRlrlaRmSkpKWLv2L3+Uq62tpaSk+R/lpk+fTnV1NQBDhw5l8+bNrF+/nv33359DDz0UgBNPPJEPfvCDrFy5stm5Xbp04eyzz26cjlJaWsq5555LRHDSSSex3377sX79+py3KEnKLGfQXgz0iYijI6IzMBKYu12fNcBwgIg4jkLQXgfcAxwfEV2LD0Z+FFiOpBaZP38+ffv2pXfv3kyaNGmH42vWrGHYsGEMGjSI/v37M2/ePABWr17Ne97zHgYOHMjAgQP53Oc+13jOV7/6VXr06MEBBxzQomu1d4MHD6ampoZVq1axZcsWZs2aRVVVVbM+PXv2ZMGCBQCsWLGCzZs3061bN9atW0dDQ+Gxkueff56amhp69erF66+/zosvvggURrDvuusujj32WADOOeccHnjgAaAwjWTLli0cdthhe+p2JUkZZFt1JKVUHxGXUQjNHYAfppSeiYjrgCUppbnAVcC0iPgnCg9GjkmFv5X+ISJuphDWEzAvpdT260NJ7UCu1TLOOussLrvsMvr06dOsfVfXas86duzILbfcQkVFBQ0NDYwdO5Z+/foxYcIEysvLqaqqYvLkyVx88cVMmTKFiGDGjBlEBL/61a+YMGECnTp1Yr/99uP73/8+73//+3nppZeoqqrizTffZOvWrQwbNqzxl5mxY8cyduxYPvShD9G5c2duv/32xgclJUntU9Z1tItrYs/brm1Ck+3lwClvce6dFJb4k/Q2NF0tA2hcLaNp0H4nq2UMGTJkp+3v5FrtRWVlJZWVlc3arrvuusbtsrIyHn744R3OO++88zjvvPN2aD/iiCPe8iHHzp07c+ed/l+eJO1N2vphSEmtLOdqGTuzq2tJkrQvM2hL+6B3ulrG27mWJEn7OoO2tJfJvVrG9t7qWpIk7esM2tJeJsdqGbvyVteSJGlfZ9CW9jJNV8s47rjjqK6ublwtY+7cwgqbkydPZtq0aQwYMIBRo0Y1Wy2jf//+DBw4kPPPP79xtQyAa665htLSUjZt2kRpaSkTJ07c5bUkSdrXxd7y5rHy8vK0ZMmSti5DkrSdo8a3/eqsqyeNaOsS/mp+j63D77H17MvfZUQ8nlIq310/R7QlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJyqBFQTsiLoiI9xW3r42In0XECXlLkyRJktqvlo5ofy2l9MeIOBX4O2A68L18ZUmSJEntW0uDdkPxvyOA21JKdwGd85QkSZIktX8dW9ivLiL+DTgN+GZE7I/zuyXt5fbllzFIkv56LQ3L1cA9QEVK6TXg/cCXslUlSZIktXMtCtoppU3Ay8CpxaZ6oCZXUZIkSVJ719JVR74OfBn4SrGpE3BnrqIkSZKk9q6lU0c+AVQBbwCklF4A3perKEmSJKm9a+nDkFtSSikiEkBEvDdjTZL+Cj7AJ0nSu0NLR7RnF1cdOTgiLgbuB6blK0uSJElq31o0op1S+lZEnAZsBPoCE1JK92WtTJIkSWrHdhu0I6IDcH9KaRhguJYkSZJaYLdTR1JKDcDWiDhoD9QjSZIk7RVa+jDk68BTEXEfxZVHAFJKl2epSpIkSWrnWhq0f1b8J0mSJKkFWvow5O0R0Rk4ptj0bErpz/nKkiRJktq3FgXtiPgYcDuwGgigR0RclFL6Vb7SJEmSpParpVNHJgOnp5SeBYiIY4CZwIm5CpMkSZLas5a+sKbTtpANkFJaCXTKU5IkSZLU/rV0RHtJRPwAuLO4/0lgSZ6SJEmSpPavpUH7UmAcsG05v4eA72apSJIkSdoLtDRodwS+k1K6GRrfFrl/tqokSZKkdq6lc7QXAO9psv8e4P7WL0eSJEnaO7Q0aHdJKb2+bae43TVPSZIkSVL719Kg/UZEnLBtJyLKgT/lKUmSJElq/1o6R/sK4CcR8UJx/0jg7/OUJEmSJLV/uxzRjojBEfE3KaXFwLHAfwB/BuYDq/ZAfZIkSVK7tLupI/8GbCluDwX+GbgV+ANwW8a6JEmSpHZtd1NHOqSUXi1u/z1wW0rpp8BPI2JZ3tIkSZKk9mt3I9odImJbGB8O/LLJsZbO75YkSZL2ObsLyzOBByNiPYVVRh4CiIjewIbMtUmSJEnt1i6DdkrphohYQGGVkXtTSql4aD/gC7mLkyRJktqr3U7/SCk9upO2lXnKkSRJkvYOLX1hjSRJkqS3waAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJyiBr0I6IMyLi2Yh4LiLG7+R4z4h4ICKWRsSTEVG5k+OvR8TVOeuUJEmSWlu2oB0RHYBbgTOBMmBURJRt1+1aYHZKaRAwEvjudsdvBu7OVaMkSZKUS84R7ZOA51JKz6eUtgCzgLO365OAA4vbBwEvbDsQEecAq4BnMtYoSZIkZZEzaJcAa5vs1xbbmpoIfCoiaoF5wBcAIuIA4MvAv+zqAyLikohYEhFL1q1b11p1S5IkSX+1tn4YchQwI6VUClQCd0TEfhQC+JSU0uu7OjmldFtKqTylVN6tW7f81UqSJEkt1DHjteuAHk32S4ttTX0aOAMgpfRIRHQBDgNOBs6PiBuBg4GtEbE5pXRLxnolSZKkVpMzaC8G+kTE0RQC9kjgwu36rAGGAzMi4jigC7AupfS32zpExETgdUO2JEmS2pNsU0dSSvXAZcA9wAoKq4s8ExHXRURVsdtVwMUR8RtgJjAmpZRy1SRJkiTtKTlHtEkpzaPwkGPTtglNtpcDp+zmGhOzFCdJkiRl1NYPQ0qSJEl7JYO2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMjBoS5IkSRkYtCVJkqQMDNqSJElSBgZtSZIkKQODtiRJkpSBQVuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpA4O2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMjBoS5IkSRkYtCVJkqQMDNqSJElSBgZtSZIkKQODtiRJkpSBQVuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpA4O2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMsgatCPijIh4NiKei4jxOzneMyIeiIilEfFkRFQW20+LiMcj4qnifz+es069e8yfP5++ffvSu3dvJk2atMPxNWvWMGzYMAYNGkT//v2ZN28eAK+88grDhg3jgAMO4LLLLtvptauqqvjQhz7UuP/qq69y2mmn0adPH0477TT+8Ic/5LkpSZK0T8oWtCOiA3ArcCZQBoyKiLLtul0LzE4pDQJGAt8ttq8HzkopHQ9cBNyRq069ezQ0NDBu3Djuvvtuli9fzsyZM1m+fHmzPtdffz3V1dUsXbqUWbNm8fnPfx6ALl268K//+q9861vf2um1f/azn3HAAQc0a5s0aRLDhw+npqaG4cOH7zTYS5IkvVM5R7RPAp5LKT2fUtoCzALO3q5PAg4sbh8EvACQUlqaUnqh2P4M8J6I2D9jrXoXWLRoEb1796ZXr1507tyZkSNHMmfOnGZ9IoKNGzcCsGHDBrp37w7Ae9/7Xk499VS6dOmyw3Vff/11br75Zq699tpm7XPmzOGiiy4C4KKLLuLnP/95jtuSJEn7qI4Zr10CrG2yXwucvF2ficC9EfEF4L3A3+3kOucBT6SU3tz+QERcAlwC0LNnz1YoWW2prq6OHj16NO6Xlpby2GOPNeszceJETj/9dKZOncobb7zB/fffv9vrfu1rX+Oqq66ia9euzdpfeukljjzySAD+5m/+hpdeeqkV7kKSJKmgrR+GHAXMSCmVApXAHRHRWFNE9AO+CXx2ZyenlG5LKZWnlMq7deu2RwpW25o5cyZjxoyhtraWefPmMXr0aLZu3fqW/ZctW8bvfvc7PvGJT+zyuhFBRLR2uZIkaR+WM2jXAT2a7JcW25r6NDAbIKX0CNAFOAwgIkqB/wL+IaX0u4x16l2ipKSEtWv/8keQ2tpaSkpKmvWZPn061dXVAAwdOpTNmzezfv36t7zmI488wpIlSzjqqKM49dRTWblyJR/72McAOOKII3jxxRcBePHFFzn88MNb+Y4kSdK+LGfQXgz0iYijI6IzhYcd527XZw0wHCAijqMQtNdFxMHAXcD4lNLDGWvUu8jgwYOpqalh1apVbNmyhVmzZlFVVdWsT8+ePVmwYAEAK1asYPPmzezqrxmXXnopL7zwAqtXr+bXv/41xxxzDAsXLgQKq5DcfvvtANx+++2cffb2jxBIkiS9c9nmaKeU6iPiMuAeoAPww5TSMxFxHbAkpTQXuAqYFhH/ROHByDEppVQ8rzcwISImFC95ekrp5Vz1qu117NiRW265hYqKChoaGhg7diz9+vVjwoQJlJeXU1VVxeTJk7n44ouZMmUKEcGMGTMap3wcddRRbNy4kS1btvDzn/+ce++9l7Ky7Re6+Yvx48dTXV3N9OnT+cAHPsDs2bP31K1KkqR9QM6HIUkpzQPmbdc2ocn2cuCUnZx3PXB9ztr07lRZWUllZWWztuuuu65xu6ysjIcf3vkfOVavXr3Lax911FE8/fTTjfuHHnpo4+i4JElSa2vrhyH3GjletPLVr36VHj167LD+880330xZWRn9+/dn+PDh/P73v893Y5IkSXpHDNqtINeLVs466ywWLVq0Q/ugQYNYsmQJTz75JOeffz7XXHNNnhuTJEnSO2bQbgW5XrQyZMiQxnWemxo2bFjjmtBDhgyhtra2tW9JkiRJf6Wsc7T3FbletNIS06dP58wzz2yVa0mSJKn1OKK9h7zdF620xJ133smSJUv40pe+1EpVSpIkqbU4ot0KWvqilfnz5wPNX7TyTl+Scv/993PDDTfw4IMPsv/++7/z4iVJkpSFI9qtIMeLVnZl6dKlfPazn2Xu3Lm+zVCSJOldyqDdCpq+aOW4446jurq68UUrc+cWXoY5efJkpk2bxoABAxg1atQOL1q58sormTFjBqWlpY0rllxzzTWUlpayadMmSktLmThxIgBf+tKXeP3117ngggsYOHDgDqFekiRJbc+pI60kx4tWbrzxRm688cYd2lvrQUpJkiTl44i2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMvBhSL1rHDX+rrYugdWTRrR1CZIkaS/hiLYkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGrqPdClz/WZIkSdtzRFuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpA4O2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMjBoS5IkSRkYtCVJkqQMDNqSJElSBgZtSZIkKQODtiRJkpSBQVuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCmDrEE7Is6IiGcj4rmIGL+T4z0j4oGIWBoRT0ZEZZNjXyme92xEVOSsU5IkSWptHXNdOCI6ALcCpwG1wOKImJtSWt6k27XA7JTS9yKiDJgHHFXcHgn0A7oD90fEMSmlhlz1SpIkSa0p54j2ScBzKaXnU0pbgFnA2dv1ScCBxe2DgBeK22cDs1JKb6aUVgHPFa8nSZIktQs5g3YJsLbJfm2xramJwKciopbCaPYX3sa5RMQlEbEkIpasW7euteqWJEmS/mpt/TDkKGBGSqkUqATuiIgW15RSui2lVJ5SKu/WrVu2IiVJkqS3K9scbaAO6NFkv7TY1tSngTMAUkqPREQX4LAWnitJkiS9a+Uc0V4M9ImIoyOiM4WHG+du12cNMBwgIo4DugDriv1GRsT+EXE00AdYlLFWSZIkqVVlG9FOKdVHxGXAPUAH4IcppWci4jpgSUppLnAVMC0i/onCg5FjUkoJeCYiZgPLgXpgnCuOSJIkqT3JOXWElNI8Cg85Nm2b0GR7OXDKW5x7A3BDzvokSZKkXNr6YUhJkiRpr2TQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpA4O2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMjBoS5IkSRkYtCVJkqQMDNqSJElSBgZtSZIkKQODtiRJkpSBQVuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgYGbUmSJCkDg7YkSZKUgUFbkiRJysCgLUmSJGVg0JYkSZIyMGhLkiRJGRi0JUmSpAwM2pIkSVIGBm1JkiQpA4O2JEmSlIFBW5IkScrAoC1JkiRlYNCWJEmSMjBoS5IkSRkYtCVJkqQMDNqSJElSBgZtSZIkKQODtiRJkpSBQVuSJEnKwKAtSZIkZWDQliRJkjIwaEuSJEkZGLQlSZKkDAzakiRJUgZZg3ZEnBERz0bEcxExfifHp0TEsuK/lRHxWpNjN0bEMxGxIiL+X0REzlolSZKk1tQx14UjogNwK3AaUAssjoi5KaXl2/qklP6pSf8vAIOK2x8GTgH6Fw//GvgosDBXvZIkSVJryjmifRLwXErp+ZTSFmAWcPYu+o8CZha3E9AF6AzsD3QCXspYqyRJktSqIqWU58IR5wNnpJQ+U9wfDZycUrpsJ30/ADwKlKaU/v/2zjzervH6/++PGGtMEIovIVpTS/s11exHv+YaSluzRs1DzUOLKtUoRWusmkoiahahCVpDDTUUpfWlfKmWqqGGUqKUfH5/rOfk7qQhuTfn3H3vOev9enndnH32OXd57t77Wc961lqfD8uxU4DdAAFn2T5qCp/bA9ijvFwKeLIV/y/9hPmAV+s2og3IcWwOOY7NI8eyOeQ4Noccx+aQ49g86hrLxWzPP7WTWpY60k22Ba6uONlLAssAi5T3fylpLdt3VT9k+zzgvF61tI8i6UHbK9VtR38nx7E55Dg2jxzL5pDj2BxyHJtDjmPz6Otj2crUkReA/6q8XqQcmxLb0pU2ArAVcJ/tt22/DYwDVmuJlUmSJEmSJEnSAlrpaP8W+JSkxSXNTDjTYyY/SdLSwEDg3srh54B1JM0oaSaiEPKJFtqaJEmSJEmSJE2lZY627Q+A/YCbCSf5Stv/K+l4SZtXTt0WuNyTJotfDTwD/AF4FHjU9g2tsrVNyBSa5pDj2BxyHJtHjmVzyHFsDjmOzSHHsXn06bFsWTFkkiRJkiRJknQyqQyZJEmSJEmSJC0gHe0kSZIkSZIkaQHpaCdJkiRJkiRJC0hHO0m6Semik7QQSQPqtiEJJA2u24YkSaZMPiv7PulodziShkhaoG47+guSPg0cLmmeum1pV0rLz70kzVq3LZ1O+VvcK2nVum3pL0gaKmmfdIACSQtIWrxuO9oRScsAF0qaTZLqtqedkLSwpKWa8V3paHcw5SIaA3yhblv6A8XpuBx43vY/6ranHSnX5BXAe7b/Vbc9nYykZYFzgRNt31+3Pf2ByvX7RkPpuJMpz8wrgE/WbUu7Ua61Cwhxv3edLeSaRmWuX2Rq504L6Wh3KGUSHQkMt329gtnrtquvIum/gGuAi2xfImmApNUyitA8JC1J9NAfbvuCMsar1G1XJyLpE8Bo4MXK3+Knklas27a+iqQhwFjgLNs/L4Jry9ZrVX0UR3AUcLHt35Rj6XM0AUmLAHcB59s+V9LMkraTNEvdtvV3ipN9GfBT27eWY9PlG+VF34GUHOOjgBlsX14OXwysWZtRfZiSJvIl4AXgbklzE0JMG2UUoTmUB9nywOLA7eXwTcBXazOqg7E9HjgCWELSDsDPgbdtP1SvZX0TSQOBLYE/0qVyPI4OfaaW8fgh8KDti8uxEUBTtuITBgK/AxopjNcCS9p+rz6T+j9lHjoWeNb2peXYCGC56fre9BM6j5I7uAKwB/AvYCjwhO3DazWsD1IiUicDBwAbAKsAnwHutf3NOm1rF8oYXwysC3wdOBB4FbjV9jG1GdaBSJrZ9vuSPmF7vKTNgPOJ58N65ZwZi/JvwsQ82QOAS4E1gMHA/yOu38PqtK0OJC0BrAh8ikgZ+RWwJ/B0PjObh6TlgcOBLwIjO/FaawUlsLA88BywOfC47YOm5zszot1BSJpd0hzAQNsPA6cDSwBDG052dtToomx9jgQusf2M7Z8AdxBO4PWSZqzTvnagjPGlwOm2x9s+BzgOWIzIkSPHuXcoW6Y/kXQpcKOk9W3fCOwMDJS0NYDtDzJlKijF0ZcAd9m+u/z7ZeDvwPWV8zpivCp1P/+2PRz4P2Lh/H7Dyc4i0Z4haTFJ20vaTdJWtn8PnATcB4yvnJd+XTeRNEvZqcb2KOBOYCNgpoaTPT3zUEa0O4QSdTkRmAWYDXjU9gGSlgMOBl4DjrQ9QdIMtifUaG7tFKfjOiIi82nbr1Te2w/4PHAD8Cvbb9djZf+mMim/Bexk+4+SZNuS9iGihMMa+Z1J6yh/iyuBc4BngWWBfYAf2L5I0qbA94Af2R5Zn6V9hzJmvwLG216qcnw+YHdie/922+NqMrFXqYzHObZPrBzfHViZqHG51/ZbNZnYbynz0dXArcDcwNrAbcROwWeAQ4C/2T6yNiP7KcU3Oh4YRMxFjwNHA+sAWwOPAtfZfq0xP3X7d6Sj3f6UB+BlwNnAPcROxjXAQ7Z3KhfaoUSnh33qs7RvUFpRXU6M14zAfsAutv9QOWcv4ka8AhjT6QuT7lJx7EYROwSrAqNs31k5Z08iX25r2/dO8YuS6UbSp4gcz+NtX1U5vjXhXB9k+2ZJWwLfJ7aqX+rk+gRJQ4nr98fAN4DXbG9deX9BYidgUeAX7e5sVwofXyTyhnez/WTl/QMIh3AscIvtd2oxtB9SgmHnEoWPI8qxOYm5/AHbu0n6LHAM0RHrkPqs7V9UfKPziCj2EsAuwMzAVsDGwIbAn4id7R51G0tHu80plfC/A3awPbaRX6noUfwgseW5t6TPExHEk2w/UZ/F9SNpE2Bm26PL60OB7Ymo6/9Wztsf+HXZwku6gaS1gCG2RxanZSsip3Nk2YJvnLcv8JjtX9dkattTHOpLgOVs/6WRp13e25OIzq5v+01J89v+e5321k1JA9kdeNP2FeXYA4STU3W2PwnsClxj+4+1GNsLSJoLuIgIOIyQdATwFWLOqTrbhxFFZYdXdwiTKVOuMwEPEKk4q5Xjs9l+tzjbvwe+WzphrVjOy/loGlB0EnsQOLikizSODyaCbM/bPljSjkQL5JNtP9ej35WOdnsjaRCxGjvF9gnl2Ky2/yVpYSKS9VWio8Zstv9Zn7V9j0oqw8HAjsCOth+v2652ozjbWxBdCUbYvmey93u0ZZd8NOX+H2T7D5VUnZ1t31/yEScQUcjjgW2yALILSTPZ/ne1MFTS/cBfJ3O2Jy5a2hlJS9p+uvxbRJHelJztIbb/XI+V/QtJc9l+qziEdxM7ft8u7zXm8JOIBd/wfEZ2j5KOcyaR3jW8HJuhpM+uSezyf6Xc55+0/WJPf1cmzbcpkhaXtKPt1wnnZXdJpwKUG3QWYiJ9h0gZ+SCd7P+k8eCyfRowArhO0mfqtap9aBSJ2X6GKB57AthD0trV83ICaQlbA2dLWqEUoZ4OjJC0SnkeTADmBz4APtEpBX3Tgu1/l58fNIqkbK8KLChpXOW8tnayK/fvRCfbwUnAVcDFJTWRct6fazG0n1F2nO+QtLvt54luNjtLGg4xh5dT3wKyRqgbSFpU0sFll+lgYBlJPwSopID+BViY6CDE9DjZkI52W1KqjgcCZ0n6uu2XibZ021Sc7feIfKQPgY52YhTiHFPF9o+JVmcpv94kqg50cbZ/AfwBeL02o9ocSUtK2tX2GUTP8u9J+lzF2b5U0hBJKxARn4ttv5WLnSkzmbO9BjBY0n/XbFavMPk1Mdn9fBLRS/wySblQm0ZKfr+BbwGHSdrJ9l+J9IWdJZ1YzluNqAN4CDIYMS0U3+gTRODx8FJ39UNigXxy5dQlic5B46fwNd3/vfm3aS8U7aa2s32cpHWJ3Lnv275Q0gJEvtd5RLHftcB3bF//kV/Y5pRiiOOBgwgVvLwhaqaRg1i3He1IpWjtdJfuIZKOBVYCjrH9SEkjOQ74B3BAqe3o2G1pSQNc5NSr/57CedlffApIWsL2n+q2oz9Q0hnOA46yfZekDYhCyGNLPcsiRBHkw0S09URHC85kKpSx/SKRf70cETQbY/tERU/yw4iOI7cBPyV8ozFN+d0d+uxsS9QlHXqGu9S41gF+BpzgaNO1APAYMC+wqe1xnTqJlvG6CLjQ9oXd/GxHjtn0UIrDxtt+s25bOhFFYfStwPdsX1zSxxa2/SdJ3yNErI6x/WgpAHrF9i31WVw/ZYxWBJ4mVEvnJoRoPsrZbqRSWM+Vad0AAByZSURBVG3eJlVF1KhuO9oFdXXAOLPcnwNsfyjpfwjHb3Jn+0Db1+VcNHXU1eXqLNvnl2OfpcvZHl5SQocTwnRfbmaAIVNH2oRSMHENcFG5SWeStKajW8OuwNGVNJJlgHVdWk514k1aFhzjgBtLtH9GScMUxaNTOn9A+dnYIu64MesJDcdD0qpE3/ED9RGiSJUxzudSk9GUJcJvJEQZcChwPgScKunzti+1fUtu9zMHEf06l8g5fuljnOwB5bnQcLbbzsmu3M9LAYeUxdtHnZv38TSiKUjWE/ntn7H9S2Av4JiS8vVX4FPpZE8bkpYkepAPt32+pAGSVi1pI7sBm0k6wvZjwLeBdWyPhebN83kjtAGS5gG+RHQOuVuhcDSOWJlh+w5gGHCypN1sv2r7ThXqsrtmBhHiCh8q+maPBpZ0FI9OQiWyMA8xhvP2sq39lhLZ25hIz7kb2BvYW9Ls1fMqYzwQ+NHk7yc9R11iVQ8SvWL3kPQw8EjJywbA9nHA/cCAyrGOnsRtvwY8BaxPqMJOsY9u5fqdmwhqzNF7VvYe5X7eEDiN6B++s6IP+ySU8ZigUCP+Qq8b2o9QSNZ/kVB4fF/SFpLGAm8U54+ys3QgcJyiW9AH5XhH359To8wjyxO7UbeXwzcTHXEo47s7sIOkY2w/Zvv+8tmm+UbpaPdzJC1LSFjfTCgZHkA4kI/b/k45Z0CJbH8NeKbxWRd63+r6kDSfpEUcvcLPBuYibsCXbR9VzlHl/OoEOhq4vky+yVQo67i5CdWys2wfCGxOXIcHNSLbJbe1sZC5ihjjFLRoApp2ifCZAGwfZfvBOmztS1Qit6sTzvWGRP7mnmV3BkkDJc1aUkQaz4gbidSStuwEoSiQPYMo1DuWSEH8qqTFKudU7+ebiM5WyRTQtEnWz1SusbHA522/0I67Jc2m+Ea3E9fg4UQQ8jeEOumhjfMc2hjbE2l1VI43zTfqsXZ7Uj/lJh1JyCQ/A/xE0nhgQeB6dRXnNGTVby+f68jtpjJe5wE3S7rD9m/K9uZcwD8lLVBSayZSmTCuA452RUwl+XjKNfampMeBOSXNYvsBST8gFofPEr1hPyiR7CuA42zfVaPZbYMmlQgfBWD7JUkXEVHrzSXNaXucS7u6JCiR2y0IZ/Iw27dK+iewE7Cpos/uusBetl8oz4hrgG95sh7wbcYQIojze+D3kv5KqIXOJOki28+V+3keIif2KFcUdZMuNKlk/WgA22dIehdYuewc3Ovopd0I/mSQZxooY3spcGqpIzhH0pvAyUQjiIm98GFiZLtlZES7n6Io5BsNDAUmqubZvoRokbY9kXs0RwlcT6ic04lO9jLEg/8iQv3yNwC2HwbOIe6FwyQtVh2fEuk7i6hATid7KlQigUPV1W/8KWA1YpKG2FW5DzhJ0hfKZ0YT3XHSyW4CCgGgy4icwxclXdN4z/arRIH0q4TTuHE9VvZdFDUvRwNbFSd7KJF7/SNiN+CLRD3MC4qex+cTEvZt9Yyo3M+NoNxvCad6cwDbt5ZjQ4nONY1UxmuJAvw7e93ofoC6uv88AmxSXgPgKNb7X2AbYH1JszfmpE6cu7tLZZfgQ0rrwxJcHEUsCq+VtLpDiKZXUmez60g/pOQUX06kPswI7AfsUo0cSNoLWIeIEo7p5K2mMklcCtxt+6zK8a8T+VtHAIsC+5e3jnSXIACaTlWoTkPSpkRO8G8JWfXdiJSmOYCZgP8GNiOig7eWeoHBTlnmplAmj5QInw5Kys3PgVMJDYIhwCbA5rZvUqUFZdmNmcf2s3XZ20oULebWBV6wfbakAwgxj9eIuovTgF8CnybSwr5EtEr9bT0W922UkvUtQ13dRUYRgYRViV3TOyvn7EnsVG1t+94pflGz7UpHu/8haRNg5sZ2k6RDiQj2TiXfqHHe/sCvyzZfRyPpSuC7LvLpknYhcuEeA2YjJojPAf+0/VQ5pyNTbHpCY6yKg/ITYAfCQTkVWLqk4Pw3UZTyKLGwOYdoMfnMR31v0jOUEuHdonL9Lk2IJb0ObAd8mRDsuV7SMOK6PcFdfbXbsoWfumpTVgMuIISLjiS6r4wEPk/c47MD3y0/v0loOGQv8amglKxvCZLWAoY42iAOBbYigj0jq7tNkvYFHnPUrrXervQj+jeVCeJgYEdgx4YzmXQh6TLixhpeXq8DPGv7OUk/A36Y49Z9NFkvXUnzEYs+lZ/bOfo0r2f7tnLO0kQ+67aZv9l6JnO27wHesp3pIpNRAhgnEGlM/0NM0q+V5+s6xMLwmyVdoi2RtBBxfbxdooNHAvfYvkDRv/lKYKztE8r5sxIdWU4Avm770bps7w9MHrypvi6R7S2BXR3F+sl0UpztLYClgBGT10/0VjAtc7T7OZXcrdOAEcB1ldzYjkXSQpKWKWk2ECk0/1UKTLD96+Jkr0nkF3Z8VK+7KKq6L5N0vaQvK3qQz050aBgGbFmc7NWBH5c8eUqawtrpZPcO7mCJ8GmlPCeOJxydfxApThOAGcp7ZxEpZe3sZM8EfJWI2kO0QJ0fWK9EVv8KbA1sI+kUgJJiNwTYPp3sqTO5U1d97ZSsbxqNsSu7pdcDTxBtTdeuntdbO9YZ0e4HTB41nMq5hwL3tVtRTncoEdNRwBtEK7OfE617jgLmIVooXUcI95wCHGr75nqs7Z8Up/liojhsCLGV/APbv5O0GVF0MgKYldiC/5btGzIdp/koJcJ7RGU3cBCh+LgDUZx2NLEz+LSk9YkWYQva/ls7X7/FOZmF6MJ0MrAPsDSRx/80cG0JTiwELNZb+a2dhlKyvukoer1vAdzkFncYmeLvb9NnRttQtu+OBw4iCkzyD/YxlCjrKOBg4Eli+3dB28cohGY2IqJW8wLvAj+1PaYue/sjkmYDxgLv2d6oHDuGiGYfXSKoqxCFKAOBO23f0c5OSl0oJcJ7RMXJXpdQ3TuA6Le7CLB4SZ1Yi8g/3tX2X2oztsVI+gQwv+2/KFT0BgFfJ1pAHkgsor8GvAhcZvu5umztz3QnYJY0H1UKmHv9d+e813cpkdmLgAttX9jNz3akU1NSQe60PUN5vSQhsHA0UQj2Sjk+IzCL7Xc6dax6gqS5HH1dNwB2AR61fbKkI4lt5QlEms6j7bzN3lcoi8cvAxsT7dU280cUP1cK3DrWwa4iaT1CQGmc7ZslrQEcBzxAFEkfRhRQX/8xX9PvKamGexBtC1cknOsBRDerOcvPlYmdqR+6TburtILKgm4pIi1n5EcVN+Z92TMU3ZPG236zbls+iszR7qNIWoDI17rR9oWSZpQ0rGxzTun8AeVnIxezIx3HkjKziaTG1tvKhANyPpG/fkVxEg2ML5/pyLHqLiXydV4pePwVMaYrS7qB2CXYmehIMCswUtKSmWfYWpwS4d2mck1uRXTKeL28fhDYExgMLEHkZF/fAdfwHwlf4BBCQfTPwJ+J7kFvAhcSY/PtdLK7R3GyU7K+yTTuSYVK6w3AgSpKw1M4t+Eb1ebvpqPddxlEODMflmKc0cCStl+f/MTKJDoPcHKJcnUstm8C9pP0NnCw7cFEF4Ftid6ar9j+MB3s7lG2Pfcmrs0dbN9B9HL/JHC77Sdsn+Xo7LK07adzjFtDZaJJifBppOIwzwlge39it+tKSbPafs/2M7Z3s31Co26jHa9hFSCKZYH7iYXz6pI2LM/Hpwhho+eBT9me4iIu+WiUkvUtoSxgNibSau8m5qW9Jc1ePa/iGw0EfjT5+71FOtp9DEnzSVrE0d7nbKIw5XbgZdtHlXNUOb8aqRoNXF+iXB2N7bHEtvAny+vXbT9ve1/bj9RrXf+jMim/QUT7vidp++JsHwoMkfSdykc6zpHrTcpEswXRDWM+2/cRkZ0BhNrjIUQx6rwlUtZQ62t3ifApUhYblrQRcIGki8v1eyAxLg+WHZtJnq/tiguSViw7fA8QjuAYIkixokIdc02iyLnjtRh6yBCKZL1DtXkMIegzTNKiMLErUErWTyNljTg3sQNzVrmHNyfqCA5qRLYnW8BcRfhGtSxi0tHuQ5Q8rmuI7aXVi0N4DTGBvlzSSSahciGNJgrReqUBe3/A0bd5N0mvlBVt0kPKpLyepFXLjsEewJGSdizO9gXAimX3hcw1bC1KifBpovy/UxYbnyPSIX4C/A5YQdKxtg8BHgYerXN7uTdQtD1dq/x7I+BqIu3rarq6rlxH9Ay/G3gqI9nTTmWnKSXrW0RZI75J7ODNKWkW2w8AP6BL+KexgBlILGCOK/5ALWQxZB9B0S7tciKX61JXugaU93Yh5NbPdKUCXtH79GdE94y7etfq/oFCiGJ8cQiTbiBNLOb5DPEg2wBY0/YDJfdwOBFV+JmkQVNKbUqaj1IifKqUwMUw4DTbr0jalFAi3ac41F8guo0c7ui4sXw7R25LruqBROrXcUTR8pm2b1MUhm4P/ML2dYruTbPZfqg+i/snSsn6plOZh4YS1+VjkvYjhGjOsv2kpOWIQMOywDZEOtQdwHfqDkC29eq9v1BWv8cC59u+xF09cb8u6TSiddf5hKN9UCNKA2D738Bh6WR/NLbHurSXq9uW/kZ5uG1IRAUuAH4K3CJpjZLD+h3gMEkLpZPdOiqRsqUlDQb+BPyYiN7cbntLopBv1ZJO1nCyZ7D9Rgc62UsDlwFPunQaAv4CrCVpfdsTbP+GmAMb4j293l+3NynzylPAZkSe+pPA8uV6uY1wSg5StKF7PJ3saUddBXerEc7ec8Rz8Uhit+BO4HPAEUQ+8c3EDtQA22PSyf54yjy0KbHbcpCkO4FbiPv32wrl52uI3u8XAjOX2oqv1O1kQzhuSc2ULY4ZgIlbG5J2IaItjxGRq68RHR3+6VDjmrjKs/1iDWb3O5zbNz1lZaLF5GhgtKTfAzdI2sD2LyT9tuLMJC2gTDRTkgi/tLy3DpEr/83qblgnpvCUtJqriNSZnxUnaENCtOpsYDuF6MojRETsz9AZY+UQjVqfEJT6PRFp/QJwT3n9EtGRKZkG9J+S9XsAP3JI1t9IBChmdEjW36j/lKxPAamPoRLJ/jShjbEBsYN3KvCM7X0VKreLE0GfRQlfaQRAX5mXMqLdd/iAyJVr8GdgC9s7Af8ElrL9kKMSHEjHMWkNU4j8v0MoxDXe+xnwENGt4bN95WHWziglwrvDUoTD+HCJ/o8D1rH9HlHvMhbYnchx/67t39VmaYuRtKikjTVpS7mrCWf6V8D7RPHjlcClwOWuSdSjv6GUrG8ZKoXJFR/ndUJK/WuEwvOGpT5tPdsP276GklpLRLGfqcPujyJztGuirITnBv5l+1lFB4GNgNGuyIErBFiGE+pkT9djbdIpVCIIqwHzEeqZdxKO9Tjbh0tam5hAAJ63fUpN5rY1lb9FSoR3E0nbETnrywAP2N5nsvdnJea/d9t5zEq071vEvXwFIZjyjqTrgCdsf7tEC5ciBL1+187j0UxK0CEl65tMqQ8YTqTWXEKkNM1JFOguDGxs+0VFa9Nzga85urQhaV73wa5r6WjXQMkfHAW8AbxMpIbcSqzU5gH+j8hFWgY4BTi06nwnSSspOdmnEJHAtQixikOJSODfgDUINcK1CAfvqJpMbVsqTva6dLhE+LQyuYNYitIOIHI2b3HpHd5pjmRZVKwGfA/4A/AMkc96OqEzkAGcbqCUrG8ZisYPFxN57kOIsfxBWQBuRqQ8jSBE0bYj2pXe0Nfv6XS0e5myWhtF5Bs9SeRZLmj7GIXQzEbE9vC8RDTxp7bH1GVv0lmU7dDLidzf68qx+wgn+/vAQCLSsDyRJ7e97cdrMretUUqEd5uySFyb2IW5nciH3YFYpNxk+9UazasVRXvYZYnrZl5gOeAg2+fXalg/QylZ3xIkzUbMM+/Z3qgcOwaYnWhd/IGkVYBViXnoTpcmB33ZyYbM0a6DQcAKtm+3/Tei+njFssU3wPYo218hkv6/antMdstIWkmjYh4mdrF5hVjkNRhGFJu4OCoDiG35XdLJbj6V+z0lwrtBCWJ8n5iYtwSOIZztEcQOzCbq6m/ccdh+ucw7mxBb81cRO1RJ90jJ+iYjaa5SG3Ai8Jqkw8tb/yYWy/dIOhiY0/aZto93adfb151syIh2LSiEAs6xvUTJJTydkLn9F/BX4ka9FZjQHy6ipH9Siuhet/2mQkXrg3J8f2ISWcMhePJFIq1pC9tvlXNmKcVlSZOopIvMVRnnHwNbAMuUQqqkQmXMFgLWI6JhV5W0ms2JZ+pwoq/xi+5wVVhFu8cJ5d8NVeE+HxGsm8ZitjFOknYidvWWAM5rpHaWaPeORMpI2/ZkbyYlFeciYifgdWJHal8iPWR+ItCzPpELv195/5n+dM127Oq+ThyCEvtJepsoSBlcCp5mB44EXnGlRVeStIihRGeGxW3/Q9LMtt+3fWZJY7pF0s3AxkSdwFuND6aT3VwaDlBZhO8qaTyRV3ygpA8JifBVbI9Px6iL4mRvCpxEdG56DrjK9l2SJgDbEhoF38oxm7SFYWOOyXGZOhUHe0Ui7eYBIs/9K0TXlleJncCGZH2qaU4j5Zm2N+FUb2x7ZFnXnAL8shQ6Noodz6rOQ/2FjGjXSMnBHGF7kbptSTqT4tidDaxk+w1Js7qrT/umwAtEH9gH08FrPpON9+eIIuhdiWjZQoSi6XGSRhAFbUtRVIjrsrkvIemzhGLpIUTbw18CV9s+rry/FvB323+sz8qkv1J2SoaWhdtGRHpIo0j8DMLhXpFI6VoQGOYapb77G9U5pYzvuUTKzWWlEHxPIhh5fDlnBvfDfvcZ0a4Rh/TtbpJeIfpkv1G3TUln0dhdISKmK7uoOypa+G1M5AG/Xc5N566JqEiESzrN0Yt8YaLw8XZJv6ZIhEtazPbOConwfjfJtApJcxN9jJclgkYvKdqkXquQoD/SqZib9JBSu7IdMEjS/YTT9w1PKln/qu2LSsF4StZ3k7IjtR7wTpmL9gBOKQ71peVv8M2y6/psf33+ZTFkzdi+iWgNtELNpiQdiu1xRO7bgwCSliMKpW5tONlJc1FKhPeIauGn7TeBC4BfAHtLWsr2n4ge71tqUpGWJOkWTsn6ltG4j0tO+8HAXSU17haiK85BkoY5BLiG9fei0oxo9wFsj4XO6++a9B1sj5O0r6R3icr5PW2Pzmuy+SglwntEpfBxU0KCfn6ij/hIoi3qnpLOt/2EpM87FQ6T6cQpWd8Syn28IdEv+9vAs0RN0KaOVqYzAj+UdLOjO1u/JiPafYh0aJI6KZHtTYF90sluKSkR3gPK5LwaUfg4jhAEOY0IGP2ccHj2KV0M3q/N0KTfopSs701WBi60Pdr2/kQk+wZJK9n+BbBuOzjZkMWQSZJMgXSyW4tSInyakLQY8D+2LyivDwQWtX1web0vIR7SKBR9x/ZTddmb9G+UkvUtY/JxknQQsKzt3UsqyQBiAT2UaCX7h5pMbTqZOpIkyX+QE0fzqU40tn8u6TVCIvxXkubwpBLhE3tmd/jfYjBwZGk9eQ7wNLCcpAVtv2T77NJZZLGM/CfTi+2HFT2yG5L1K0h6hsgjPl3SkmUh91TlM518f04TlbSv1YhFzLtEmtxDkk62fbik1YHHy38bAuloJ0mSJNNOJS+xKhE+gJAI/4Skm2y/mhN3UCbn3yoU4U6S9A+i/eEwYPvSCeI9oli0o/PXk+ZRFrm3S9qaLsn6rxCS9f+PWOwl3aDy7DuFrvaIDwIrAWMljQTWIBRc1yJaJbYN6WgnSZL0AuqSCL+bkAhfs7z+ENgLmEHSZS4KnZ1OmZw3J0RnHiMKHz8k2qwdBaxCTMiH2H6yLjuT9sT2y8DLhNO9BXHPtkXOcG8jaSYixes7tq8rx+4DjgA2AAYCIvQDvkG0Tmwb0tFOkiRpEZUt04WIyOtJnlQi/ChCInwAIRGeTnZB0nzE9v1+wENE9OsM4APbB5VuLQvY/lvmySatoCGQYvt6STc6JeunmdIGsaE++m+FXki1cHQYcHi87VclLUjUrexi+/Het7h1ZNeRJEmSFlFpR3cLcCiwUzl+FzAamIeQCL/J9iO1GdqHqPTK/hB4HnjU9nhiJ+By4FxJ+9j+sNGVIB2fpBU4Jeu7jaTFJc1dFiXVYO7jxL27cHm9MDAEmAPA9kuEKuSjvWpwL5AR7SRJkhahkAjfB9iGIhEu6Vjbx9m+R9IMhER4x0/elUjhYOBl229IeonoOb6h7QmSniK6QXS8eE+S9FGGEq1LF7f9j1LI/L7tMyXNS/TLvplQHj7U9luND5YWp21HtvdLkiRpAQqJ8EOBHYFNipDKEsC1RAT7yFoN7INI2oTok30fUTR1E3Am8FlCmOYQYFvb9+UWfpL0TSRtRHQVWaksmGdtdFIqO3wvADPafrAT7uN0tJMkSZrEFHrFLkZ0LZgAnG37SUlDCdnwL9n+v5pM7XNIWgnYnxACWZqIjD1m+wJJuwIzAc86ZJqTJOnDSNoYOAtY2fbr5djawFeBIxvtTDuBdLSTJEmawMdIhA8iJMLnBhoS4bM5FeUmUraUfw08YntHSbMAWwOrAk8ScvX/+rjvSJKkb1Gc7bNtLyFpOeA2YK9G55FOIYshkyRJmkBFkCElwruJ7deAE4GNJG1TcjWvBB4mehkvUKd9SZJ0H9vjgH0lvQvcCuxp+7pKwXNHkBHtJEmSHpIS4T2jEv1fkYj0P2P7L5K2Ao4Djrd9delaMLjRXSRJkv6HpPWAeWxf2wk52ZOTXUeSJEl6TkqE94DiZG8MnA6MAK6Q9OUS7foQ+HHpYXwlKRKSJP0a27fBf9awdArpaCdJkvSAlAjvOUUlczjR4mtJ4APgQknftD2miNG8WqeNSZI0l050siFTR5IkSXpMRSJ8JmAF4BgiF/Eo4JOERPiptm+ozcg+QFHGnBt42/bz5dhSwHzA6bZXknQ4cALRM/v2ck5HRsCSJGkfMqKdJEnSA1IifNqQtDQwCngDeEnSGNtXllaHqwMPlFMfAH5DRaa5U8csSZL2IR3tJEmSblBxmieRCJdUlQhfoORsd7REeEkRGQUcTLTp24roItLgWWATSWcA6wJ72L6vt+1MkiRpFdneL0mSZBqotKQaDGD7DaAhEY7tCUBKhE/KIGAF27eXziE3A6tIWkHSQrbvAC4glOKOSCc7SZJ2I3O0kyRJppGUCO8+RY75nCJasR3RaeR5opf4E8Ao27eWc3PMkiRpKzJ1JEmSZBooEuFfI9IglgbWBgbZ/kZFInxi6kM6jIHtmyTtJ+lt4AnbgyUNAuYAjiBytxvn5pglSdJWZEQ7SZJkKqRE+PRTRCtG2F6kbluSJEl6i8zRTpIkmQopET79FNGK3SS9Imlg3fYkSZL0BhnRTpIkmYyUCG8dJc99fCmETJIkaWsyRztJkmQyUiK8ddgeC1n4mCRJZ5COdpIkyWSkRHjrSSc7SZJOIFNHkiTpeFIiPEmSJGkFGdFOkqSjSYnwJEmSpFWko50kSceSEuFJkiRJK0lHO0mSTmaiRDiApJuBMyStAPzd9h2lZ/bnSInwJEmSpJtkjnaSJB1NSoQnSZIkrSIj2kmSdDQpEZ4kSZK0ioxoJ0mSkBLhSZIkSfNJCfYkSRJSIjxJkiRpPhnRTpIkqZAS4UmSJEmzSEc7SZJkCmThY5IkSTK9pKOdJEmSJEmSJC0gc7STJEmSJEmSpAWko50kSZIkSZIkLSAd7SRJkiRJkiRpAeloJ0mSJEmSJEkLSEc7SZIkSZIkSVpAOtpJkiRJkiRJ0gL+P5/3xejr6DCdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dic_results = {'RandomForest_TF': eval_RF_tf_tts,\n",
    "               'RandomForest_TFIDF': eval_RF_tfidf_tts,\n",
    "               'LogiReg_TF': eval_LR_tf_tts,\n",
    "               'LogiReg_TFIDF': eval_LR_tfidf_tts,\n",
    "               'RGF_TF': eval_RGF_tf_tts,\n",
    "               'RGF_TFIDF': eval_RGF_tfidf_tts,\n",
    "               'Voting_TF': eval_vot_tf_tts,\n",
    "               'Voting_TFIDF': eval_vot_tfidf_tts,\n",
    "              }\n",
    "\n",
    "import operator\n",
    "tup_results = sorted(dic_results.items(), key=operator.itemgetter(1))\n",
    "\n",
    "N = len(dic_results)\n",
    "ind = np.arange(N)  # the x locations for the groups\n",
    "width = 0.40       # the width of the bars\n",
    "\n",
    "fig = plt.figure(figsize=(12,8))\n",
    "ax = fig.add_subplot(111)\n",
    "rects = ax.bar(ind, list(zip(*tup_results))[1], width,)\n",
    "for rect in rects:\n",
    "    height = rect.get_height()\n",
    "    ax.text(rect.get_x()+rect.get_width()/2., \n",
    "            1.005*height, \n",
    "            '{0:.4f}'.format(height), \n",
    "            ha='center', \n",
    "            va='bottom',)\n",
    "\n",
    "ax.set_ylabel('Scores')\n",
    "ax.set_ylim(ymin=0.78,ymax = 0.92)\n",
    "ax.set_title(\"Classificators' performance\")\n",
    "ax.set_xticks(ind + width/2.)\n",
    "ax.set_xticklabels(list(zip(*tup_results))[0], rotation=45)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a Submission  \n",
    "--\n",
    "\n",
    "All that remains is to run the best classifier on our test set and create a submission file. If you haven't already done so, download testData.tsv from the Data page. This file contains another 25,000 reviews and ids; our task is to predict the sentiment label.\n",
    "\n",
    "Note that when we use the Bag of Words for the test set, we only call \"transform\", not \"fit_transform\" as we did for the training set. In machine learning, you shouldn't use the test set to fit your model, otherwise you run the risk of overfitting. For this reason, we keep the test set off-limits until we are ready to make predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 25000 entries, 0 to 24999\n",
      "Data columns (total 2 columns):\n",
      "id        25000 non-null object\n",
      "review    25000 non-null object\n",
      "dtypes: object(2)\n",
      "memory usage: 390.7+ KB\n"
     ]
    }
   ],
   "source": [
    "test.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"12311_10\"</td>\n",
       "      <td>\"Naturally in a film who's main themes are of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"8348_2\"</td>\n",
       "      <td>\"This movie is a disaster within a disaster fi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"5828_4\"</td>\n",
       "      <td>\"All in all, this is a movie for kids. We saw ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"7186_2\"</td>\n",
       "      <td>\"Afraid of the Dark left me with the impressio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"12128_7\"</td>\n",
       "      <td>\"A very accurate depiction of small time mob l...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id                                             review\n",
       "0  \"12311_10\"  \"Naturally in a film who's main themes are of ...\n",
       "1    \"8348_2\"  \"This movie is a disaster within a disaster fi...\n",
       "2    \"5828_4\"  \"All in all, this is a movie for kids. We saw ...\n",
       "3    \"7186_2\"  \"Afraid of the Dark left me with the impressio...\n",
       "4   \"12128_7\"  \"A very accurate depiction of small time mob l..."
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 5000)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data_features_tfidf.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the logistic regression with tfidf vectors to make sentiment label predictions\n",
    "result = clf_LR_tfidf.predict(test_data_features_tfidf)\n",
    "result_prob = clf_LR_tfidf.predict_proba(test_data_features_tfidf)\n",
    "output = pd.DataFrame(data={\"id\":test[\"id\"], \"sentiment\":result,})# \"probs\":result_prob[:,1]})\n",
    "# Use pandas to write the comma-separated output file\n",
    "output.to_csv(os.path.join(outputs,'LR_tfidf_model.csv'), index=False, quoting=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"12311_10\"</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"8348_2\"</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"5828_4\"</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"7186_2\"</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"12128_7\"</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id  sentiment\n",
       "0  \"12311_10\"          0\n",
       "1    \"8348_2\"          0\n",
       "2    \"5828_4\"          0\n",
       "3    \"7186_2\"          0\n",
       "4   \"12128_7\"          0"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part 2: Alternative Vectors\n",
    "--\n",
    "\n",
    "In the subsequent sections, we are going to explore alternate ways to codify text into vectors. We are going to explore three techniques, namely Latent Semantic Indexing (LSI), Latent Dirichlet Allocation (LDA) and Word2vec."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Topic Modeling and Topic Vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To build the models, the more we have texts, the better. The size of the Corpus is essential for having good results. We don't need labels in order to create the models, so we will use the train examples and also some unlabeled reviews. The list of cleaned sentences will be used for all the subsequent models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing sentences from training set\n",
      "Parsing sentences from unlabeled set\n",
      "Parsing sentences from test set\n",
      "Parsing sentences with stopwords from training set\n",
      "Parsing sentences with stopwords from unlabeled set\n",
      "Parsing sentences with stopwords from test set\n"
     ]
    }
   ],
   "source": [
    "print(\"Parsing sentences from training set\")\n",
    "labeled_sentences = Text_Cleaning_Utilities.df_to_list_of_tokens(train,\n",
    "                                                                 'review', \n",
    "                                                                 remove_html=True,\n",
    "                                                                 remove_stopwords=True,)\n",
    "print(\"Parsing sentences from unlabeled set\")\n",
    "all_sentences = labeled_sentences + Text_Cleaning_Utilities.df_to_list_of_tokens(unlabeled_train,\n",
    "                                                                                 'review', \n",
    "                                                                                 remove_html=True,\n",
    "                                                                                 remove_stopwords=True,)\n",
    "\n",
    "print(\"Parsing sentences from test set\")\n",
    "test_labeled_sentences = Text_Cleaning_Utilities.df_to_list_of_tokens(test,\n",
    "                                                                      'review', \n",
    "                                                                      remove_html=True,\n",
    "                                                                      remove_stopwords=True,)\n",
    "\n",
    "print(\"Parsing sentences with stopwords from training set\")\n",
    "labeled_sentences_sw = Text_Cleaning_Utilities.df_to_list_of_tokens(train,\n",
    "                                                                    'review', \n",
    "                                                                    remove_html=True,\n",
    "                                                                    remove_stopwords=False,)\n",
    "\n",
    "print(\"Parsing sentences with stopwords from unlabeled set\")\n",
    "all_sentences_sw = labeled_sentences_sw + Text_Cleaning_Utilities.df_to_list_of_tokens(unlabeled_train,\n",
    "                                                                                       'review', \n",
    "                                                                                       remove_html=True,\n",
    "                                                                                       remove_stopwords=False,)\n",
    "\n",
    "print(\"Parsing sentences with stopwords from test set\")\n",
    "test_labeled_sentences_sw = Text_Cleaning_Utilities.df_to_list_of_tokens(test,\n",
    "                                                                         'review', \n",
    "                                                                         remove_html=True,\n",
    "                                                                         remove_stopwords=False,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000\n",
      "75000\n",
      "25000\n",
      "25000\n",
      "75000\n",
      "25000\n"
     ]
    }
   ],
   "source": [
    "print(len(labeled_sentences))\n",
    "print(len(all_sentences))\n",
    "print(len(test_labeled_sentences))\n",
    "print(len(labeled_sentences_sw))\n",
    "print(len(all_sentences_sw))\n",
    "print(len(test_labeled_sentences_sw))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving sentences to a Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(outputs, 'sentences.pkl'),'wb') as f:\n",
    "    pickle.dump((labeled_sentences,\n",
    "                 all_sentences,\n",
    "                 test_labeled_sentences,\n",
    "                 labeled_sentences_sw,\n",
    "                 all_sentences_sw,\n",
    "                 test_labeled_sentences_sw,), f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading sentences from a Pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(outputs, 'sentences.pkl'),'rb') as f:\n",
    "    (labeled_sentences,\n",
    "     all_sentences,\n",
    "     test_labeled_sentences,\n",
    "     labeled_sentences_sw,\n",
    "     all_sentences_sw,\n",
    "     test_labeled_sentences_sw,) = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nltk_stopwords():\n",
    "    return set(nltk.corpus.stopwords.words('english'))\n",
    "\n",
    "def prep_corpus(docs, additional_stopwords=set(), no_below=4, no_above=0.7):\n",
    "    print('Building dictionary...')\n",
    "    dictionary = corpora.Dictionary(docs)\n",
    "    print('{} Tokens extracted from {} texts'.format(len(dictionary.keys()), dictionary.num_docs))\n",
    "    stopwords = nltk_stopwords().union(additional_stopwords)\n",
    "    #stopword_ids = [dictionary.token2id[sw] for sw in stopwords if sw in dictionary.token2id]\n",
    "    stopword_ids = map(dictionary.token2id.get, stopwords)\n",
    "    dictionary.filter_tokens(stopword_ids)\n",
    "    #low_freq_ids = [tokenid for tokenid, docfreq in dictionary.dfs.items() if docfreq < 4]\n",
    "    #dictionary.filter_tokens(low_freq_ids)\n",
    "    dictionary.filter_extremes(no_below=no_below, no_above=no_above, keep_n=None)\n",
    "    dictionary.compactify()\n",
    "    print('{} Tokens after cleaning'.format(len(dictionary.keys())))\n",
    "    #print('Building corpus...')\n",
    "    #corpus = [dictionary.doc2bow(doc) for doc in docs]\n",
    "    return dictionary #, corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compacting and saving the dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building dictionary...\n",
      "256506 Tokens extracted from 75000 texts\n",
      "56367 Tokens after cleaning\n",
      "dictionary done\n",
      "dictionary saved\n"
     ]
    }
   ],
   "source": [
    "additional_stopwords=set(['n\\'t', 'movie'])\n",
    "\n",
    "dictionary = prep_corpus(all_sentences, additional_stopwords)\n",
    "dictionary.compactify()\n",
    "print('dictionary done')\n",
    "\n",
    "dictionary.save(os.path.join(outputs, 'reviews.dict'))\n",
    "print('dictionary saved')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "81\n"
     ]
    }
   ],
   "source": [
    "#print(dictionary.token2id['movie']) #verify if these words were in the stopwords list\n",
    "#print(dictionary.token2id['n\\'t'])\n",
    "print(dictionary.token2id['like'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = corpora.dictionary.Dictionary.load(os.path.join(outputs, 'reviews.dict'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generating the Corpora (tf and tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "corpus tf done\n",
      "corpus tfidf done\n"
     ]
    }
   ],
   "source": [
    "corpus_tf = [dictionary.doc2bow(sentence) for sentence in all_sentences]\n",
    "print('corpus tf done')\n",
    "tfidf = models.TfidfModel(corpus_tf)\n",
    "corpus_tfidf = tfidf[corpus_tf]\n",
    "print('corpus tfidf done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving the Corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpora.MmCorpus.serialize(os.path.join(outputs, 'corpus_tf.mm'), corpus_tf)\n",
    "corpora.MmCorpus.serialize(os.path.join(outputs, 'corpus_tfidf.mm'), corpus_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the Corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_tf = corpora.MmCorpus(os.path.join(outputs, 'corpus_tf.mm'))\n",
    "corpus_tfidf = corpora.MmCorpus(os.path.join(outputs, 'corpus_tfidf.mm'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Latent Semantic Indexing\n",
    "https://en.wikipedia.org/wiki/Latent_semantic_analysis  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generating the Models and the Corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "corpus lsi - TF done\n",
      "corpus lsi - TFIDF done\n"
     ]
    }
   ],
   "source": [
    "lsi_tf = models.LsiModel(corpus_tf, id2word=dictionary, num_topics=10)\n",
    "corpus_lsi_tf = lsi_tf[corpus_tf]\n",
    "print('corpus lsi - TF done')\n",
    "lsi_tfidf = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=10)\n",
    "corpus_lsi_tfidf = lsi_tfidf[corpus_tfidf]\n",
    "print('corpus lsi - TFIDF done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving the Models and the Corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "lsi_tf.save(os.path.join(outputs, 'model_tf.lsi'))\n",
    "corpora.MmCorpus.serialize(os.path.join(outputs, 'corpus_lsi_tf.mm'), corpus_lsi_tf)\n",
    "lsi_tfidf.save(os.path.join(outputs, 'model_tfidf.lsi'))\n",
    "corpora.MmCorpus.serialize(os.path.join(outputs, 'corpus_lsi_tfidf.mm'), corpus_lsi_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the Models and the Corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "lsi_tf = models.LsiModel.load(os.path.join(outputs, 'model_tf.lsi'))\n",
    "corpus_lsi_tf = corpora.MmCorpus(os.path.join(outputs, 'corpus_lsi_tf.mm'))\n",
    "lsi_tfidf = models.LsiModel.load(os.path.join(outputs, 'model_tfidf.lsi'))\n",
    "corpus_lsi_tfidf = corpora.MmCorpus(os.path.join(outputs, 'corpus_lsi_tfidf.mm'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visualizing the topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.508*\"film\" + 0.290*\"one\" + 0.218*\"like\" + 0.147*\"would\" + 0.145*\"good\" + 0.137*\"even\" + 0.125*\"time\" + 0.123*\"really\" + 0.119*\"story\" + 0.117*\"see\"'),\n",
       " (1,\n",
       "  '-0.833*\"film\" + 0.214*\"one\" + 0.195*\"like\" + 0.113*\"show\" + 0.105*\"movies\" + 0.105*\"good\" + 0.101*\"would\" + 0.093*\"even\" + 0.092*\"really\" + 0.085*\"get\"'),\n",
       " (2,\n",
       "  '-0.728*\"one\" + 0.456*\"like\" + 0.179*\"bad\" + 0.164*\"really\" + 0.152*\"good\" + 0.124*\"would\" + 0.095*\"people\" + 0.079*\"think\" + -0.076*\"two\" + 0.072*\"could\"'),\n",
       " (3,\n",
       "  '-0.492*\"one\" + -0.424*\"like\" + 0.332*\"story\" + -0.243*\"bad\" + 0.138*\"show\" + 0.138*\"great\" + 0.137*\"also\" + 0.137*\"life\" + 0.129*\"love\" + 0.119*\"character\"'),\n",
       " (4,\n",
       "  '-0.622*\"good\" + 0.408*\"like\" + -0.311*\"bad\" + -0.248*\"really\" + 0.157*\"would\" + 0.144*\"people\" + 0.131*\"show\" + -0.129*\"acting\" + 0.118*\"life\" + -0.116*\"great\"'),\n",
       " (5,\n",
       "  '-0.523*\"like\" + 0.477*\"would\" + 0.235*\"bad\" + -0.226*\"story\" + 0.223*\"even\" + -0.197*\"good\" + 0.195*\"could\" + -0.184*\"great\" + -0.144*\"also\" + 0.100*\"get\"'),\n",
       " (6,\n",
       "  '-0.689*\"show\" + 0.390*\"story\" + -0.179*\"great\" + 0.173*\"even\" + -0.161*\"good\" + -0.147*\"series\" + 0.147*\"bad\" + -0.114*\"really\" + -0.103*\"episode\" + -0.091*\"shows\"'),\n",
       " (7,\n",
       "  '-0.679*\"would\" + -0.275*\"story\" + 0.226*\"bad\" + 0.200*\"get\" + -0.187*\"good\" + 0.169*\"time\" + -0.161*\"one\" + -0.156*\"like\" + 0.139*\"really\" + 0.138*\"people\"'),\n",
       " (8,\n",
       "  '-0.509*\"really\" + -0.340*\"people\" + -0.326*\"see\" + 0.268*\"even\" + -0.241*\"story\" + 0.195*\"good\" + -0.141*\"think\" + 0.135*\"time\" + 0.129*\"first\" + -0.122*\"love\"'),\n",
       " (9,\n",
       "  '-0.472*\"story\" + -0.387*\"show\" + 0.349*\"good\" + -0.245*\"even\" + -0.244*\"characters\" + 0.219*\"would\" + -0.187*\"bad\" + 0.169*\"see\" + 0.143*\"man\" + 0.116*\"get\"')]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsi_tf.print_topics(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('film', -0.8328195635576635),\n",
       " ('one', 0.2135196263031167),\n",
       " ('like', 0.19501541415258306),\n",
       " ('show', 0.11342031440814408),\n",
       " ('movies', 0.10545667270756386),\n",
       " ('good', 0.10523861901988525),\n",
       " ('would', 0.1006687021716853),\n",
       " ('even', 0.09279793115514301),\n",
       " ('really', 0.09158421010788059),\n",
       " ('get', 0.08474444565434824)]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsi_tf.show_topic(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.146*\"film\" + 0.103*\"bad\" + 0.101*\"good\" + 0.101*\"like\" + 0.101*\"really\" + 0.095*\"one\" + 0.094*\"would\" + 0.091*\"story\" + 0.089*\"see\" + 0.088*\"even\"'),\n",
       " (1,\n",
       "  '0.334*\"bad\" + 0.199*\"worst\" + 0.133*\"waste\" + 0.129*\"movies\" + 0.128*\"acting\" + 0.115*\"terrible\" + 0.114*\"ever\" + 0.106*\"horrible\" + 0.106*\"horror\" + 0.101*\"stupid\"'),\n",
       " (2,\n",
       "  '0.672*\"show\" + 0.215*\"episode\" + 0.212*\"series\" + -0.169*\"film\" + 0.151*\"episodes\" + 0.147*\"tv\" + 0.141*\"season\" + -0.138*\"horror\" + 0.124*\"funny\" + 0.116*\"shows\"'),\n",
       " (3,\n",
       "  '0.231*\"show\" + -0.225*\"book\" + -0.221*\"great\" + 0.173*\"horror\" + -0.114*\"movies\" + -0.111*\"seen\" + -0.111*\"love\" + -0.108*\"read\" + -0.103*\"best\" + 0.103*\"killer\"'),\n",
       " (4,\n",
       "  '-0.237*\"horror\" + -0.232*\"series\" + -0.171*\"action\" + -0.164*\"show\" + 0.153*\"life\" + 0.153*\"people\" + -0.142*\"effects\" + -0.128*\"original\" + -0.122*\"great\" + -0.112*\"episode\"'),\n",
       " (5,\n",
       "  '0.378*\"book\" + -0.365*\"funny\" + -0.269*\"comedy\" + 0.179*\"series\" + 0.155*\"read\" + -0.131*\"jokes\" + -0.130*\"great\" + -0.123*\"fun\" + -0.117*\"laugh\" + -0.094*\"humor\"'),\n",
       " (6,\n",
       "  '0.423*\"horror\" + -0.247*\"bad\" + -0.145*\"worst\" + 0.133*\"scary\" + 0.115*\"gore\" + -0.114*\"script\" + -0.114*\"show\" + 0.105*\"saw\" + 0.103*\"kids\" + -0.099*\"acting\"'),\n",
       " (7,\n",
       "  '-0.279*\"book\" + 0.257*\"ever\" + 0.222*\"worst\" + 0.202*\"seen\" + -0.168*\"characters\" + -0.152*\"really\" + -0.128*\"character\" + 0.123*\"dvd\" + 0.119*\"horror\" + 0.119*\"years\"'),\n",
       " (8,\n",
       "  '0.515*\"book\" + -0.184*\"film\" + 0.159*\"read\" + -0.145*\"show\" + 0.141*\"version\" + -0.128*\"people\" + -0.104*\"films\" + -0.103*\"characters\" + 0.099*\"original\" + 0.098*\"series\"'),\n",
       " (9,\n",
       "  '-0.365*\"horror\" + 0.312*\"action\" + 0.288*\"game\" + -0.200*\"book\" + -0.181*\"comedy\" + -0.180*\"funny\" + -0.170*\"show\" + -0.140*\"film\" + 0.121*\"war\" + 0.104*\"series\"')]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsi_tfidf.print_topics(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('bad', 0.3341800294443265),\n",
       " ('worst', 0.19876667926857763),\n",
       " ('waste', 0.1325875480378994),\n",
       " ('movies', 0.12876426810253105),\n",
       " ('acting', 0.1284150471131665),\n",
       " ('terrible', 0.11486275640415528),\n",
       " ('ever', 0.11396154942214519),\n",
       " ('horrible', 0.10597840366885512),\n",
       " ('horror', 0.10555413670315475),\n",
       " ('stupid', 0.1013886678729634)]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsi_tfidf.show_topic(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Latent Dirichlet Allocation\n",
    "\n",
    "https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generating the Models and the Corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "corpus lda tf done\n",
      "corpus lda tfidf done\n"
     ]
    }
   ],
   "source": [
    "lda_tf = models.LdaModel(corpus_tf, id2word=dictionary, num_topics=10, passes=10)\n",
    "corpus_lda_tf = lda_tf[corpus_tf]\n",
    "print('corpus lda tf done')\n",
    "\n",
    "lda_tfidf = models.LdaModel(corpus_tfidf, id2word=dictionary, num_topics=10, passes=10)\n",
    "corpus_lda_tfidf = lda_tfidf[corpus_tfidf]\n",
    "print('corpus lda tfidf done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving the Models and the Corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_tf.save(os.path.join(outputs, 'model_tf.lda'))\n",
    "corpora.MmCorpus.serialize(os.path.join(outputs, 'corpus_lda_tf.mm'), corpus_lda_tf)\n",
    "\n",
    "lda_tfidf.save(os.path.join(outputs, 'model_tfidf.lda'))\n",
    "corpora.MmCorpus.serialize(os.path.join(outputs, 'corpus_lda_tfidf.mm'), corpus_lda_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the Models and the Corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_tf = models.LdaModel.load(os.path.join(outputs, 'model_tf.lda'))\n",
    "corpus_lda_tf = corpora.MmCorpus(os.path.join(outputs, 'corpus_lda_tf.mm'))\n",
    "\n",
    "lda_tfidf = models.LdaModel.load(os.path.join(outputs, 'model_tfidf.lda'))\n",
    "corpus_lda_tfidf = corpora.MmCorpus(os.path.join(outputs, 'corpus_lda_tfidf.mm'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visualizing the topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.009*\"role\" + 0.009*\"good\" + 0.008*\"character\" + 0.008*\"john\" + 0.007*\"jack\" + 0.007*\"performance\" + 0.007*\"best\"'),\n",
       " (1,\n",
       "  '0.017*\"film\" + 0.013*\"horror\" + 0.009*\"one\" + 0.006*\"also\" + 0.006*\"killer\" + 0.005*\"films\" + 0.004*\"plot\"'),\n",
       " (2,\n",
       "  '0.026*\"film\" + 0.009*\"story\" + 0.008*\"people\" + 0.007*\"one\" + 0.007*\"life\" + 0.006*\"characters\" + 0.005*\"many\"'),\n",
       " (3,\n",
       "  '0.009*\"one\" + 0.008*\"like\" + 0.006*\"action\" + 0.006*\"get\" + 0.004*\"bad\" + 0.004*\"back\" + 0.004*\"effects\"'),\n",
       " (4,\n",
       "  '0.026*\"film\" + 0.010*\"one\" + 0.008*\"great\" + 0.007*\"best\" + 0.007*\"cast\" + 0.007*\"role\" + 0.006*\"films\"'),\n",
       " (5,\n",
       "  '0.008*\"young\" + 0.007*\"man\" + 0.007*\"woman\" + 0.007*\"girl\" + 0.007*\"wife\" + 0.006*\"father\" + 0.006*\"family\"'),\n",
       " (6,\n",
       "  '0.023*\"film\" + 0.012*\"like\" + 0.012*\"bad\" + 0.011*\"even\" + 0.011*\"one\" + 0.009*\"would\" + 0.009*\"could\"'),\n",
       " (7,\n",
       "  '0.009*\"action\" + 0.008*\"rock\" + 0.007*\"school\" + 0.007*\"lee\" + 0.007*\"harry\" + 0.006*\"high\" + 0.006*\"arts\"'),\n",
       " (8,\n",
       "  '0.015*\"like\" + 0.015*\"one\" + 0.012*\"good\" + 0.011*\"really\" + 0.011*\"see\" + 0.010*\"great\" + 0.010*\"show\"'),\n",
       " (9,\n",
       "  '0.012*\"war\" + 0.007*\"american\" + 0.007*\"film\" + 0.006*\"years\" + 0.005*\"time\" + 0.004*\"history\" + 0.004*\"one\"')]"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_tf.print_topics(num_topics=10, num_words=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('young', 0.0077159572),\n",
       " ('man', 0.0073920405),\n",
       " ('woman', 0.007067852),\n",
       " ('girl', 0.006798674),\n",
       " ('wife', 0.0065720584),\n",
       " ('father', 0.006171629),\n",
       " ('family', 0.0061167222),\n",
       " ('love', 0.0060660196),\n",
       " ('life', 0.0057403566),\n",
       " ('mother', 0.005687796)]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_tf.show_topic(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.003*\"troma\" + 0.002*\"chicago\" + 0.002*\"holden\" + 0.002*\"campbell\" + 0.002*\"christie\" + 0.002*\"marx\" + 0.002*\"cells\"'),\n",
       " (1,\n",
       "  '0.002*\"quaid\" + 0.002*\"irene\" + 0.002*\"louise\" + 0.002*\"streep\" + 0.001*\"welles\" + 0.001*\"wasting\" + 0.001*\"niro\"'),\n",
       " (2,\n",
       "  '0.002*\"shark\" + 0.002*\"ninja\" + 0.002*\"holocaust\" + 0.002*\"powell\" + 0.002*\"tarantino\" + 0.002*\"pink\" + 0.001*\"eastwood\"'),\n",
       " (3,\n",
       "  '0.003*\"bogart\" + 0.003*\"wilder\" + 0.003*\"punk\" + 0.002*\"todd\" + 0.002*\"row\" + 0.002*\"hudson\" + 0.002*\"movie.i\"'),\n",
       " (4,\n",
       "  '0.003*\"film\" + 0.002*\"good\" + 0.002*\"like\" + 0.002*\"bad\" + 0.002*\"one\" + 0.002*\"really\" + 0.002*\"story\"'),\n",
       " (5,\n",
       "  '0.001*\"moore\" + 0.001*\"scorsese\" + 0.001*\"teenagers\" + 0.001*\"ice\" + 0.001*\"sarah\" + 0.001*\"michelle\" + 0.001*\"predator\"'),\n",
       " (6,\n",
       "  '0.002*\"alley\" + 0.002*\"dont\" + 0.002*\"dolph\" + 0.002*\"sheep\" + 0.002*\"morals\" + 0.002*\"bullock\" + 0.002*\"manga\"'),\n",
       " (7,\n",
       "  '0.004*\"mitchum\" + 0.003*\"sopranos\" + 0.003*\"damme\" + 0.003*\"wesley\" + 0.003*\"mins\" + 0.002*\"sharpe\" + 0.002*\"dud\"'),\n",
       " (8,\n",
       "  '0.001*\"sandler\" + 0.001*\"former\" + 0.001*\"holmes\" + 0.001*\"henry\" + 0.001*\"chuck\" + 0.001*\"return\" + 0.001*\"chaplin\"'),\n",
       " (9,\n",
       "  '0.003*\"khan\" + 0.003*\"franco\" + 0.003*\"costello\" + 0.002*\"juliet\" + 0.002*\"campus\" + 0.002*\"rani\" + 0.002*\"cate\"')]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_tfidf.print_topics(num_topics=10, num_words=7)\n",
    "#lda_tfidf.print_topics(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  [('troma', 0.0029572477),\n",
       "   ('chicago', 0.0024606744),\n",
       "   ('holden', 0.0023822414),\n",
       "   ('campbell', 0.0023791383),\n",
       "   ('christie', 0.0022772383),\n",
       "   ('marx', 0.0020162796),\n",
       "   ('cells', 0.001843523),\n",
       "   ('sarandon', 0.001827529),\n",
       "   ('candidate', 0.0018166029),\n",
       "   ('angela', 0.0017963027),\n",
       "   ('christine', 0.0017839267),\n",
       "   ('21', 0.0017588589),\n",
       "   ('dundee', 0.0016579692),\n",
       "   ('beowulf', 0.0016569836),\n",
       "   ('dates', 0.0016428282),\n",
       "   ('farrah', 0.0016176904),\n",
       "   ('jules', 0.0016058786),\n",
       "   ('agatha', 0.0015885248),\n",
       "   ('caine', 0.0015868405),\n",
       "   ('hockey', 0.0015569164)]),\n",
       " (1,\n",
       "  [('quaid', 0.002209951),\n",
       "   ('irene', 0.0018733585),\n",
       "   ('louise', 0.0015487766),\n",
       "   ('streep', 0.0015391754),\n",
       "   ('welles', 0.0014544016),\n",
       "   ('wasting', 0.0014451345),\n",
       "   ('niro', 0.0014290058),\n",
       "   ('twin', 0.0013793036),\n",
       "   ('jenny', 0.0012598671),\n",
       "   ('chicken', 0.0012583536),\n",
       "   ('unpleasant', 0.0012221851),\n",
       "   ('troubled', 0.0012134993),\n",
       "   ('illness', 0.0012111578),\n",
       "   ('bernie', 0.0011949404),\n",
       "   ('storm', 0.001179316),\n",
       "   ('shields', 0.0011688041),\n",
       "   ('gorilla', 0.0011669915),\n",
       "   ('jill', 0.0011603693),\n",
       "   ('cassavetes', 0.001147288),\n",
       "   ('pretends', 0.0011308873)])]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_tfidf.show_topics(formatted=False, num_words=20)[0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualizing using PyLDAvis  \n",
    "http://nbviewer.jupyter.org/github/bmabey/pyLDAvis/blob/master/notebooks/pyLDAvis_overview.ipynb  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/pyLDAvis/_prepare.py:387: DeprecationWarning: \n",
      ".ix is deprecated. Please use\n",
      ".loc for label based indexing or\n",
      ".iloc for positional indexing\n",
      "\n",
      "See the documentation here:\n",
      "http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated\n",
      "  topic_term_dists = topic_term_dists.ix[topic_order]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el1437551404698944397044237069456\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el1437551404698944397044237069456_data = {\"mdsDat\": {\"Freq\": [21.775693893432617, 16.94452476501465, 16.244983673095703, 8.819421768188477, 8.718465805053711, 8.077996253967285, 7.723182678222656, 5.885867595672607, 3.719498634338379, 2.0903635025024414], \"cluster\": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], \"topics\": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], \"x\": [0.1650851653641202, 0.13851786117902296, 0.15382370571790288, 0.01653646142194692, 0.05164250076709957, 0.06378147643827244, -0.012106455844236095, 0.006707408930647591, -0.18131813582209075, -0.4026699881526859], \"y\": [0.04973699527648374, 0.01677649462871187, -0.0006881556713676289, 0.09015624172491701, 0.1799479435843167, 0.00038282124998353695, -0.20887384715052937, -0.05155092130698725, -0.2438546129314231, 0.1679670405958948]}, \"tinfo\": {\"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic5\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic6\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic7\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic8\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic9\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\", \"Topic10\"], \"Freq\": [117884.0, 27038.0, 43300.0, 10177.0, 27211.0, 16980.0, 10312.0, 34704.0, 6023.0, 58839.0, 18912.0, 9371.0, 17378.0, 13026.0, 20484.0, 11272.0, 19807.0, 8258.0, 27723.0, 21069.0, 18407.0, 9363.0, 22479.0, 19103.0, 33745.0, 27452.0, 10575.0, 7637.0, 7980.0, 6739.0, 914.6619262695312, 4335.8330078125, 589.273681640625, 543.2855224609375, 403.1495666503906, 384.41229248046875, 370.92919921875, 368.6746826171875, 334.8665466308594, 857.329345703125, 289.2114562988281, 278.3360290527344, 276.0347900390625, 263.0599365234375, 247.02125549316406, 235.60195922851562, 233.66822814941406, 194.1149444580078, 181.89816284179688, 177.88002014160156, 176.5033721923828, 175.05203247070312, 160.67018127441406, 160.5576171875, 155.57298278808594, 154.249267578125, 151.40423583984375, 140.43690490722656, 140.1122589111328, 137.94525146484375, 2012.4263916015625, 7444.28466796875, 385.01806640625, 1511.48828125, 4615.79736328125, 1467.1375732421875, 1695.076904296875, 4284.1875, 638.395263671875, 433.1548767089844, 3825.37353515625, 579.9306030273438, 4967.9912109375, 3379.702392578125, 1119.38916015625, 723.4860229492188, 21403.76171875, 6877.63134765625, 717.5801391601562, 1049.99169921875, 1919.8907470703125, 4444.32373046875, 1600.5859375, 3472.127685546875, 6713.96923828125, 12547.0107421875, 8979.2822265625, 1436.1787109375, 2257.568115234375, 12062.5654296875, 3704.815673828125, 16502.396484375, 19931.0703125, 2238.62841796875, 4592.94677734375, 43554.58203125, 3238.069091796875, 17629.912109375, 11664.8427734375, 5018.2626953125, 22377.27734375, 7137.92724609375, 8551.16796875, 5388.41748046875, 14019.0625, 6877.51513671875, 16007.6904296875, 8273.6962890625, 9758.041015625, 19831.716796875, 6520.4365234375, 7321.841796875, 10386.544921875, 11160.3837890625, 7207.736328125, 9050.7509765625, 7714.7041015625, 8164.38232421875, 7725.42919921875, 8838.4677734375, 7627.30419921875, 7770.00537109375, 7367.91748046875, 7193.3349609375, 1734.9759521484375, 1530.9444580078125, 780.5415649414062, 769.6229858398438, 701.7748413085938, 597.78173828125, 413.1301574707031, 569.5257568359375, 339.79754638671875, 331.4488220214844, 289.5357360839844, 283.8790588378906, 260.7586975097656, 216.91539001464844, 214.1055145263672, 197.46533203125, 192.90550231933594, 185.2196807861328, 184.2484130859375, 179.64451599121094, 170.2903594970703, 161.8174591064453, 160.5159912109375, 155.85910034179688, 153.79931640625, 151.6093292236328, 150.82872009277344, 142.61459350585938, 141.91090393066406, 139.0572052001953, 489.4582214355469, 1808.9586181640625, 1923.9769287109375, 3916.933837890625, 1436.1197509765625, 510.5643005371094, 14660.9501953125, 858.56201171875, 404.04974365234375, 3381.06103515625, 1224.7999267578125, 5627.69775390625, 3510.6806640625, 6373.0439453125, 3680.380615234375, 3860.708740234375, 6969.89208984375, 1045.2572021484375, 1236.1116943359375, 2766.60693359375, 4407.9990234375, 15028.4306640625, 4527.3525390625, 7775.74951171875, 2507.739990234375, 10726.0546875, 11796.685546875, 15951.8310546875, 16034.6435546875, 10450.7001953125, 3657.455810546875, 17600.095703125, 21630.0546875, 11192.3076171875, 4472.40771484375, 8572.1533203125, 5237.66943359375, 21144.677734375, 7038.3525390625, 12715.775390625, 11532.134765625, 7620.52099609375, 9945.7939453125, 6077.17529296875, 7293.69921875, 7102.421875, 5982.57861328125, 8266.474609375, 5723.244140625, 7236.93603515625, 7740.45068359375, 8125.40087890625, 6348.115234375, 7221.6123046875, 7120.7353515625, 6981.05029296875, 7093.01513671875, 1963.1171875, 565.1730346679688, 521.7711181640625, 417.6272277832031, 339.1147155761719, 329.7833251953125, 319.2347717285156, 286.9775390625, 285.613037109375, 280.3320007324219, 272.1143798828125, 271.277587890625, 255.05679321289062, 243.8313446044922, 240.7655487060547, 240.4490966796875, 233.83615112304688, 232.37542724609375, 225.42591857910156, 222.1154327392578, 216.84371948242188, 216.66278076171875, 210.21966552734375, 206.66734313964844, 200.87469482421875, 195.25685119628906, 193.89955139160156, 193.60928344726562, 192.11268615722656, 187.2212677001953, 2854.405029296875, 296.36083984375, 237.93081665039062, 516.1334838867188, 1762.14990234375, 306.9811706542969, 1068.439453125, 1587.1326904296875, 631.0365600585938, 1269.554443359375, 1451.5250244140625, 809.7268676757812, 727.0096435546875, 711.9398193359375, 1428.543701171875, 1563.0634765625, 1879.642578125, 913.0806274414062, 789.447509765625, 3233.072265625, 747.9640502929688, 1598.25537109375, 872.9312744140625, 544.0424194335938, 2032.494140625, 6565.71728515625, 1103.2421875, 6541.32373046875, 9469.80859375, 844.078125, 1431.3074951171875, 2297.36669921875, 36304.09765625, 2070.6083984375, 2569.28515625, 10525.037109375, 12103.0791015625, 1765.8878173828125, 3318.989990234375, 3360.245361328125, 8136.857421875, 1922.119384765625, 1826.95068359375, 6700.80712890625, 5089.35546875, 3305.889404296875, 3917.683837890625, 3602.386962890625, 5837.4052734375, 3781.048828125, 3510.217041015625, 6048.42724609375, 10051.654296875, 5583.0830078125, 6119.46484375, 4571.24609375, 5228.01611328125, 5204.74365234375, 3568.956298828125, 5514.53857421875, 3890.927001953125, 5859.95947265625, 4156.25927734375, 3599.87646484375, 3697.76416015625, 3543.798583984375, 3485.560791015625, 1758.6107177734375, 1498.45947265625, 1083.788330078125, 736.5951538085938, 677.2265014648438, 648.86865234375, 558.5570678710938, 554.9824829101562, 458.3091125488281, 434.6596374511719, 400.58013916015625, 346.375244140625, 332.6044616699219, 332.3001403808594, 308.90679931640625, 307.033935546875, 298.6238708496094, 287.18841552734375, 276.99822998046875, 255.86581420898438, 250.04241943359375, 237.79403686523438, 234.8423309326172, 227.39340209960938, 226.99374389648438, 225.5158233642578, 214.0464324951172, 212.50584411621094, 212.209716796875, 209.67433166503906, 3039.100830078125, 281.2674865722656, 3379.235595703125, 4286.07421875, 4650.66943359375, 568.5272827148438, 3254.73681640625, 4952.41552734375, 5326.0244140625, 526.942626953125, 1648.745849609375, 909.934326171875, 5123.18310546875, 544.89306640625, 1989.94580078125, 2894.975341796875, 882.3496704101562, 1366.3370361328125, 5814.40771484375, 4609.29443359375, 1778.91650390625, 3206.65234375, 3173.478515625, 2384.7392578125, 2551.469970703125, 1873.2410888671875, 2280.441162109375, 5570.318359375, 1127.95458984375, 1102.952880859375, 1218.06640625, 3091.325439453125, 3533.596923828125, 4571.0869140625, 4325.681640625, 3359.132568359375, 2328.38427734375, 3482.310791015625, 3859.31640625, 2864.16943359375, 2764.47021484375, 2569.002197265625, 3496.176025390625, 2171.365478515625, 1669.436767578125, 1367.2650146484375, 1036.4613037109375, 1014.6754760742188, 761.6212768554688, 559.731201171875, 522.8854370117188, 504.817626953125, 484.0948181152344, 479.3291931152344, 461.20196533203125, 412.5929870605469, 389.783935546875, 368.8175964355469, 348.30279541015625, 339.72711181640625, 337.238037109375, 335.3629455566406, 331.3726501464844, 326.7899169921875, 302.8639221191406, 297.98602294921875, 291.4053955078125, 279.0997314453125, 275.8830261230469, 275.0037841796875, 265.8647155761719, 260.5110778808594, 259.2149963378906, 257.05853271484375, 945.2482299804688, 881.5210571289062, 1879.7327880859375, 811.6838989257812, 1196.545166015625, 919.1948852539062, 346.0127258300781, 568.711669921875, 1607.7003173828125, 614.8113403320312, 950.2615966796875, 1659.3355712890625, 1183.75537109375, 1442.9866943359375, 752.5714721679688, 1654.0484619140625, 621.5165405273438, 2259.06591796875, 2500.019775390625, 1095.90283203125, 4538.4130859375, 3254.974853515625, 2033.72021484375, 1733.791259765625, 1109.912353515625, 899.2139892578125, 1759.74462890625, 728.7001953125, 1814.038330078125, 2391.947021484375, 2982.829833984375, 1736.9285888671875, 3263.285888671875, 4509.44873046875, 2782.203857421875, 6319.80517578125, 6607.29052734375, 1729.9007568359375, 3041.27587890625, 3310.97314453125, 2221.64501953125, 1837.8072509765625, 2612.655517578125, 1880.698486328125, 2071.16455078125, 2524.44580078125, 2345.779541015625, 1998.3720703125, 2005.0615234375, 2022.0311279296875, 1859.448974609375, 1219.4730224609375, 1173.406494140625, 1013.939208984375, 691.192626953125, 617.52490234375, 520.2979125976562, 509.9898681640625, 498.81365966796875, 496.1658020019531, 485.8641357421875, 469.23419189453125, 459.7816467285156, 445.23724365234375, 443.60833740234375, 418.2053527832031, 382.94537353515625, 410.7806091308594, 359.4635314941406, 346.97998046875, 305.3597412109375, 302.48150634765625, 296.2767639160156, 282.1846008300781, 278.46722412109375, 275.6992492675781, 266.8927917480469, 266.87384033203125, 264.0797424316406, 239.02015686035156, 238.4247589111328, 423.26446533203125, 791.5164794921875, 3831.427490234375, 768.3886108398438, 2197.80615234375, 8732.578125, 887.561767578125, 2807.25244140625, 1436.6512451171875, 2205.632568359375, 667.7637329101562, 1308.6068115234375, 594.5608520507812, 1610.975341796875, 1057.4656982421875, 1401.59912109375, 2104.89453125, 1648.9759521484375, 1492.6676025390625, 2233.339111328125, 2045.0888671875, 1025.418212890625, 1868.1944580078125, 1781.913818359375, 2283.537841796875, 2389.78369140625, 2343.92919921875, 11796.12890625, 1484.4742431640625, 1263.271728515625, 4137.8974609375, 3606.866943359375, 6391.9033203125, 3093.73876953125, 2431.4755859375, 2946.836669921875, 2351.900634765625, 3006.038330078125, 2427.706787109375, 2724.3447265625, 2284.53125, 1774.784912109375, 1923.3309326171875, 1701.534423828125, 1964.21142578125, 1967.0396728515625, 1731.4547119140625, 2209.434814453125, 1306.951171875, 881.4293823242188, 801.6221923828125, 800.6875, 770.8548583984375, 718.598876953125, 698.7440795898438, 511.1880187988281, 508.98004150390625, 478.0890808105469, 462.20867919921875, 450.9638671875, 447.72186279296875, 440.57586669921875, 439.1006164550781, 411.6143798828125, 347.85736083984375, 322.8476867675781, 317.38446044921875, 312.6617126464844, 303.76953125, 300.7120666503906, 291.9182434082031, 284.3955993652344, 281.8510437011719, 281.8087158203125, 281.4986572265625, 268.8218994140625, 264.3143615722656, 964.89892578125, 701.0892944335938, 1368.2698974609375, 843.9874267578125, 1627.477783203125, 768.4228515625, 1114.9678955078125, 580.59228515625, 1011.5383911132812, 1585.2037353515625, 675.5639038085938, 715.0435180664062, 4015.3271484375, 4388.03662109375, 1563.122314453125, 1602.8076171875, 4473.2919921875, 1736.8460693359375, 2465.334716796875, 1008.852783203125, 17047.599609375, 2495.47216796875, 2567.111572265625, 4753.0, 2976.826904296875, 1542.697509765625, 5214.8291015625, 2342.458251953125, 4023.214111328125, 1608.3721923828125, 6793.68798828125, 1724.8424072265625, 1880.1500244140625, 2035.59375, 3400.328369140625, 1766.6309814453125, 3316.8134765625, 1950.6055908203125, 1777.5321044921875, 1746.4385986328125, 1961.375, 2278.088623046875, 1954.1728515625, 1819.2515869140625, 1679.8890380859375, 1695.3040771484375, 1733.7081298828125, 1707.2315673828125, 1710.294921875, 6022.87451171875, 1383.4739990234375, 680.1478881835938, 669.3220825195312, 474.0889892578125, 451.43670654296875, 431.5091247558594, 412.7670593261719, 409.0692138671875, 351.6473388671875, 322.1426696777344, 293.1236572265625, 292.9236145019531, 292.6224670410156, 289.7826843261719, 281.7777404785156, 280.6816101074219, 276.15008544921875, 270.0996398925781, 267.1131896972656, 266.820068359375, 264.99420166015625, 259.8439636230469, 255.6787872314453, 252.6554718017578, 245.39125061035156, 210.08734130859375, 210.05435180664062, 208.32850646972656, 203.38368225097656, 1202.11767578125, 1979.875732421875, 889.0390014648438, 1231.3795166015625, 737.6416625976562, 650.1172485351562, 492.6054382324219, 2082.6416015625, 709.5663452148438, 593.7556762695312, 976.7166137695312, 366.5652770996094, 1949.446533203125, 834.9934692382812, 658.0715942382812, 3482.636474609375, 911.16552734375, 370.555419921875, 2186.193359375, 781.64306640625, 878.793701171875, 1330.533203125, 763.6713256835938, 1410.5960693359375, 1398.0936279296875, 1345.4415283203125, 1103.3248291015625, 882.4994506835938, 2766.4384765625, 1051.908203125, 1003.4359741210938, 873.0496826171875, 2378.681640625, 1615.811767578125, 3429.785400390625, 1890.927001953125, 1800.683837890625, 1320.8765869140625, 2154.98779296875, 1202.0650634765625, 1655.532470703125, 1161.7486572265625, 1077.357177734375, 1081.1661376953125, 1137.6011962890625, 1101.258056640625, 784.6056518554688, 639.3048706054688, 604.7518920898438, 527.7647705078125, 496.3140563964844, 488.11395263671875, 485.36834716796875, 478.1675109863281, 468.0965881347656, 442.6495361328125, 419.73065185546875, 397.0778503417969, 379.31964111328125, 359.674560546875, 358.87164306640625, 335.3717346191406, 327.6318359375, 313.6392822265625, 301.83441162109375, 293.1678466796875, 286.85986328125, 285.21905517578125, 266.6203308105469, 264.3385009765625, 260.721923828125, 257.9065246582031, 253.693603515625, 251.0128631591797, 235.68023681640625, 228.02940368652344, 600.0228271484375, 587.8187255859375, 670.0105590820312, 923.9404296875, 926.418212890625, 2348.46826171875, 427.27081298828125, 346.5133972167969, 373.1505126953125, 1060.75927734375, 847.658203125, 925.6361694335938, 674.403076171875, 1514.1085205078125, 1088.943603515625, 859.9300537109375, 599.871337890625, 823.968994140625, 1196.621826171875, 2524.485595703125, 632.6998901367188, 1396.9373779296875, 2904.23046875, 1428.566650390625, 2308.14599609375, 739.788818359375, 1732.622314453125, 2035.049560546875, 817.2945556640625, 1691.9761962890625, 2551.306640625, 1599.809326171875, 1844.89892578125, 2184.07373046875, 1176.778076171875, 2798.4736328125, 1032.1751708984375, 1989.2244873046875, 1483.45849609375, 1688.119873046875, 987.682373046875, 1107.9835205078125, 955.5052490234375, 871.8748168945312, 991.3363647460938, 986.2979736328125, 833.656005859375, 566.5687866210938, 547.5164794921875, 520.775146484375, 476.7263488769531, 471.81695556640625, 436.781005859375, 410.2228088378906, 368.67242431640625, 338.7643127441406, 336.9042053222656, 751.3635864257812, 312.6552429199219, 312.22735595703125, 294.3414001464844, 270.7018737792969, 270.0586853027344, 261.65911865234375, 260.2052307128906, 252.65353393554688, 250.2634735107422, 249.78024291992188, 239.81167602539062, 236.24026489257812, 229.65316772460938, 214.9892120361328, 210.27328491210938, 208.26881408691406, 244.7768096923828, 880.1952514648438, 854.8551025390625, 353.1084899902344, 1365.1163330078125, 376.6260986328125, 1188.4931640625, 1198.1365966796875, 797.7096557617188, 557.7677612304688, 521.394775390625, 326.90679931640625, 817.3114624023438, 329.1241149902344, 644.153076171875, 1204.0582275390625, 1694.3262939453125, 1073.200439453125, 793.1605834960938, 554.642578125, 480.6022033691406, 646.2938842773438, 433.18511962890625, 421.3907775878906, 412.2239990234375], \"Term\": [\"film\", \"bad\", \"good\", \"horror\", \"great\", \"show\", \"action\", \"really\", \"war\", \"like\", \"best\", \"role\", \"life\", \"funny\", \"character\", \"cast\", \"watch\", \"performance\", \"people\", \"think\", \"love\", \"comedy\", \"movies\", \"plot\", \"see\", \"get\", \"young\", \"woman\", \"family\", \"john\", \"redeeming\", \"waste\", \"amateurish\", \"uninteresting\", \"unwatchable\", \"drivel\", \"miserably\", \"1/10\", \"disjointed\", \"whatsoever\", \"pitiful\", \"appalling\", \"abysmal\", \"non-existent\", \"tripe\", \"lowest\", \"crowe\", \"cells\", \"lesbians\", \"paltrow\", \"self-indulgent\", \"unlikeable\", \"tempted\", \"nikita\", \"artsy\", \"steaming\", \"donnie\", \"embarrassingly\", \"marcus\", \"dreck\", \"poorly\", \"worst\", \"insulting\", \"pointless\", \"awful\", \"pathetic\", \"wasted\", \"terrible\", \"insult\", \"wasting\", \"worse\", \"remotely\", \"boring\", \"horrible\", \"garbage\", \"unfunny\", \"bad\", \"minutes\", \"rubbish\", \"excuse\", \"avoid\", \"poor\", \"mess\", \"supposed\", \"script\", \"acting\", \"nothing\", \"badly\", \"crap\", \"plot\", \"stupid\", \"could\", \"even\", \"annoying\", \"money\", \"film\", \"dialogue\", \"would\", \"make\", \"anything\", \"like\", \"thing\", \"better\", \"least\", \"really\", \"actors\", \"good\", \"ever\", \"made\", \"one\", \"actually\", \"say\", \"much\", \"time\", \"something\", \"get\", \"seen\", \"movies\", \"watch\", \"see\", \"people\", \"story\", \"characters\", \"think\", \"season\", \"disney\", \"sandler\", \"cartoons\", \"bugs\", \"bambi\", \"candyman\", \"batman\", \"movie.i\", \"porky\", \"carrey\", \"lol\", \"im\", \"looney\", \"zohan\", \"abc\", \"programme\", \"hangs\", \"lewton\", \"lohan\", \"buffy\", \"rodman\", \"contestants\", \"canceled\", \"aladdin\", \"previews\", \"taped\", \"p.s\", \"reruns\", \"ebay\", \"seasons\", \"animation\", \"episodes\", \"episode\", \"cartoon\", \"ninja\", \"show\", \"anime\", \"aired\", \"loved\", \"na\", \"tv\", \"liked\", \"series\", \"remember\", \"kids\", \"saw\", \"animated\", \"awesome\", \"enjoyed\", \"watched\", \"great\", \"dvd\", \"funny\", \"favorite\", \"watch\", \"movies\", \"see\", \"really\", \"think\", \"fan\", \"good\", \"like\", \"first\", \"fun\", \"seen\", \"thought\", \"one\", \"still\", \"would\", \"time\", \"love\", \"well\", \"years\", \"know\", \"best\", \"watching\", \"get\", \"lot\", \"made\", \"much\", \"story\", \"ever\", \"people\", \"also\", \"could\", \"even\", \"society\", \"religion\", \"cultural\", \"documentaries\", \"spiritual\", \"philosophy\", \"inspiring\", \"bunuel\", \"kurosawa\", \"aids\", \"beliefs\", \"fellini\", \"prostitution\", \"grief\", \"explores\", \"compassion\", \"addiction\", \"9/11\", \"loneliness\", \"muslim\", \"reflect\", \"bible\", \"o'toole\", \"akira\", \"vienna\", \"expressed\", \"accessible\", \"insights\", \"subtly\", \"studied\", \"documentary\", \"che\", \"thought-provoking\", \"insight\", \"emotional\", \"attitudes\", \"emotions\", \"social\", \"emotionally\", \"issues\", \"culture\", \"understanding\", \"humanity\", \"religious\", \"nature\", \"subject\", \"message\", \"relationships\", \"media\", \"human\", \"deeply\", \"powerful\", \"moral\", \"inner\", \"reality\", \"us\", \"complex\", \"world\", \"life\", \"themes\", \"personal\", \"viewer\", \"film\", \"experience\", \"lives\", \"people\", \"story\", \"view\", \"audience\", \"true\", \"characters\", \"art\", \"important\", \"many\", \"real\", \"different\", \"may\", \"feel\", \"character\", \"without\", \"yet\", \"way\", \"one\", \"also\", \"see\", \"love\", \"much\", \"well\", \"work\", \"would\", \"man\", \"like\", \"time\", \"films\", \"even\", \"never\", \"make\", \"married\", \"grant\", \"davis\", \"marry\", \"cary\", \"dunne\", \"marie\", \"bette\", \"irene\", \"nikki\", \"daughters\", \"pammy\", \"snoopy\", \"streep\", \"divorce\", \"angela\", \"loy\", \"tarzan\", \"o'hara\", \"gable\", \"sophie\", \"panther\", \"dakota\", \"meryl\", \"tribe\", \"marries\", \"madame\", \"noah\", \"colbert\", \"sellers\", \"husband\", \"waitress\", \"daughter\", \"mother\", \"father\", \"aunt\", \"son\", \"wife\", \"woman\", \"emily\", \"sister\", \"mom\", \"girl\", \"wealthy\", \"finds\", \"boy\", \"boyfriend\", \"meets\", \"young\", \"family\", \"parents\", \"home\", \"women\", \"town\", \"friend\", \"brother\", \"wants\", \"man\", \"decides\", \"mary\", \"baby\", \"gets\", \"old\", \"love\", \"life\", \"back\", \"goes\", \"two\", \"get\", \"new\", \"little\", \"go\", \"one\", \"comes\", \"alien\", \"zombie\", \"zombies\", \"aliens\", \"shark\", \"vampires\", \"godzilla\", \"seagal\", \"predator\", \"jaws\", \"werewolf\", \"scientists\", \"virus\", \"robots\", \"crow\", \"mst3k\", \"bullets\", \"dinosaurs\", \"helicopter\", \"puppet\", \"ants\", \"highlander\", \"sharks\", \"lab\", \"terminator\", \"mutant\", \"sinbad\", \"rubber\", \"troma\", \"dinosaur\", \"creature\", \"scientist\", \"monster\", \"monsters\", \"giant\", \"plane\", \"cave\", \"jungle\", \"gun\", \"robot\", \"planet\", \"space\", \"science\", \"sci-fi\", \"machine\", \"earth\", \"creatures\", \"car\", \"game\", \"fire\", \"action\", \"effects\", \"kill\", \"hero\", \"water\", \"computer\", \"fight\", \"guns\", \"evil\", \"special\", \"guy\", \"guys\", \"back\", \"get\", \"around\", \"like\", \"one\", \"head\", \"scene\", \"bad\", \"gets\", \"dead\", \"man\", \"looks\", \"look\", \"even\", \"people\", \"take\", \"go\", \"time\", \"two\", \"slasher\", \"serial\", \"murders\", \"supernatural\", \"murderer\", \"atmospheric\", \"scares\", \"freddy\", \"karloff\", \"chainsaw\", \"gruesome\", \"connery\", \"eerie\", \"miike\", \"spooky\", \"thrillers\", \"killings\", \"argento\", \"giallo\", \"sadako\", \"visitor\", \"columbo\", \"investigating\", \"phantasm\", \"hellraiser\", \"cronenberg\", \"emmanuelle\", \"carpenter\", \"orphanage\", \"liam\", \"13th\", \"gory\", \"killer\", \"massacre\", \"thriller\", \"horror\", \"dracula\", \"murder\", \"creepy\", \"gore\", \"killers\", \"detective\", \"haunted\", \"suspense\", \"ghost\", \"atmosphere\", \"police\", \"crime\", \"mystery\", \"dark\", \"genre\", \"mysterious\", \"violence\", \"blood\", \"house\", \"dead\", \"death\", \"film\", \"scary\", \"killing\", \"also\", \"films\", \"one\", \"plot\", \"quite\", \"well\", \"scenes\", \"good\", \"little\", \"story\", \"first\", \"ending\", \"director\", \"night\", \"much\", \"really\", \"bit\", \"musical\", \"allen\", \"holmes\", \"woody\", \"powell\", \"flynn\", \"chaplin\", \"kelly\", \"mgm\", \"lugosi\", \"brooks\", \"welles\", \"broadway\", \"errol\", \"hopkins\", \"lane\", \"watson\", \"francisco\", \"jazz\", \"lana\", \"hoffman\", \"nathan\", \"andre\", \"marx\", \"bela\", \"everett\", \"violin\", \"musicals\", \"orson\", \"olivia\", \"crawford\", \"warner\", \"dance\", \"douglas\", \"stage\", \"wayne\", \"robin\", \"arthur\", \"singing\", \"songs\", \"gene\", \"singer\", \"performance\", \"role\", \"supporting\", \"song\", \"cast\", \"roles\", \"performances\", \"silent\", \"film\", \"play\", \"screen\", \"best\", \"music\", \"score\", \"great\", \"john\", \"films\", \"fine\", \"one\", \"wonderful\", \"version\", \"actor\", \"also\", \"hollywood\", \"well\", \"plays\", \"star\", \"classic\", \"director\", \"good\", \"scene\", \"script\", \"excellent\", \"played\", \"character\", \"actors\", \"made\", \"war\", \"german\", \"united\", \"nazi\", \"indians\", \"wwii\", \"germany\", \"revolution\", \"civil\", \"nazis\", \"19th\", \"asoka\", \"spaghetti\", \"russia\", \"historically\", \"hitler\", \"sharpe\", \"soviet\", \"germans\", \"cassavetes\", \"communist\", \"troops\", \"aircraft\", \"enterprise\", \"fairbanks\", \"roman\", \"russians\", \"maris\", \"laughton\", \"leopard\", \"historical\", \"western\", \"russian\", \"century\", \"north\", \"europe\", \"westerns\", \"french\", \"u.s\", \"france\", \"soldiers\", \"union\", \"british\", \"president\", \"ford\", \"american\", \"military\", \"le\", \"history\", \"states\", \"army\", \"america\", \"americans\", \"novel\", \"country\", \"english\", \"battle\", \"government\", \"years\", \"king\", \"general\", \"political\", \"time\", \"world\", \"film\", \"first\", \"made\", \"series\", \"one\", \"later\", \"would\", \"set\", \"version\", \"last\", \"also\", \"two\", \"cagney\", \"bogart\", \"keaton\", \"fu\", \"mickey\", \"chaney\", \"kung\", \"quaid\", \"murray\", \"football\", \"jenny\", \"burt\", \"lenny\", \"pacino\", \"stanwyck\", \"daniels\", \"buscemi\", \"melanie\", \"niro\", \"nicholson\", \"daffy\", \"damme\", \"seth\", \"pryor\", \"boris\", \"buster\", \"morris\", \"timothy\", \"hogan\", \"bachchan\", \"wilson\", \"penn\", \"khan\", \"gangster\", \"chris\", \"jack\", \"parker\", \"sherlock\", \"robinson\", \"steve\", \"dennis\", \"tony\", \"al\", \"tom\", \"martin\", \"match\", \"matt\", \"moore\", \"paul\", \"john\", \"jr\", \"james\", \"role\", \"michael\", \"performance\", \"jane\", \"actor\", \"comedy\", \"smith\", \"plays\", \"character\", \"played\", \"cast\", \"best\", \"mr\", \"good\", \"robert\", \"great\", \"funny\", \"also\", \"job\", \"one\", \"big\", \"david\", \"arts\", \"martial\", \"murphy\", \"blah\", \"hong\", \"woo\", \"chan\", \"ha\", \"eastwood\", \"punk\", \"costello\", \"clint\", \"berry\", \"kong\", \"li\", \"stan\", \"ginger\", \"costner\", \"beowulf\", \"troy\", \"bands\", \"troll\", \"randolph\", \"norris\", \"meg\", \"abbott\", \"kidman\", \"elvis\", \"gremlins\", \"avery\", \"chow\", \"chuck\", \"eddie\", \"hudson\", \"rock\", \"hardy\", \"harry\", \"lee\", \"band\", \"metal\", \"jackie\", \"jet\", \"bruce\", \"beatles\", \"chinese\", \"school\", \"action\", \"high\", \"fight\", \"scott\", \"dirty\", \"black\", \"cop\", \"gang\", \"boys\"], \"Total\": [117884.0, 27038.0, 43300.0, 10177.0, 27211.0, 16980.0, 10312.0, 34704.0, 6023.0, 58839.0, 18912.0, 9371.0, 17378.0, 13026.0, 20484.0, 11272.0, 19807.0, 8258.0, 27723.0, 21069.0, 18407.0, 9363.0, 22479.0, 19103.0, 33745.0, 27452.0, 10575.0, 7637.0, 7980.0, 6739.0, 915.5548706054688, 4340.28662109375, 590.1669311523438, 544.178466796875, 404.0425109863281, 385.30523681640625, 371.8221740722656, 369.5846862792969, 335.7596740722656, 859.677734375, 290.10491943359375, 279.22906494140625, 276.927734375, 263.952880859375, 247.91424560546875, 236.49490356445312, 234.56149291992188, 195.00831604003906, 182.79470825195312, 178.77305603027344, 177.39645385742188, 175.94503784179688, 161.5632781982422, 161.45071411132812, 156.4660186767578, 155.14247131347656, 152.29739379882812, 141.3303985595703, 141.00558471679688, 138.8381805419922, 2029.6937255859375, 7692.67578125, 389.5094299316406, 1553.6932373046875, 4799.77197265625, 1512.1627197265625, 1755.04638671875, 4543.44677734375, 656.8763427734375, 442.364990234375, 4114.55078125, 598.653564453125, 5488.234375, 3741.634033203125, 1189.292236328125, 758.0113525390625, 27038.626953125, 8568.552734375, 768.4632568359375, 1157.19091796875, 2233.643798828125, 5572.9140625, 1840.0841064453125, 4326.349609375, 8953.0166015625, 17967.951171875, 12658.7158203125, 1656.7880859375, 2746.64404296875, 19103.853515625, 4952.27783203125, 28522.634765625, 36899.8125, 2828.101318359375, 6819.0205078125, 117884.6875, 4540.7470703125, 39911.4765625, 23752.083984375, 8106.234375, 58839.05078125, 13267.2333984375, 17224.4375, 9088.9853515625, 34704.04296875, 12773.43359375, 43300.515625, 17489.2890625, 23230.71875, 77621.6953125, 12446.234375, 15397.1826171875, 28435.765625, 35237.34375, 15196.0439453125, 27452.83984375, 19204.955078125, 22479.025390625, 19807.15625, 33745.6953125, 27723.748046875, 34402.09375, 21547.822265625, 21069.3125, 1735.8682861328125, 1531.836669921875, 781.433837890625, 770.5151977539062, 702.667236328125, 598.6738891601562, 414.0224914550781, 570.9938354492188, 340.6899108886719, 332.34112548828125, 290.42803955078125, 284.77130126953125, 261.6510925292969, 217.80767822265625, 214.99794006347656, 198.35763549804688, 193.79803466796875, 186.11289978027344, 185.1416015625, 180.5368194580078, 171.1826171875, 162.70993041992188, 161.4082489013672, 156.75135803222656, 154.69158935546875, 152.50160217285156, 151.72105407714844, 143.5069122314453, 142.80316162109375, 139.94949340820312, 498.5839538574219, 1867.9757080078125, 2010.017822265625, 4174.80224609375, 1517.5677490234375, 531.2496948242188, 16980.298828125, 918.5447387695312, 422.6588439941406, 3960.82421875, 1381.087646484375, 7277.3486328125, 4388.90380859375, 8492.04296875, 4683.66943359375, 4954.69580078125, 9667.9013671875, 1206.3955078125, 1474.2855224609375, 3710.537109375, 6342.52783203125, 27211.140625, 6791.54833984375, 13026.2861328125, 3481.511474609375, 19807.15625, 22479.025390625, 33745.6953125, 34704.04296875, 21069.3125, 5699.36279296875, 43300.515625, 58839.05078125, 25737.203125, 7793.57373046875, 19204.955078125, 9878.751953125, 77621.6953125, 15826.8408203125, 39911.4765625, 35237.34375, 18407.8515625, 28690.794921875, 13062.80078125, 18114.43359375, 18912.87109375, 13200.37109375, 27452.83984375, 12369.013671875, 23230.71875, 28435.765625, 34402.09375, 17489.2890625, 27723.748046875, 27075.703125, 28522.634765625, 36899.8125, 1964.0098876953125, 566.065673828125, 522.6664428710938, 418.5201110839844, 340.01348876953125, 330.6759948730469, 320.1274719238281, 287.8725891113281, 286.5057067871094, 281.2247009277344, 273.0070495605469, 272.17022705078125, 255.94967651367188, 244.72418212890625, 241.65817260742188, 241.34170532226562, 234.72901916503906, 233.26809692382812, 226.31854248046875, 223.00804138183594, 217.73635864257812, 217.55540466308594, 211.11273193359375, 207.56027221679688, 201.76771545410156, 196.1494598388672, 194.7924041748047, 194.50192260742188, 193.0054168701172, 188.11398315429688, 2912.8564453125, 298.367431640625, 239.3861541748047, 523.6522216796875, 1839.2791748046875, 310.7303161621094, 1122.2796630859375, 1721.7066650390625, 660.8409423828125, 1373.2584228515625, 1621.379638671875, 872.5790405273438, 785.60009765625, 775.6061401367188, 1669.5748291015625, 1845.9364013671875, 2262.386962890625, 1031.2874755859375, 888.46923828125, 4304.673828125, 841.451416015625, 2037.617431640625, 1029.63818359375, 599.208984375, 2731.240478515625, 10835.5341796875, 1366.9326171875, 11152.703125, 17378.2265625, 1017.8290405273438, 1942.98095703125, 3574.795166015625, 117884.6875, 3150.0439453125, 4175.7880859375, 27723.748046875, 34402.09375, 2614.53466796875, 6186.67578125, 6291.04638671875, 21547.822265625, 2994.517333984375, 2856.409423828125, 19888.14453125, 14074.2802734375, 7475.53076171875, 9902.69921875, 8681.345703125, 20484.87109375, 9624.1220703125, 8511.6103515625, 23422.7578125, 77621.6953125, 27075.703125, 33745.6953125, 18407.8515625, 28435.765625, 28690.794921875, 12124.283203125, 39911.4765625, 16213.833984375, 58839.05078125, 35237.34375, 20824.89453125, 36899.8125, 19269.353515625, 23752.083984375, 1759.5048828125, 1499.3536376953125, 1084.6824951171875, 737.4893188476562, 678.1207275390625, 649.7626953125, 559.4512329101562, 555.876708984375, 459.2032165527344, 435.5541076660156, 401.4744567871094, 347.26971435546875, 333.49853515625, 333.1943054199219, 309.8009338378906, 307.92816162109375, 299.51800537109375, 288.08258056640625, 277.89251708984375, 256.7599182128906, 250.9366455078125, 238.6881561279297, 235.73654174804688, 228.28753662109375, 227.8879852294922, 226.40989685058594, 214.94064331054688, 213.4000244140625, 213.1038818359375, 210.56845092773438, 3080.736328125, 283.1495666503906, 3614.259765625, 4817.80615234375, 5384.60888671875, 595.2386474609375, 3883.768310546875, 6226.49560546875, 7637.9619140625, 578.5703735351562, 2150.174560546875, 1090.654541015625, 8203.47265625, 606.3887939453125, 2814.869384765625, 4418.1806640625, 1077.51416015625, 1824.168701171875, 10575.78515625, 7980.951171875, 2537.651611328125, 5377.22021484375, 5316.07861328125, 3830.09814453125, 4333.27197265625, 2896.245849609375, 3783.030517578125, 16213.833984375, 1556.39404296875, 1543.1695556640625, 1828.838134765625, 9212.478515625, 11659.1669921875, 18407.8515625, 17378.2265625, 14178.095703125, 7300.298828125, 20094.861328125, 27452.83984375, 12530.369140625, 19265.041015625, 14586.78515625, 77621.6953125, 7333.7373046875, 1670.33251953125, 1368.16064453125, 1037.35693359375, 1015.5711669921875, 762.5169677734375, 560.6270751953125, 523.7810668945312, 505.7134094238281, 484.9905700683594, 480.2249450683594, 462.09771728515625, 413.4887390136719, 390.6796875, 369.71337890625, 349.1986389160156, 340.6228942871094, 338.1338195800781, 336.2586975097656, 332.2684326171875, 327.6857604980469, 303.7596740722656, 298.8818054199219, 292.3011474609375, 279.9954833984375, 276.77880859375, 275.8995361328125, 266.76055908203125, 261.4068298339844, 260.11077880859375, 257.9542541503906, 956.9449462890625, 923.2066650390625, 2075.118896484375, 893.66748046875, 1379.1259765625, 1037.2833251953125, 354.5676574707031, 612.5215454101562, 1946.5540771484375, 673.1814575195312, 1106.4361572265625, 2095.483642578125, 1437.4608154296875, 1834.42333984375, 866.4176025390625, 2223.820068359375, 703.1215209960938, 3401.036865234375, 3937.23291015625, 1507.1507568359375, 10312.3564453125, 6641.234375, 3644.205078125, 3094.11767578125, 1617.730712890625, 1193.28173828125, 3338.86376953125, 884.8314819335938, 3860.751953125, 6096.9697265625, 9184.4716796875, 4171.2353515625, 14178.095703125, 27452.83984375, 10651.6845703125, 58839.05078125, 77621.6953125, 4246.62060546875, 16287.9169921875, 27038.626953125, 9212.478515625, 5586.34423828125, 16213.833984375, 6783.1845703125, 12274.90625, 36899.8125, 27723.748046875, 10211.3857421875, 14586.78515625, 35237.34375, 20094.861328125, 1220.3680419921875, 1174.3016357421875, 1014.8343505859375, 692.0877075195312, 618.4201049804688, 521.20458984375, 510.885009765625, 499.7088317871094, 497.0609130859375, 486.7592468261719, 470.12933349609375, 460.67694091796875, 446.13238525390625, 444.50341796875, 419.10052490234375, 383.84051513671875, 411.7791442871094, 360.358642578125, 347.8750305175781, 306.25482177734375, 303.37677001953125, 297.17181396484375, 283.079833984375, 279.3623046875, 276.59442138671875, 267.7878723144531, 267.76904296875, 264.9748840332031, 239.9154815673828, 239.31996154785156, 426.5197448730469, 813.2481689453125, 4122.2109375, 793.6625366210938, 2372.974609375, 10177.3076171875, 939.3807373046875, 3167.3671875, 1586.9150390625, 2633.7578125, 717.7451171875, 1504.1766357421875, 648.44580078125, 2099.745361328125, 1317.6361083984375, 1885.194091796875, 3340.410400390625, 2454.901123046875, 2226.85205078125, 3941.12060546875, 3698.684326171875, 1403.4697265625, 3388.6123046875, 3256.5263671875, 5091.716796875, 5586.34423828125, 5925.10888671875, 117884.6875, 2966.061279296875, 2247.91259765625, 27075.703125, 20824.89453125, 77621.6953125, 19103.853515625, 11061.75, 28690.794921875, 15218.09375, 43300.515625, 19265.041015625, 34402.09375, 25737.203125, 7193.6328125, 12043.2001953125, 6343.6826171875, 28435.765625, 34704.04296875, 9082.89453125, 2210.325927734375, 1307.84228515625, 882.3204345703125, 802.5132446289062, 801.57861328125, 771.745849609375, 719.4898681640625, 699.6351318359375, 512.0790405273438, 509.8710632324219, 478.9802551269531, 463.0997009277344, 451.8548889160156, 448.6129150390625, 441.4669494628906, 439.9916687011719, 412.50543212890625, 348.7484436035156, 323.73883056640625, 318.27557373046875, 313.55279541015625, 304.66082763671875, 301.6031188964844, 292.8094787597656, 285.2866516113281, 282.74224853515625, 282.6998596191406, 282.3896484375, 269.71295166015625, 265.2054443359375, 971.0306396484375, 730.804443359375, 1619.75732421875, 948.7169189453125, 2027.815673828125, 875.7819213867188, 1373.3675537109375, 650.6238403320312, 1258.73974609375, 2246.77001953125, 802.1104125976562, 885.54443359375, 8258.1162109375, 9371.5205078125, 2525.40087890625, 2654.041015625, 11272.1796875, 3256.699951171875, 5459.37744140625, 1506.4013671875, 117884.6875, 6523.0966796875, 6857.33837890625, 18912.87109375, 8968.998046875, 3150.839111328125, 27211.140625, 6739.421875, 20824.89453125, 3753.30029296875, 77621.6953125, 4436.78369140625, 5356.7216796875, 6410.86083984375, 27075.703125, 5195.97509765625, 28690.794921875, 6827.7197265625, 5427.9423828125, 5151.236328125, 12043.2001953125, 43300.515625, 16287.9169921875, 8953.0166015625, 5924.431640625, 7857.953125, 20484.87109375, 12773.43359375, 23230.71875, 6023.76806640625, 1384.3675537109375, 681.04541015625, 670.215576171875, 474.98260498046875, 452.3302307128906, 432.4026794433594, 413.6605529785156, 409.96270751953125, 352.54083251953125, 323.0363464355469, 294.0172119140625, 293.8171691894531, 293.5159912109375, 290.6762390136719, 282.6712646484375, 281.5750732421875, 277.0435485839844, 270.99310302734375, 268.00701904296875, 267.71356201171875, 265.88775634765625, 260.7375183105469, 256.5724182128906, 253.54898071289062, 246.2848358154297, 210.98086547851562, 210.9479217529297, 209.22210693359375, 204.27728271484375, 1240.5771484375, 2168.93701171875, 972.8571166992188, 1399.2041015625, 811.8048095703125, 717.5852661132812, 537.6838989257812, 2799.774169921875, 817.1680908203125, 671.738525390625, 1201.4786376953125, 386.2450866699219, 2732.6455078125, 1043.8056640625, 801.5125122070312, 6530.90283203125, 1241.9560546875, 396.9862365722656, 3875.437744140625, 1063.1651611328125, 1246.1590576171875, 2296.528076171875, 1078.897705078125, 2664.387939453125, 2672.4716796875, 2550.376953125, 1937.6470947265625, 1362.2342529296875, 13062.80078125, 2476.925048828125, 2449.69921875, 1689.8155517578125, 35237.34375, 11152.703125, 117884.6875, 25737.203125, 23230.71875, 8492.04296875, 77621.6953125, 6299.1396484375, 39911.4765625, 6581.13720703125, 5356.7216796875, 8846.138671875, 27075.703125, 20094.861328125, 785.4987182617188, 640.197998046875, 605.6449584960938, 528.657958984375, 497.2071838378906, 489.0070495605469, 486.2615051269531, 479.0606689453125, 469.0357971191406, 443.542724609375, 420.6238098144531, 397.97100830078125, 380.2127380371094, 360.567626953125, 359.7647705078125, 336.2648620605469, 328.52496337890625, 314.5326232910156, 302.72747802734375, 294.06097412109375, 287.7530212402344, 286.1121520996094, 267.5134582519531, 265.2315673828125, 261.6153259277344, 258.79962158203125, 254.5867919921875, 251.906005859375, 236.57334899902344, 228.9225311279297, 602.61083984375, 609.52001953125, 724.0986938476562, 1038.6168212890625, 1048.624267578125, 2920.282470703125, 453.6442565917969, 358.940673828125, 390.37469482421875, 1279.841796875, 1024.867919921875, 1158.623779296875, 818.2228393554688, 2338.79541015625, 1545.930908203125, 1260.70703125, 760.4579467773438, 1191.158935546875, 2065.583984375, 6739.421875, 828.3880615234375, 2815.88134765625, 9371.5205078125, 3253.225341796875, 8258.1162109375, 1141.42724609375, 6410.86083984375, 9363.7734375, 1429.3380126953125, 6827.7197265625, 20484.87109375, 7857.953125, 11272.1796875, 18912.87109375, 3898.10498046875, 43300.515625, 3173.08984375, 27211.140625, 13026.2861328125, 27075.703125, 6335.88330078125, 77621.6953125, 9951.0537109375, 3053.97119140625, 992.2300415039062, 987.1915893554688, 834.5496826171875, 567.4624633789062, 548.4100952148438, 521.6688842773438, 477.6199645996094, 472.7106018066406, 437.6746826171875, 411.1165466308594, 369.56610107421875, 339.6579895019531, 337.7979736328125, 753.4573364257812, 313.548828125, 313.12109375, 295.23504638671875, 271.5956726074219, 270.9523620605469, 262.5528259277344, 261.0989074707031, 253.5472412109375, 251.15725708007812, 250.6738739013672, 240.70538330078125, 237.13389587402344, 230.54696655273438, 215.8828887939453, 211.16693115234375, 209.16262817382812, 246.0686492919922, 964.3429565429688, 986.1279296875, 379.1215515136719, 1836.089599609375, 415.5182189941406, 1800.1275634765625, 2190.59326171875, 1247.50927734375, 768.608154296875, 726.6689453125, 366.2481994628906, 1559.5760498046875, 370.8909912109375, 1101.3719482421875, 4466.029296875, 10312.3564453125, 5516.4697265625, 3338.86376953125, 1533.9080810546875, 1001.22119140625, 6047.8056640625, 2167.926513671875, 1709.44287109375, 1932.3671875], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 1.5233999490737915, 1.523300051689148, 1.5228999853134155, 1.5226999521255493, 1.5221999883651733, 1.5220999717712402, 1.5219999551773071, 1.5219000577926636, 1.5217000246047974, 1.5216000080108643, 1.521299958229065, 1.5211999416351318, 1.5211000442504883, 1.5210000276565552, 1.520799994468689, 1.5205999612808228, 1.5205999612808228, 1.5197999477386475, 1.5195000171661377, 1.5194000005722046, 1.5192999839782715, 1.5192999839782715, 1.5188000202178955, 1.5188000202178955, 1.5187000036239624, 1.5185999870300293, 1.5184999704360962, 1.5180000066757202, 1.5180000066757202, 1.517899990081787, 1.5157999992370605, 1.4916000366210938, 1.5127999782562256, 1.4967999458312988, 1.4852999448776245, 1.4940999746322632, 1.4895999431610107, 1.4656000137329102, 1.4958000183105469, 1.5032999515533447, 1.4515000581741333, 1.4925999641418457, 1.4248000383377075, 1.4226000308990479, 1.4637999534606934, 1.4778000116348267, 1.2906999588012695, 1.3046000003814697, 1.455899953842163, 1.4271999597549438, 1.3730000257492065, 1.2980999946594238, 1.3848999738693237, 1.3043999671936035, 1.2366000413894653, 1.1653000116348267, 1.180899977684021, 1.381500005722046, 1.3282999992370605, 1.0645999908447266, 1.2342000007629395, 0.9771999716758728, 0.9083999991416931, 1.2905999422073364, 1.1291999816894531, 0.5286999940872192, 1.1863000392913818, 0.7073000073432922, 0.8133000135421753, 1.044800043106079, 0.5576000213623047, 0.9045000076293945, 0.8241000175476074, 1.0016000270843506, 0.617900013923645, 0.9053000211715698, 0.5292999744415283, 0.7759000062942505, 0.6570000052452087, 0.1597999930381775, 0.8779000043869019, 0.7810999751091003, 0.5171999931335449, 0.37459999322891235, 0.7785000205039978, 0.4147999882698059, 0.6122999787330627, 0.5116000175476074, 0.5827999711036682, 0.18459999561309814, 0.2337999939918518, 0.0364999994635582, 0.451200008392334, 0.4496999979019165, 1.7747000455856323, 1.7746000289916992, 1.7740999460220337, 1.7740999460220337, 1.7740000486373901, 1.7736999988555908, 1.7731000185012817, 1.7726999521255493, 1.7726000547409058, 1.7725000381469727, 1.7720999717712402, 1.7720999717712402, 1.7718000411987305, 1.7711000442504883, 1.7711000442504883, 1.7706999778747559, 1.7705999612808228, 1.770400047302246, 1.770400047302246, 1.770300030708313, 1.7699999809265137, 1.769700050354004, 1.769700050354004, 1.7695000171661377, 1.7694000005722046, 1.7694000005722046, 1.7692999839782715, 1.7690000534057617, 1.7690000534057617, 1.7688000202178955, 1.7568000555038452, 1.7431000471115112, 1.7315000295639038, 1.7115000486373901, 1.7201000452041626, 1.7354999780654907, 1.6283999681472778, 1.7077000141143799, 1.7302000522613525, 1.6169999837875366, 1.6550999879837036, 1.5182000398635864, 1.5520000457763672, 1.4881999492645264, 1.5341999530792236, 1.5256999731063843, 1.4479999542236328, 1.6318999528884888, 1.5989999771118164, 1.4816999435424805, 1.4113999605178833, 1.18149995803833, 1.3696999549865723, 1.2592999935150146, 1.447100043296814, 1.1619000434875488, 1.1304999589920044, 1.0259000062942505, 1.003100037574768, 1.0741000175476074, 1.3315999507904053, 0.875, 0.7745000123977661, 0.9424999952316284, 1.2199000120162964, 0.9685999751091003, 1.1406999826431274, 0.4747999906539917, 0.964900016784668, 0.6313999891281128, 0.65829998254776, 0.8932999968528748, 0.7157999873161316, 1.0099999904632568, 0.8654999732971191, 0.795799970626831, 0.9837999939918518, 0.574999988079071, 1.0046000480651855, 0.6089000105857849, 0.4740000069141388, 0.3321000039577484, 0.7617999911308289, 0.4300000071525574, 0.43959999084472656, 0.3677000105381012, 0.12610000371932983, 1.8169000148773193, 1.8157999515533447, 1.8157000541687012, 1.8152999877929688, 1.8147000074386597, 1.8147000074386597, 1.8145999908447266, 1.8142999410629272, 1.8142999410629272, 1.8142000436782837, 1.8141000270843506, 1.8141000270843506, 1.8138999938964844, 1.8136999607086182, 1.8136999607086182, 1.8136999607086182, 1.813599944114685, 1.813599944114685, 1.8134000301361084, 1.8134000301361084, 1.8133000135421753, 1.8133000135421753, 1.813099980354309, 1.813099980354309, 1.812999963760376, 1.8128000497817993, 1.8128000497817993, 1.8128000497817993, 1.8128000497817993, 1.812600016593933, 1.7970999479293823, 1.8106000423431396, 1.8113000392913818, 1.802899956703186, 1.7745000123977661, 1.8051999807357788, 1.7682000398635864, 1.7359999418258667, 1.7711999416351318, 1.7388999462127686, 1.7066999673843384, 1.7425999641418457, 1.73989999294281, 1.7316999435424805, 1.6614999771118164, 1.6510000228881836, 1.6319999694824219, 1.6956000328063965, 1.6992000341415405, 1.5311000347137451, 1.6995999813079834, 1.5744999647140503, 1.6523000001907349, 1.7208000421524048, 1.5219000577926636, 1.3164000511169434, 1.6030999422073364, 1.2838000059127808, 1.2102999687194824, 1.6302000284194946, 1.5118000507354736, 1.3752000331878662, 0.6395999789237976, 1.3977999687194824, 1.3316999673843384, 0.8489000201225281, 0.7727000117301941, 1.4249999523162842, 1.194599986076355, 1.1902999877929688, 0.843500018119812, 1.3739999532699585, 1.3704999685287476, 0.7294999957084656, 0.8001999855041504, 1.0015000104904175, 0.8901000022888184, 0.9377999901771545, 0.5619999766349792, 0.8830999732017517, 0.9315999746322632, 0.4634999930858612, -0.22669999301433563, 0.23849999904632568, 0.10999999940395355, 0.4244000017642975, 0.12380000203847885, 0.1103999987244606, 0.5943999886512756, -0.16189999878406525, 0.3901999890804291, -0.489300012588501, -0.32010000944137573, 0.06210000067949295, -0.4830999970436096, 0.12409999966621399, -0.10170000046491623, 2.4277000427246094, 2.4275999069213867, 2.4274001121520996, 2.427000045776367, 2.4268999099731445, 2.426800012588501, 2.4265999794006348, 2.4265999794006348, 2.426300048828125, 2.4261999130249023, 2.4260001182556152, 2.425600051879883, 2.42549991607666, 2.42549991607666, 2.425299882888794, 2.425299882888794, 2.4251999855041504, 2.425100088119507, 2.424999952316284, 2.4247000217437744, 2.4245998859405518, 2.424499988555908, 2.4244000911712646, 2.424299955368042, 2.424299955368042, 2.424299955368042, 2.4240000247955322, 2.4240000247955322, 2.4240000247955322, 2.4240000247955322, 2.414599895477295, 2.4214999675750732, 2.3610000610351562, 2.311300039291382, 2.2816998958587646, 2.3822999000549316, 2.251499891281128, 2.1993000507354736, 2.067699909210205, 2.334700107574463, 2.1626999378204346, 2.2471001148223877, 1.9573999643325806, 2.3213000297546387, 2.081399917602539, 2.005500078201294, 2.2283999919891357, 2.13919997215271, 1.8300000429153442, 1.8791999816894531, 2.072999954223633, 1.9112999439239502, 1.9122999906539917, 1.9543999433517456, 1.8985999822616577, 1.9924999475479126, 1.9220999479293823, 1.3597999811172485, 2.1061999797821045, 2.092400074005127, 2.0218000411987305, 1.336300015449524, 1.2344000339508057, 1.0351999998092651, 1.037600040435791, 0.9882000088691711, 1.2855000495910645, 0.6754000186920166, 0.46619999408721924, 0.9523000121116638, 0.4867999851703644, 0.6916000247001648, -0.671999990940094, 1.2110999822616577, 2.439199924468994, 2.4391000270843506, 2.4388999938964844, 2.438800096511841, 2.4386000633239746, 2.4381000995635986, 2.437999963760376, 2.437999963760376, 2.4379000663757324, 2.4379000663757324, 2.4377999305725098, 2.4375998973846436, 2.4374001026153564, 2.437299966812134, 2.4372000694274902, 2.4370999336242676, 2.4370999336242676, 2.4370999336242676, 2.437000036239624, 2.437000036239624, 2.436800003051758, 2.4367001056671143, 2.4367001056671143, 2.436500072479248, 2.436500072479248, 2.436500072479248, 2.4363999366760254, 2.436300039291382, 2.436300039291382, 2.436199903488159, 2.4274001121520996, 2.3935000896453857, 2.3408000469207764, 2.3434998989105225, 2.2976999282836914, 2.3189001083374023, 2.415299892425537, 2.365499973297119, 2.248500108718872, 2.3489999771118164, 2.287600040435791, 2.206399917602539, 2.245500087738037, 2.199700117111206, 2.2988998889923096, 2.143699884414673, 2.3164000511169434, 2.030600070953369, 1.9854999780654907, 2.1210999488830566, 1.61899995803833, 1.7266000509262085, 1.8565000295639038, 1.8604999780654907, 2.062999963760376, 2.1568000316619873, 1.799299955368042, 2.2455999851226807, 1.684399962425232, 1.503999948501587, 1.3150999546051025, 1.563599944114685, 0.97079998254776, 0.633400022983551, 1.0973000526428223, 0.2085999995470047, -0.023900000378489494, 1.541700005531311, 0.7616000175476074, 0.33970001339912415, 1.0174000263214111, 1.3279999494552612, 0.6141999959945679, 1.1569000482559204, 0.6603000164031982, -0.24250000715255737, -0.029899999499320984, 0.8086000084877014, 0.4553000032901764, -0.41830000281333923, 0.05950000137090683, 2.5153000354766846, 2.5153000354766846, 2.5151000022888184, 2.514699935913086, 2.5146000385284424, 2.5143001079559326, 2.5143001079559326, 2.51419997215271, 2.51419997215271, 2.51419997215271, 2.5141000747680664, 2.5141000747680664, 2.5139999389648438, 2.5139999389648438, 2.5139000415802, 2.513700008392334, 2.5136001110076904, 2.5134999752044678, 2.5134999752044678, 2.5130999088287354, 2.5130999088287354, 2.513000011444092, 2.5129001140594482, 2.5127999782562256, 2.5127999782562256, 2.512700080871582, 2.512700080871582, 2.5125999450683594, 2.5123000144958496, 2.5123000144958496, 2.5083999633789062, 2.4888999462127686, 2.4428999423980713, 2.4837000370025635, 2.439300060272217, 2.3629000186920166, 2.4593000411987305, 2.3952999114990234, 2.4165000915527344, 2.338599920272827, 2.4437999725341797, 2.376699924468994, 2.42930006980896, 2.2511000633239746, 2.296099901199341, 2.219599962234497, 2.0541999340057373, 2.1180999279022217, 2.115999937057495, 1.948099970817566, 1.9234999418258667, 2.202199935913086, 1.9206000566482544, 1.913100004196167, 1.7141000032424927, 1.6669000387191772, 1.5887000560760498, 0.21410000324249268, 1.8238999843597412, 1.9397000074386597, 0.6376000046730042, 0.7627000212669373, 0.019200000911951065, 0.6955000162124634, 1.0010000467300415, 0.2401999980211258, 0.6488000154495239, -0.15150000154972076, 0.4447000026702881, -0.019899999722838402, 0.094200000166893, 1.1165000200271606, 0.6815999746322632, 1.2000999450683594, -0.15649999678134918, -0.35429999232292175, 0.8586000204086304, 2.560499906539917, 2.56030011177063, 2.5599000453948975, 2.559799909591675, 2.559799909591675, 2.559799909591675, 2.5597000122070312, 2.5597000122070312, 2.5592000484466553, 2.5592000484466553, 2.5590999126434326, 2.559000015258789, 2.559000015258789, 2.559000015258789, 2.5589001178741455, 2.5589001178741455, 2.558799982070923, 2.5583999156951904, 2.558199882507324, 2.5580999851226807, 2.5580999851226807, 2.558000087738037, 2.558000087738037, 2.5578999519348145, 2.557800054550171, 2.557800054550171, 2.557800054550171, 2.557800054550171, 2.5576000213623047, 2.5576000213623047, 2.5546000003814697, 2.519399881362915, 2.392199993133545, 2.444000005722046, 2.3410000801086426, 2.4302000999450684, 2.3524999618530273, 2.4470999240875244, 2.3422999382019043, 2.2121999263763428, 2.38919997215271, 2.347100019454956, 1.839900016784668, 1.8021999597549438, 2.081199884414673, 2.0566000938415527, 1.6367000341415405, 1.9322999715805054, 1.7659000158309937, 2.1600000858306885, 0.6272000074386597, 1.600100040435791, 1.5784000158309937, 1.179900050163269, 1.4579999446868896, 1.8467999696731567, 0.9088000059127808, 1.5041999816894531, 0.9168999791145325, 1.7135000228881836, 0.1251000016927719, 1.6160999536514282, 1.5139000415802002, 1.413699984550476, 0.4862000048160553, 1.482100009918213, 0.4034000039100647, 1.3080999851226807, 1.444599986076355, 1.4793000221252441, 0.7461000084877014, -0.383899986743927, 0.4404999911785126, 0.9674000144004822, 1.3006000518798828, 1.0273000001907349, 0.09149999916553497, 0.5483999848365784, -0.0478999987244606, 2.8324999809265137, 2.8320000171661377, 2.8313000202178955, 2.8313000202178955, 2.830699920654297, 2.8306000232696533, 2.8304998874664307, 2.8304998874664307, 2.830399990081787, 2.8301000595092773, 2.8297998905181885, 2.8296000957489014, 2.8296000957489014, 2.8296000957489014, 2.8294999599456787, 2.8294999599456787, 2.829400062561035, 2.829400062561035, 2.8292999267578125, 2.8292999267578125, 2.8292999267578125, 2.829200029373169, 2.829200029373169, 2.8290998935699463, 2.8290998935699463, 2.8289999961853027, 2.828399896621704, 2.828399896621704, 2.8282999992370605, 2.828200101852417, 2.801100015640259, 2.7414000034332275, 2.742500066757202, 2.7047998905181885, 2.736799955368042, 2.7339000701904297, 2.7451000213623047, 2.5367000102996826, 2.6914000511169434, 2.709199905395508, 2.625499963760376, 2.7802999019622803, 2.4948999881744385, 2.6094000339508057, 2.6354000568389893, 2.203900098800659, 2.522900104522705, 2.763700008392334, 2.2600998878479004, 2.5250000953674316, 2.483299970626831, 2.286799907684326, 2.4870998859405518, 2.196700096130371, 2.1847000122070312, 2.1930999755859375, 2.2695000171661377, 2.3984999656677246, 1.280400037765503, 1.976199984550476, 1.9400999546051025, 2.1721999645233154, 0.1370999962091446, 0.9007999897003174, -0.7045999765396118, 0.22169999778270721, 0.275299996137619, 0.9718000292778015, -0.7513999938964844, 1.176200032234192, -0.3499000072479248, 1.0982999801635742, 1.2288000583648682, 0.7307000160217285, -0.33709999918937683, -0.0714000016450882, 3.2904000282287598, 3.2901999950408936, 3.29010009765625, 3.289900064468384, 3.289799928665161, 3.289799928665161, 3.2897000312805176, 3.2897000312805176, 3.289599895477295, 3.289599895477295, 3.2894999980926514, 3.289299964904785, 3.2892000675201416, 3.289099931716919, 3.289099931716919, 3.2888998985290527, 3.2888998985290527, 3.2887001037597656, 3.288599967956543, 3.2885000705718994, 3.2885000705718994, 3.2885000705718994, 3.2881999015808105, 3.2881999015808105, 3.2881999015808105, 3.288100004196167, 3.288100004196167, 3.2880001068115234, 3.2878000736236572, 3.2876999378204346, 3.2873001098632812, 3.2553000450134277, 3.213900089263916, 3.1745998859405518, 3.1677000522613525, 3.073699951171875, 3.2316999435424805, 3.2562999725341797, 3.246500015258789, 3.103800058364868, 3.1017000675201416, 3.0671000480651855, 3.098299980163574, 2.856800079345703, 2.941200017929077, 2.9089999198913574, 3.0543999671936035, 2.9230000972747803, 2.745699882507324, 2.3096001148223877, 3.0220999717712402, 2.59060001373291, 2.1201000213623047, 2.468600034713745, 2.0167999267578125, 2.8578999042510986, 1.983199954032898, 1.7653000354766846, 2.732599973678589, 1.8964999914169312, 1.2085000276565552, 1.6999000310897827, 1.4816999435424805, 1.1328999996185303, 2.093899965286255, 0.5525000095367432, 2.1684999465942383, 0.6757000088691711, 1.11899995803833, 0.5166000127792358, 1.4329999685287476, -0.9577000141143799, 0.9484000205993652, 2.0380001068115234, 3.8668999671936035, 3.8668999671936035, 3.86680006980896, 3.866300106048584, 3.8661999702453613, 3.8661000728607178, 3.865999937057495, 3.8659000396728516, 3.865799903869629, 3.8657000064849854, 3.8654000759124756, 3.8652000427246094, 3.8652000427246094, 3.865000009536743, 3.865000009536743, 3.865000009536743, 3.864799976348877, 3.864500045776367, 3.864500045776367, 3.8643999099731445, 3.8643999099731445, 3.864300012588501, 3.864300012588501, 3.864300012588501, 3.8640999794006348, 3.8640999794006348, 3.8638999462127686, 3.8636999130249023, 3.863600015640259, 3.8635001182556152, 3.862600088119507, 3.7764999866485596, 3.7249999046325684, 3.796799898147583, 3.5713999271392822, 3.7695999145507812, 3.452699899673462, 3.264400005340576, 3.4207000732421875, 3.5471999645233154, 3.535900115966797, 3.754199981689453, 3.2216999530792236, 3.7483999729156494, 3.3315000534057617, 2.556999921798706, 2.061800003051758, 2.2307000160217285, 2.430500030517578, 2.850600004196167, 3.1338999271392822, 1.631600022315979, 2.257499933242798, 2.4674999713897705, 2.3229000568389893], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -7.617800235748291, -6.061699867248535, -8.057499885559082, -8.138799667358398, -8.437100410461426, -8.484700202941895, -8.520400047302246, -8.52649974822998, -8.622699737548828, -7.682600021362305, -8.769200325012207, -8.807600021362305, -8.815899848937988, -8.86400032043457, -8.926899909973145, -8.974300384521484, -8.982500076293945, -9.167900085449219, -9.232999801635742, -9.25529956817627, -9.263099670410156, -9.271300315856934, -9.357000350952148, -9.35770034790039, -9.389300346374512, -9.39780044555664, -9.416399955749512, -9.491600036621094, -9.494000434875488, -9.509499549865723, -6.8292999267578125, -5.521200180053711, -8.483099937438965, -7.115499973297119, -5.999199867248535, -7.145299911499023, -7.000899791717529, -6.073699951171875, -7.977399826049805, -8.365300178527832, -6.186999797821045, -8.07349967956543, -5.925600051879883, -6.3109002113342285, -7.415900230407715, -7.85230016708374, -4.465099811553955, -5.600399971008301, -7.860499858856201, -7.479899883270264, -6.876399993896484, -6.0370001792907715, -7.058300018310547, -6.283899784088135, -5.624499797821045, -4.999199867248535, -5.333700180053711, -7.1666998863220215, -6.714399814605713, -5.03849983215332, -6.218999862670898, -4.725100040435791, -4.536399841308594, -6.722799777984619, -6.0040998458862305, -3.7546000480651855, -6.353700160980225, -4.658999919891357, -5.0721001625061035, -5.915599822998047, -4.420599937438965, -5.563199996948242, -5.382599830627441, -5.844399929046631, -4.888199806213379, -5.600399971008301, -4.7555999755859375, -5.415599822998047, -5.2505998611450195, -4.541399955749512, -5.65369987487793, -5.537799835205078, -5.1880998611450195, -5.116300106048584, -5.553500175476074, -5.325799942016602, -5.485499858856201, -5.428899765014648, -5.484099864959717, -5.3495001792907715, -5.4969000816345215, -5.478400230407715, -5.531499862670898, -5.555500030517578, -6.726799964904785, -6.851900100708008, -7.525599956512451, -7.539599895477295, -7.631899833679199, -7.792300224304199, -8.161800384521484, -7.840700149536133, -8.357199668884277, -8.382100105285645, -8.51729965209961, -8.536999702453613, -8.621999740600586, -8.805999755859375, -8.819100379943848, -8.899999618530273, -8.923299789428711, -8.96399974822998, -8.969300270080566, -8.994600296020508, -9.04800033569336, -9.099100112915039, -9.107199668884277, -9.13659954071045, -9.149900436401367, -9.164199829101562, -9.169400215148926, -9.2253999710083, -9.230299949645996, -9.250699996948242, -7.992199897766113, -6.684999942779541, -6.6234002113342285, -5.912499904632568, -6.915800094604492, -7.949999809265137, -4.592599868774414, -7.430300235748291, -8.184000015258789, -6.059599876403809, -7.074999809265137, -5.550099849700928, -6.021999835968018, -5.4257001876831055, -5.974800109863281, -5.9268999099731445, -5.33620023727417, -7.233500003814697, -7.065800189971924, -6.260200023651123, -5.794400215148926, -4.567800045013428, -5.7677001953125, -5.226799964904785, -6.358399868011475, -4.905099868774414, -4.809999942779541, -4.508200168609619, -4.502999782562256, -4.931099891662598, -5.980999946594238, -4.409900188446045, -4.203700065612793, -4.862599849700928, -5.779900074005127, -5.129300117492676, -5.6219000816345215, -4.226399898529053, -5.326399803161621, -4.734899997711182, -4.832699775695801, -5.2469000816345215, -4.980599880218506, -5.473299980163574, -5.290800094604492, -5.317399978637695, -5.488900184631348, -5.165599822998047, -5.533299922943115, -5.298600196838379, -5.231299877166748, -5.182799816131592, -5.429599761962891, -5.3007001876831055, -5.314799785614014, -5.33459997177124, -5.318699836730957, -6.561100006103516, -7.806300163269043, -7.886199951171875, -8.108799934387207, -8.317000389099121, -8.345000267028809, -8.3774995803833, -8.484000205993652, -8.488699913024902, -8.507399559020996, -8.537199974060059, -8.540200233459473, -8.601900100708008, -8.646900177001953, -8.659600257873535, -8.660900115966797, -8.688799858093262, -8.694999694824219, -8.7253999710083, -8.74020004272461, -8.764200210571289, -8.765000343322754, -8.79520034790039, -8.812299728393555, -8.840700149536133, -8.869099617004395, -8.87600040435791, -8.8774995803833, -8.885299682617188, -8.911100387573242, -6.186800003051758, -8.451800346374512, -8.67140007019043, -7.896999835968018, -6.669099807739258, -8.416600227355957, -7.169400215148926, -6.77370023727417, -7.696000099182129, -6.997000217437744, -6.86299991607666, -7.446700096130371, -7.5543999671936035, -7.575399875640869, -6.879000186920166, -6.789000034332275, -6.604599952697754, -7.326600074768066, -7.472099781036377, -6.06220006942749, -7.526000022888184, -6.76669979095459, -7.371500015258789, -7.844399929046631, -6.526400089263916, -5.353799819946289, -7.137400150299072, -5.357500076293945, -4.987500190734863, -7.405099868774414, -6.876999855041504, -6.403900146484375, -3.643699884414673, -6.507800102233887, -6.291999816894531, -4.881899833679199, -4.742199897766113, -6.666999816894531, -6.035999774932861, -6.023600101470947, -5.139200210571289, -6.582200050354004, -6.632999897003174, -5.333399772644043, -5.608500003814697, -6.039899826049805, -5.870100021362305, -5.953999996185303, -5.47130012512207, -5.905600070953369, -5.980000019073486, -5.435800075531006, -4.9278998374938965, -5.515900135040283, -5.424200057983398, -5.715799808502197, -5.581600189208984, -5.586100101470947, -5.963399887084961, -5.528200149536133, -5.876999855041504, -5.46750020980835, -5.810999870300293, -5.954699993133545, -5.9278998374938965, -5.970399856567383, -5.986999988555908, -6.060299873352051, -6.220399856567383, -6.544300079345703, -6.930500030517578, -7.014599800109863, -7.057300090789795, -7.207200050354004, -7.213600158691406, -7.40500020980835, -7.458000183105469, -7.539599895477295, -7.684999942779541, -7.725599765777588, -7.726500034332275, -7.799499988555908, -7.805600166320801, -7.833399772644043, -7.872399806976318, -7.9085001945495605, -7.9878997802734375, -8.010899543762207, -8.061200141906738, -8.073599815368652, -8.105899810791016, -8.107600212097168, -8.1141996383667, -8.166399955749512, -8.173600196838379, -8.175000190734863, -8.187000274658203, -5.513199806213379, -7.8933000564575195, -5.407199859619141, -5.169400215148926, -5.087800025939941, -7.189499855041504, -5.444699764251709, -5.024899959564209, -4.952199935913086, -7.265500068664551, -6.124800205230713, -6.719200134277344, -4.991000175476074, -7.23199987411499, -5.936699867248535, -5.561800003051758, -6.75, -6.312699794769287, -4.864500045776367, -5.096700191497803, -6.048799991607666, -5.45959997177124, -5.46999979019165, -5.75570011138916, -5.6880998611450195, -5.997099876403809, -5.8003997802734375, -4.907400131225586, -6.50439977645874, -6.526800155639648, -6.427499771118164, -5.496200084686279, -5.362500190734863, -5.105100154876709, -5.160200119018555, -5.413099765777588, -5.779600143432617, -5.377099990844727, -5.2743000984191895, -5.572500228881836, -5.607999801635742, -5.681300163269043, -5.3730998039245605, -5.849400043487549, -6.100800037384033, -6.30049991607666, -6.577499866485596, -6.598700046539307, -6.8856000900268555, -7.193600177764893, -7.26170015335083, -7.296800136566162, -7.338799953460693, -7.348700046539307, -7.387199878692627, -7.498600006103516, -7.555500030517578, -7.6107001304626465, -7.668000221252441, -7.69290018081665, -7.700300216674805, -7.7058000564575195, -7.717800140380859, -7.7316999435424805, -7.807799816131592, -7.823999881744385, -7.84630012512207, -7.889500141143799, -7.901100158691406, -7.904300212860107, -7.9380998611450195, -7.958399772644043, -7.963399887084961, -7.971700191497803, -6.669600009918213, -6.7393999099731445, -5.9822001457214355, -6.821899890899658, -6.433899879455566, -6.697500228881836, -7.674600124359131, -7.177700042724609, -6.138500213623047, -7.099699974060059, -6.664299964904785, -6.106900215148926, -6.4446001052856445, -6.246600151062012, -6.897600173950195, -6.110099792480469, -7.088900089263916, -5.798299789428711, -5.697000026702881, -6.521699905395508, -5.1006999015808105, -5.43310022354126, -5.90339994430542, -6.063000202178955, -6.508999824523926, -6.7195000648498535, -6.048099994659424, -6.929800033569336, -6.0177001953125, -5.741199970245361, -5.520400047302246, -6.061200141906738, -5.430600166320801, -5.107100009918213, -5.590000152587891, -4.769599914550781, -4.725100040435791, -6.065199851989746, -5.500999927520752, -5.415999889373779, -5.815000057220459, -6.004700183868408, -5.652900218963623, -5.981599807739258, -5.885200023651123, -5.687300205230713, -5.760700225830078, -5.921000003814697, -5.917600154876709, -5.909200191497803, -5.993000030517578, -6.338600158691406, -6.377099990844727, -6.523099899291992, -6.906300067901611, -7.019000053405762, -7.190299987792969, -7.210400104522705, -7.232500076293945, -7.237800121307373, -7.258800029754639, -7.293600082397461, -7.314000129699707, -7.346099853515625, -7.349800109863281, -7.40880012512207, -7.4969000816345215, -7.426700115203857, -7.560100078582764, -7.5954999923706055, -7.723299980163574, -7.732699871063232, -7.753499984741211, -7.802199840545654, -7.815400123596191, -7.825399875640869, -7.857900142669678, -7.857999801635742, -7.868500232696533, -7.968200206756592, -7.970699787139893, -7.396699905395508, -6.7708001136779785, -5.19379997253418, -6.80049991607666, -5.749499797821045, -4.369900226593018, -6.656300067901611, -5.504799842834473, -6.174699783325195, -5.745999813079834, -6.940800189971924, -6.26800012588501, -7.0569000244140625, -6.060200214385986, -6.481100082397461, -6.199399948120117, -5.792699813842773, -6.036799907684326, -6.13640022277832, -5.733500003814697, -5.821599960327148, -6.511899948120117, -5.9120001792907715, -5.9593000411987305, -5.711299896240234, -5.665800094604492, -5.685200214385986, -4.069200038909912, -6.141900062561035, -6.303299903869629, -5.116799831390381, -5.254199981689453, -4.682000160217285, -5.407599925994873, -5.648499965667725, -5.456299781799316, -5.68179988861084, -5.436399936676025, -5.650000095367432, -5.534800052642822, -5.7108001708984375, -5.9633002281188965, -5.882900238037109, -6.005499839782715, -5.8618998527526855, -5.860499858856201, -5.98799991607666, -5.6992998123168945, -6.224400043487549, -6.618299961090088, -6.713200092315674, -6.714399814605713, -6.752299785614014, -6.822500228881836, -6.850500106811523, -7.163099765777588, -7.167399883270264, -7.230000019073486, -7.263800144195557, -7.288400173187256, -7.2957000732421875, -7.311699867248535, -7.315100193023682, -7.379700183868408, -7.547999858856201, -7.622600078582764, -7.639699935913086, -7.654699802398682, -7.683599948883057, -7.693699836730957, -7.723400115966797, -7.749499797821045, -7.758500099182129, -7.758600234985352, -7.759699821472168, -7.805799961090088, -7.822700023651123, -6.5278000831604, -6.8471999168396, -6.178500175476074, -6.6616997718811035, -6.005000114440918, -6.755499839782715, -6.383200168609619, -7.035799980163574, -6.480599880218506, -6.031400203704834, -6.884300231933594, -6.827499866485596, -5.1020002365112305, -5.013199806213379, -6.045400142669678, -6.020299911499023, -4.993899822235107, -5.940000057220459, -5.589700222015381, -6.48330020904541, -3.656100034713745, -5.577600002288818, -5.549300193786621, -4.933300018310547, -5.401199817657471, -6.058499813079834, -4.84060001373291, -5.640900135040283, -5.099999904632568, -6.016900062561035, -4.576099872589111, -5.946899890899658, -5.8607001304626465, -5.781300067901611, -5.268199920654297, -5.922999858856201, -5.293099880218506, -5.82390022277832, -5.916800022125244, -5.934500217437744, -5.818399906158447, -5.668700218200684, -5.8221001625061035, -5.893599987030029, -5.973299980163574, -5.964200019836426, -5.941800117492676, -5.957200050354004, -5.955399990081787, -4.424799919128418, -5.8958001136779785, -6.605800151824951, -6.6219000816345215, -6.966800212860107, -7.015699863433838, -7.0609002113342285, -7.105299949645996, -7.114299774169922, -7.265500068664551, -7.3531999588012695, -7.4475998878479, -7.448200225830078, -7.4492998123168945, -7.459000110626221, -7.486999988555908, -7.490900039672852, -7.507199764251709, -7.529399871826172, -7.540500164031982, -7.541600227355957, -7.548399925231934, -7.5680999755859375, -7.584199905395508, -7.596099853515625, -7.62529993057251, -7.780600070953369, -7.780799865722656, -7.789000034332275, -7.8130998611450195, -6.036300182342529, -5.537399768829346, -6.3379998207092285, -6.01230001449585, -6.524700164794922, -6.651000022888184, -6.928400039672852, -5.486800193786621, -6.563499927520752, -6.741700172424316, -6.24399995803833, -7.223999977111816, -5.5528998374938965, -6.400700092315674, -6.638800144195557, -4.972599983215332, -6.313399791717529, -7.213200092315674, -5.438199996948242, -6.466800212860107, -6.349599838256836, -5.934800148010254, -6.489999771118164, -5.876399993896484, -5.885300159454346, -5.923699855804443, -6.122099876403809, -6.345399856567383, -5.2027997970581055, -6.1697998046875, -6.2170000076293945, -6.356200218200684, -5.353899955749512, -5.740600109100342, -4.9878997802734375, -5.5833001136779785, -5.632199764251709, -5.9421000480651855, -5.452600002288818, -6.036399841308594, -5.716300010681152, -6.070499897003174, -6.145899772644043, -6.142399787902832, -6.0914998054504395, -6.123899936676025, -6.004000186920166, -6.208799839019775, -6.264400005340576, -6.4004998207092285, -6.461999893188477, -6.478600025177002, -6.484300136566162, -6.499199867248535, -6.520500183105469, -6.576399803161621, -6.6296000480651855, -6.685100078582764, -6.730800151824951, -6.783999919891357, -6.786200046539307, -6.854000091552734, -6.877299785614014, -6.920899868011475, -6.9593000411987305, -6.988399982452393, -7.010200023651123, -7.015900135040283, -7.083399772644043, -7.0920000076293945, -7.1057000160217285, -7.116600036621094, -7.1331000328063965, -7.143700122833252, -7.206699848175049, -7.239699840545654, -6.272200107574463, -6.292799949645996, -6.161900043487549, -5.8404998779296875, -5.837900161743164, -4.907700061798096, -6.611800193786621, -6.821300029754639, -6.747200012207031, -5.702499866485596, -5.926700115203857, -5.838699817657471, -6.155399799346924, -5.34660005569458, -5.676199913024902, -5.912300109863281, -6.272500038146973, -5.955100059509277, -5.581900119781494, -4.835400104522705, -6.219200134277344, -5.427199840545654, -4.695300102233887, -5.404799938201904, -4.925000190734863, -6.06279993057251, -5.2118000984191895, -5.050899982452393, -5.963200092315674, -5.235499858856201, -4.82480001449585, -5.291600227355957, -5.14900016784668, -4.980199813842773, -5.598700046539307, -4.732399940490723, -5.729800224304199, -5.073699951171875, -5.367099761962891, -5.237800121307373, -5.773799896240234, -5.658899784088135, -5.807000160217285, -5.898499965667725, -5.193900108337402, -5.198999881744385, -5.367099761962891, -5.753300189971924, -5.787499904632568, -5.837600231170654, -5.926000118255615, -5.936299800872803, -6.013500213623047, -6.076200008392334, -6.183000087738037, -6.267600059509277, -6.273099899291992, -5.471099853515625, -6.347799777984619, -6.3491997718811035, -6.408199787139893, -6.4918999671936035, -6.49429988861084, -6.525899887084961, -6.531499862670898, -6.5609002113342285, -6.570400238037109, -6.572400093078613, -6.613100051879883, -6.6280999183654785, -6.656400203704834, -6.722400188446045, -6.744500160217285, -6.7540998458862305, -6.592599868774414, -5.31279993057251, -5.3420000076293945, -6.226200103759766, -4.873899936676025, -6.1616997718811035, -5.012499809265137, -5.00439977645874, -5.411200046539307, -5.769000053405762, -5.836400032043457, -6.303299903869629, -5.386899948120117, -6.296500205993652, -5.625, -4.999499797821045, -4.657899856567383, -5.114500045776367, -5.416900157928467, -5.774600028991699, -5.917900085449219, -5.621699810028076, -6.0218000411987305, -6.0493998527526855, -6.071400165557861]}, \"token.table\": {\"Topic\": [1, 5, 6, 8, 3, 10, 2, 1, 3, 1, 2, 3, 5, 6, 7, 9, 1, 2, 3, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 6, 7, 8, 9, 1, 2, 3, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 3, 3, 8, 2, 8, 3, 3, 5, 8, 9, 10, 2, 5, 5, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 8, 10, 1, 2, 3, 4, 5, 6, 7, 8, 10, 2, 3, 5, 8, 7, 4, 2, 3, 5, 7, 2, 3, 5, 2, 5, 1, 2, 5, 5, 1, 2, 3, 4, 5, 6, 1, 6, 5, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 7, 8, 10, 7, 8, 10, 1, 8, 1, 2, 3, 6, 7, 8, 6, 3, 8, 1, 2, 3, 6, 7, 2, 4, 7, 10, 1, 2, 3, 4, 5, 6, 8, 2, 5, 6, 8, 9, 1, 2, 5, 2, 4, 5, 6, 7, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 5, 1, 4, 5, 2, 2, 5, 7, 8, 10, 10, 2, 7, 3, 5, 8, 9, 10, 7, 3, 10, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 4, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 3, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 10, 5, 6, 9, 1, 2, 5, 9, 1, 2, 3, 4, 5, 6, 1, 4, 6, 1, 2, 3, 4, 5, 9, 10, 1, 2, 4, 6, 7, 8, 7, 7, 2, 3, 4, 5, 6, 9, 2, 6, 7, 9, 10, 2, 2, 5, 3, 9, 9, 9, 9, 2, 2, 1, 2, 4, 5, 6, 6, 2, 2, 5, 7, 2, 4, 8, 1, 2, 4, 6, 7, 8, 9, 10, 5, 6, 1, 3, 7, 8, 6, 10, 9, 7, 1, 2, 3, 4, 6, 7, 9, 1, 2, 3, 4, 6, 7, 9, 3, 5, 3, 5, 8, 10, 10, 2, 9, 10, 9, 10, 8, 2, 5, 6, 7, 8, 9, 10, 10, 4, 6, 1, 2, 4, 7, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 8, 3, 2, 3, 6, 8, 1, 2, 3, 5, 6, 2, 5, 6, 9, 10, 10, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 3, 4, 5, 8, 1, 2, 5, 4, 7, 5, 6, 2, 3, 5, 2, 6, 1, 3, 6, 8, 9, 6, 5, 1, 3, 2, 3, 10, 9, 4, 9, 4, 7, 10, 9, 1, 2, 3, 5, 6, 8, 4, 6, 4, 1, 2, 3, 4, 6, 7, 8, 9, 4, 1, 2, 4, 5, 6, 1, 2, 3, 4, 5, 6, 8, 4, 5, 6, 3, 4, 2, 6, 7, 9, 6, 7, 9, 1, 2, 3, 5, 6, 7, 8, 9, 1, 2, 3, 5, 6, 7, 8, 9, 5, 5, 1, 3, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 10, 1, 2, 4, 3, 3, 8, 1, 4, 7, 8, 5, 6, 1, 1, 4, 1, 2, 5, 6, 7, 8, 1, 2, 3, 5, 8, 10, 2, 9, 10, 6, 1, 2, 5, 6, 8, 10, 1, 4, 6, 7, 6, 2, 3, 9, 3, 4, 2, 3, 4, 7, 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 7, 8, 10, 1, 2, 3, 6, 7, 9, 8, 2, 6, 8, 2, 8, 7, 3, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 5, 6, 7, 8, 9, 10, 7, 2, 3, 4, 5, 6, 8, 10, 2, 3, 6, 7, 8, 9, 1, 2, 5, 10, 1, 2, 3, 6, 7, 8, 3, 3, 8, 2, 3, 4, 6, 9, 1, 2, 5, 6, 7, 9, 10, 3, 4, 6, 9, 2, 4, 6, 7, 8, 9, 1, 2, 3, 6, 3, 2, 3, 5, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 5, 6, 7, 8, 9, 10, 4, 5, 6, 9, 1, 2, 3, 4, 6, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 7, 9, 4, 7, 8, 4, 8, 7, 6, 1, 3, 4, 6, 8, 1, 2, 3, 4, 5, 6, 9, 9, 1, 2, 4, 5, 6, 7, 9, 10, 1, 2, 4, 7, 9, 10, 4, 2, 3, 4, 5, 7, 9, 4, 5, 6, 8, 9, 10, 6, 9, 1, 2, 5, 4, 7, 8, 10, 1, 2, 3, 5, 6, 8, 9, 1, 2, 3, 5, 6, 7, 8, 10, 8, 8, 8, 1, 2, 3, 4, 5, 6, 7, 9, 10, 1, 2, 3, 4, 5, 6, 7, 9, 10, 2, 6, 6, 5, 7, 10, 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, 9, 10, 5, 1, 2, 3, 4, 5, 6, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 5, 6, 5, 6, 3, 5, 8, 4, 1, 2, 3, 4, 6, 7, 8, 9, 10, 3, 6, 5, 6, 8, 9, 10, 5, 8, 10, 1, 2, 4, 5, 6, 9, 10, 1, 2, 5, 9, 10, 10, 2, 9, 10, 4, 7, 10, 3, 6, 1, 2, 3, 4, 5, 6, 7, 9, 10, 5, 6, 2, 3, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 5, 3, 8, 8, 1, 2, 3, 7, 8, 10, 8, 7, 9, 1, 2, 3, 7, 8, 9, 10, 7, 1, 2, 3, 4, 5, 6, 8, 10, 7, 1, 2, 5, 1, 5, 6, 1, 2, 4, 5, 6, 4, 8, 10, 3, 5, 3, 5, 4, 6, 2, 1, 2, 3, 4, 5, 6, 8, 9, 8, 3, 4, 6, 9, 1, 3, 3, 3, 1, 5, 1, 3, 6, 4, 1, 2, 3, 2, 5, 6, 7, 8, 9, 9, 10, 2, 4, 5, 6, 7, 8, 9, 10, 4, 9, 5, 7, 9, 5, 10, 1, 2, 3, 4, 6, 7, 9, 2, 4, 5, 6, 7, 8, 9, 10, 4, 5, 7, 8, 9, 4, 5, 6, 9, 7, 7, 9, 10, 2, 3, 4, 5, 10, 1, 2, 4, 5, 6, 5, 6, 3, 5, 6, 5, 6, 8, 6, 2, 5, 7, 8, 9, 1, 2, 3, 4, 5, 6, 10, 5, 10, 9, 3, 5, 7, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 2, 3, 4, 5, 6, 7, 8, 9, 10, 8, 7, 8, 1, 2, 3, 5, 6, 7, 8, 6, 9, 10, 9, 8, 1, 2, 10, 6, 1, 2, 3, 4, 7, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 6, 7, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 3, 4, 5, 6, 2, 2, 3, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 4, 5, 6, 7, 9, 10, 2, 1, 2, 3, 4, 5, 6, 7, 9, 1, 2, 3, 4, 7, 9, 2, 3, 4, 1, 4, 7, 3, 5, 8, 4, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 4, 8, 4, 4, 4, 10, 2, 4, 6, 7, 9, 10, 7, 2, 4, 6, 7, 9, 10, 5, 6, 1, 2, 5, 7, 8, 9, 2, 5, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 2, 3, 8, 2, 3, 4, 5, 6, 9, 10, 10, 9, 4, 1, 5, 6, 10, 1, 2, 3, 5, 10, 7, 1, 2, 4, 5, 6, 7, 8, 9, 10, 9, 6, 5, 8, 1, 2, 5, 6, 7, 1, 2, 4, 1, 2, 4, 5, 5, 6, 5, 6, 3, 4, 9, 10, 1, 2, 3, 4, 9, 2, 3, 4, 2, 1, 2, 3, 5, 6, 7, 8, 9, 1, 2, 3, 4, 5, 7, 8, 9, 10, 5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 4, 6, 8, 6, 6, 10, 9, 1, 2, 3, 5, 6, 7, 10, 7, 7, 3, 5, 2, 3, 4, 5, 6, 7, 8, 1, 2, 5, 6, 7, 2, 5, 7, 1, 3, 4, 5, 6, 8, 8, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 9, 1, 2, 4, 5, 6, 8, 9, 1, 4, 2, 8, 9, 4, 1, 10, 3, 5, 8, 1, 2, 3, 4, 5, 6, 7, 9, 1, 3, 4, 6, 7, 8, 9, 4, 3, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 6, 7, 2, 9, 1, 4, 4, 2, 3, 4, 4, 9, 1, 5, 3, 4, 5, 6, 7, 8, 9, 10, 3, 9, 1, 2, 3, 5, 6, 1, 2, 3, 4, 6, 7, 8, 9, 1, 2, 3, 4, 6, 7, 8, 9, 1, 2, 3, 4, 6, 7, 8, 9, 6, 3, 1, 4, 5, 8, 9, 2, 3, 5, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 5, 6, 7, 8, 1, 5, 4, 5, 6, 8, 10, 3, 8, 1, 2, 3, 4, 5, 8, 1, 5, 2, 7, 3, 5, 6, 7, 8, 9, 5, 5, 8, 2, 2, 3, 9, 10, 5, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 1, 2, 3, 4, 5, 6, 9, 1, 3, 3, 4, 3, 3, 5, 6, 10, 1, 2, 3, 4, 5, 7, 8, 1, 5, 2, 8, 4, 5, 6, 7, 8, 9, 2, 4, 7, 5, 9, 2, 5, 5, 2, 4, 5, 6, 9, 10, 2, 1, 2, 3, 4, 6, 7, 8, 9, 10, 1, 2, 3, 4, 7, 8, 9, 10, 8, 5, 1, 5, 8, 1, 5, 8, 8, 6, 2, 1, 2, 3, 8, 1, 2, 3, 5, 6, 6, 1, 2, 5, 6, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 10, 2, 5, 2, 3, 5, 5, 6, 5, 1, 2, 5, 6, 7, 8, 4, 7, 8, 9, 10, 1, 2, 3, 5, 6, 7, 8, 9, 1, 6, 7, 8, 9, 5, 2, 2, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 5, 6, 7, 8, 1, 4, 6, 2, 6, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 9, 5, 5, 8, 3, 9, 1, 2, 3, 4, 5, 7, 8, 9, 10, 1, 2, 6, 7, 8, 5, 4, 7, 9, 10, 2, 4, 7, 2, 4, 6, 10, 6, 2, 3, 4, 7, 8, 9, 10, 4, 3, 4, 8, 3, 5, 8, 1, 2, 3, 4, 5, 6, 2, 3, 4, 5, 6, 9, 2, 7, 9, 10, 2, 7, 9, 4, 8, 1, 2, 3, 5, 8, 8, 1, 2, 3, 5, 6, 7, 8, 9, 3, 6, 1, 3, 4, 7, 10, 10, 9, 1, 2, 4, 5, 7, 8, 9, 10, 1, 3, 4, 5, 8, 1, 2, 4, 5, 6, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 6, 7, 8, 9, 4, 3, 1, 2, 5, 1, 3, 8, 3, 6, 1, 3, 4, 6, 7, 9, 1, 2, 3, 4, 5, 6, 10, 1, 2, 6, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 4, 1, 5, 1, 2, 5, 2, 3, 6, 8, 1, 2, 3, 4, 5, 6, 1, 2, 3, 5, 1, 2, 3, 5, 1, 3, 1, 6, 6, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 9, 2, 4, 5, 6, 7, 9, 6, 9, 2, 4, 5, 6, 7, 8, 10, 4, 1, 10, 5, 8, 10, 1, 2, 3, 4, 6, 7, 8, 9, 1, 2, 3, 4, 5, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 3, 5, 8, 1, 3, 1, 2, 9, 10, 1, 4, 8, 8, 1, 1, 1, 2, 3, 4, 5, 6, 7, 8, 5, 1, 2, 5, 6, 7, 8, 9, 3, 1, 2, 3, 4, 6, 7, 8, 1, 3, 6, 7, 8, 1, 2, 3, 5, 6, 7, 5, 6, 4, 9, 1, 2, 3, 4, 5, 6, 9, 10, 8, 2, 7, 1, 5, 1, 9, 1, 2, 5, 1, 2, 3, 4, 5, 7, 9, 1, 2, 3, 8, 1, 2, 3, 5, 1, 3, 4, 5, 8, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 7, 8, 10, 4, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 7, 5, 8, 10, 8, 10, 1, 5, 1, 2, 3, 4, 6, 7, 8, 9, 4, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 3, 4, 5, 6, 1, 3, 4, 5, 6, 10, 2, 3, 4, 7, 8, 9, 10, 7, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3, 4, 5, 7, 8, 9, 1, 2, 5, 1, 2, 5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 2, 3, 4, 6, 7, 8, 9, 2, 5, 5], \"Freq\": [0.9984179735183716, 0.004689114633947611, 0.9917477369308472, 0.9967918395996094, 0.9945637583732605, 0.9952183365821838, 0.9931555986404419, 0.9966499209403992, 0.9959320425987244, 0.6982988715171814, 0.17186155915260315, 0.02354191616177559, 0.0012800568947568536, 0.0198130551725626, 0.07045878469944, 0.014692827127873898, 0.13265639543533325, 0.13498370349407196, 0.0019394209375604987, 0.44005462527275085, 0.05042494460940361, 0.0002909131289925426, 0.06749185174703598, 0.007951625622808933, 0.16426895558834076, 0.30245548486709595, 0.10061051696538925, 0.004367588087916374, 0.0004679558624047786, 0.0003119705943390727, 0.3175860643386841, 0.00405561737716198, 0.2703225016593933, 0.5384613275527954, 0.19853706657886505, 0.0725724995136261, 0.1336367428302765, 0.010490522719919682, 0.04634619131684303, 0.5238532423973083, 0.26128384470939636, 0.061785757541656494, 0.0027317500207573175, 0.07576588541269302, 0.06025918945670128, 0.008114904165267944, 0.005865227896720171, 0.00032138233655132353, 0.9968941807746887, 0.9956451058387756, 0.9971714019775391, 0.9558536410331726, 0.042587537318468094, 0.9973006844520569, 0.010999448597431183, 0.09288423508405685, 0.03788698837161064, 0.8237364888191223, 0.03299834579229355, 0.9955292344093323, 0.9992022514343262, 0.9994375705718994, 0.9993559718132019, 0.052999548614025116, 0.26300331950187683, 0.20619963109493256, 0.037007350474596024, 0.05665596202015877, 0.15283074975013733, 0.12557384371757507, 0.042030300945043564, 0.062343716621398926, 0.001403472269885242, 0.998022735118866, 0.0897006243467331, 0.2312185913324356, 0.008708798326551914, 0.04006047174334526, 0.5795705318450928, 0.050946470350027084, 0.11254186928272247, 0.07793715596199036, 0.21880589425563812, 0.0024498910643160343, 0.010412036441266537, 0.0010718272533267736, 0.0007655909284949303, 0.5333106517791748, 0.042873091995716095, 0.03244051709771156, 0.2530360519886017, 0.0055612316355109215, 0.7081301808357239, 0.9980002641677856, 0.9969857931137085, 0.866216778755188, 0.00994698703289032, 0.09864094853401184, 0.02486746571958065, 0.9684280157089233, 0.013918810524046421, 0.017130844295024872, 0.9351749420166016, 0.06423203647136688, 0.7916972637176514, 0.1803330034017563, 0.027580341324210167, 0.9974991083145142, 0.619029700756073, 0.20564419031143188, 0.12175814062356949, 0.0038242170121520758, 0.03626837953925133, 0.013446440920233727, 0.9955983757972717, 0.9962297677993774, 0.29450494050979614, 0.7053674459457397, 0.19564980268478394, 0.1387573927640915, 0.14185549318790436, 0.16992616653442383, 0.2611793577671051, 0.08327320963144302, 0.007698312867432833, 0.0002816455962602049, 9.38818629947491e-05, 0.0013143460964784026, 0.21372392773628235, 0.02204027958214283, 0.6418396830558777, 0.11854998767375946, 0.002337605459615588, 0.0013357745483517647, 0.8929890990257263, 0.10605206340551376, 0.9987603425979614, 0.9970216155052185, 0.9965403079986572, 0.0015913480892777443, 0.02015707641839981, 0.1713351458311081, 0.743690013885498, 0.05251448601484299, 0.01060898695141077, 0.9976888298988342, 0.9879950284957886, 0.009654674679040909, 0.3321654498577118, 0.0518856979906559, 0.5364754796028137, 0.028933146968483925, 0.050430960953235626, 0.03359996899962425, 0.955919086933136, 0.010079991072416306, 0.9944415092468262, 0.8595820069313049, 0.01701256074011326, 0.023280344903469086, 0.025071142241358757, 0.046112991869449615, 0.0026861936785280704, 0.025966539978981018, 0.8383722305297852, 0.062403108924627304, 0.039341092109680176, 0.027810081839561462, 0.031201554462313652, 0.9617123603820801, 0.011875564232468605, 0.02625124715268612, 0.09842314571142197, 0.6659966111183167, 0.1295904666185379, 0.10443789511919022, 0.00109359051566571, 0.9959701299667358, 0.12871968746185303, 0.30949148535728455, 0.02736615762114525, 0.2369147539138794, 0.230143740773201, 0.020242493599653244, 0.0011990326456725597, 0.04020285978913307, 0.005078256130218506, 0.0006347820162773132, 0.7916082143783569, 0.08591412752866745, 0.1224544420838356, 0.8667373061180115, 0.05130408704280853, 0.08148296177387238, 0.9988743662834167, 0.0737469494342804, 0.14268431067466736, 0.13787472248077393, 0.005611180793493986, 0.6396746039390564, 0.9957911968231201, 0.9982594847679138, 0.0017513324273750186, 0.006709168199449778, 0.42319366335868835, 0.5692471265792847, 0.1105446070432663, 0.8870530724525452, 0.9954899549484253, 0.9963112473487854, 0.9964851140975952, 0.9976377487182617, 0.08480996638536453, 0.3755114674568176, 0.0935341864824295, 0.007137996144592762, 0.0009517327998764813, 0.03986702859401703, 0.2513103485107422, 0.03050832450389862, 0.11547691375017166, 0.0009517327998764813, 0.9984228610992432, 0.49644583463668823, 0.3472391963005066, 0.02409367449581623, 0.0020900536328554153, 0.01706877164542675, 0.014514261856675148, 0.07471942156553268, 0.005631533917039633, 0.014804547652602196, 0.00336730876006186, 0.9974470734596252, 0.22932244837284088, 0.3242872655391693, 0.024721000343561172, 0.05135134607553482, 0.14862747490406036, 0.0030147561337798834, 0.1142592579126358, 0.0019093455048277974, 0.09607023000717163, 0.006632463540881872, 0.24221353232860565, 0.3301810920238495, 0.09435318410396576, 0.01376213226467371, 0.025432420894503593, 0.19057801365852356, 0.10095900297164917, 0.0017615529941394925, 0.00044038824853487313, 0.00033029119367711246, 0.06977737694978714, 0.022818192839622498, 0.23281171917915344, 0.02761332131922245, 0.18667927384376526, 0.12401191890239716, 0.013723986223340034, 0.10929583758115768, 0.10631955415010452, 0.10681559890508652, 0.9991850256919861, 0.45262953639030457, 0.547208845615387, 0.9981287121772766, 0.9052091836929321, 0.09165060520172119, 0.002915327437222004, 0.9976480007171631, 0.052736639976501465, 0.1887654811143875, 0.029876550659537315, 0.6552470922470093, 0.07197532802820206, 0.0011316875461488962, 0.0677485316991806, 0.818550705909729, 0.11229550838470459, 0.05071499943733215, 0.1940625011920929, 0.06934499740600586, 0.4636799991130829, 0.0031049998942762613, 0.006209999788552523, 0.21321000158786774, 0.0848994106054306, 0.04940267652273178, 0.0036594574339687824, 0.040619976818561554, 0.1079539954662323, 0.7132282853126526, 0.9981080293655396, 0.997953474521637, 0.09840324521064758, 0.02175229787826538, 0.6466992497444153, 0.12464411556720734, 0.034527454525232315, 0.073543481528759, 0.15196438133716583, 0.03526599332690239, 0.15581157803535461, 0.133369579911232, 0.5238603353500366, 0.9930914640426636, 0.9990504384040833, 0.9966468214988708, 0.9969688653945923, 0.9975600838661194, 0.9984020590782166, 0.9969102740287781, 0.9993650913238525, 0.9952067136764526, 0.9975303411483765, 0.01764167845249176, 0.003822363680228591, 0.2610968351364136, 0.6642091870307922, 0.05292503535747528, 0.9963208436965942, 0.9985261559486389, 0.9462509751319885, 0.017132679000496864, 0.0355832539498806, 0.9993313550949097, 0.998347282409668, 0.9962425827980042, 0.21371199190616608, 0.13094183802604675, 0.0012419958366081119, 0.07984258979558945, 0.3968176543712616, 0.013661953620612621, 0.16367730498313904, 8.871399040799588e-05, 0.9758363366127014, 0.02256269007921219, 0.9948293566703796, 0.06003412976861, 0.05931943655014038, 0.8797858953475952, 0.9984402060508728, 0.9987019896507263, 0.9979405999183655, 0.9993191361427307, 0.3011002540588379, 0.10724987834692001, 0.28494200110435486, 0.03553842380642891, 0.06199697405099869, 0.08464783430099487, 0.12453092634677887, 0.3419370949268341, 0.23872482776641846, 0.37762516736984253, 0.00023204201715998352, 0.03786925598978996, 0.0006033092504367232, 0.0030165461357682943, 0.9920653700828552, 0.0033515722025185823, 0.30598199367523193, 0.0036318339407444, 0.1053231805562973, 0.5847252607345581, 0.9956570863723755, 0.0104899350553751, 0.8830617666244507, 0.1048993468284607, 0.08606896549463272, 0.9125384092330933, 0.9976516962051392, 0.3812676966190338, 0.03067224845290184, 0.17840377986431122, 0.33894774317741394, 0.03261353075504303, 0.03436068445444107, 0.003300178563222289, 0.998062789440155, 0.9948199987411499, 0.9960567951202393, 0.210385262966156, 0.35114049911499023, 0.030543241649866104, 0.1750362664461136, 0.21732690930366516, 0.015592005103826523, 0.1577640324831009, 0.10090353339910507, 0.16376370191574097, 0.29602915048599243, 0.1107211709022522, 0.10444879531860352, 0.03913420811295509, 0.001227205153554678, 0.02536224015057087, 0.0005454245256260037, 0.9973345994949341, 0.9944406151771545, 0.04023607075214386, 0.8069161176681519, 0.11631882935762405, 0.035846684128046036, 0.025140751153230667, 0.18436551094055176, 0.03603507578372955, 0.753384530544281, 0.9985305666923523, 0.9974706768989563, 0.16006077826023102, 0.5147776007652283, 0.12500423192977905, 0.1997300237417221, 0.9984682202339172, 0.9978067874908447, 0.5785580277442932, 0.24475298821926117, 0.1220434233546257, 0.012586494907736778, 0.020510027185082436, 0.00010517962073208764, 0.016618380323052406, 0.0043824841268360615, 0.0003505987406242639, 0.00010517962073208764, 0.022451126947999, 0.2694135308265686, 0.12123608589172363, 0.06361152231693268, 0.5231112241744995, 0.8220941424369812, 0.07791326195001602, 0.10012218356132507, 0.005149167962372303, 0.9937894344329834, 0.9875175952911377, 0.011494914069771767, 0.03128904476761818, 0.08391152322292328, 0.8846266269683838, 0.09389286488294601, 0.9055305123329163, 0.02158946357667446, 0.09491217136383057, 0.6717174649238586, 0.06354634463787079, 0.14786745607852936, 0.9970578551292419, 0.9965674877166748, 0.9976062178611755, 0.9987249374389648, 0.0394725576043129, 0.8955336213111877, 0.06475966423749924, 0.9973831176757812, 0.9968755841255188, 0.9961128830909729, 0.10433661937713623, 0.8445709347724915, 0.05062486603856087, 0.9962384700775146, 0.036791566759347916, 0.14386771619319916, 0.205779030919075, 0.03958265110850334, 0.5665901303291321, 0.006850843317806721, 0.9349079132080078, 0.06474354863166809, 0.9988182187080383, 0.03143448010087013, 0.042567525058984756, 0.03339913487434387, 0.2043241262435913, 0.30615875124931335, 0.07531177997589111, 0.021283762529492378, 0.28552988171577454, 0.999370813369751, 0.08288067579269409, 0.04242488369345665, 0.11796623468399048, 0.3290165960788727, 0.42782899737358093, 0.028691455721855164, 0.033248335123062134, 0.14160077273845673, 0.11105281114578247, 0.17400524020195007, 0.39560455083847046, 0.11560969054698944, 0.7247521877288818, 0.15227505564689636, 0.12271956354379654, 0.8889402151107788, 0.10933489352464676, 0.018538974225521088, 0.08098604530096054, 0.07318016141653061, 0.8274236917495728, 0.8702435493469238, 0.07711860537528992, 0.05185561254620552, 0.7130985260009766, 0.0006606842507608235, 0.07906188070774078, 0.011892315931618214, 0.010350720025599003, 0.1766229122877121, 0.0019820528104901314, 0.00616638595238328, 0.08333856612443924, 0.3766956627368927, 0.4422428607940674, 0.00775864627212286, 0.0663498044013977, 0.02193824201822281, 0.0013376976130530238, 0.00040130928391590714, 0.9963006973266602, 0.9962567687034607, 0.3943304121494293, 0.24138101935386658, 0.15967516601085663, 0.16283047199249268, 0.031387005001306534, 0.0098810950294137, 0.000498206471092999, 0.0858951061964035, 0.02496950700879097, 0.1418268084526062, 0.032959748059511185, 0.07490852475166321, 0.15980485081672668, 0.4804133176803589, 0.997737467288971, 0.9994537830352783, 0.9974147081375122, 0.9987572431564331, 0.9797942638397217, 0.01991172693669796, 0.9914811849594116, 0.06851358711719513, 0.8896225690841675, 0.0411081537604332, 0.05429108440876007, 0.9453036189079285, 0.9939628839492798, 0.9966124892234802, 0.9988262057304382, 0.1886167824268341, 0.6665637493133545, 0.0016196600627154112, 0.006773123983293772, 0.0506511889398098, 0.08569474518299103, 0.06655214726924896, 0.06295473128557205, 0.10117725282907486, 0.7437652349472046, 0.026081247255206108, 0.9984585046768188, 0.9932154417037964, 0.13182874023914337, 0.8670274615287781, 0.9974617958068848, 0.2830798923969269, 0.15659739077091217, 0.49011972546577454, 0.06836078315973282, 0.001656318549066782, 0.9959103465080261, 0.9905865788459778, 0.910865843296051, 0.06395073235034943, 0.024197574704885483, 0.9971279501914978, 0.023378724232316017, 0.9579839706420898, 0.018485503271222115, 0.9548439979553223, 0.04388348013162613, 0.03207756578922272, 0.9516344666481018, 0.013365652412176132, 0.0026731304824352264, 0.31639090180397034, 0.2436877191066742, 0.12427657097578049, 0.06853282451629639, 0.00041703545139171183, 0.2467459887266159, 0.22702527046203613, 0.12900054454803467, 0.09371163696050644, 0.01215506624430418, 0.010194571688771248, 0.5273730158805847, 0.00039209891110658646, 0.05443955957889557, 0.7457141280174255, 0.14499248564243317, 0.01212762389332056, 0.03449635207653046, 0.008085083216428757, 0.9977689981460571, 0.9382480382919312, 0.00023953230993356556, 0.06132027134299278, 0.9572054147720337, 0.04228818044066429, 0.9986337423324585, 0.09336869418621063, 0.9058157205581665, 0.5401382446289062, 0.1922232061624527, 0.1002173125743866, 0.016314446926116943, 0.06840143352746964, 0.02739851363003254, 0.03620614483952522, 0.017615266144275665, 0.001029815524816513, 0.00043360653216950595, 0.4730895459651947, 0.36296501755714417, 0.05666325241327286, 0.03613640367984772, 0.0005717785097658634, 0.04528485983610153, 0.018925869837403297, 0.006232385989278555, 0.00017153355292975903, 0.9973748326301575, 0.08625262975692749, 0.041960738599300385, 0.001554101356305182, 0.4698566496372223, 0.3447515070438385, 0.05491158366203308, 0.000777050678152591, 0.2869473695755005, 0.2297266721725464, 0.055363960564136505, 0.2835715115070343, 0.07224321365356445, 0.07190563529729843, 0.9073697328567505, 0.0017283232882618904, 0.07604622095823288, 0.013826586306095123, 0.17650547623634338, 0.14983917772769928, 0.6574511528015137, 0.010793500579893589, 0.003809470683336258, 0.0012698235223069787, 0.9972764253616333, 0.9941398501396179, 0.9978348016738892, 0.2224045693874359, 0.19772079586982727, 0.5775001049041748, 0.0001252983493031934, 0.0021300718653947115, 0.23985137045383453, 0.6416506767272949, 0.0029827896505594254, 0.052286546677351, 0.021756818518042564, 0.01947586052119732, 0.021756818518042564, 0.0928572565317154, 0.8637582063674927, 0.015042874962091446, 0.028228605166077614, 0.7203767895698547, 0.023552987724542618, 0.017808357253670692, 0.18871113657951355, 0.013499883934855461, 0.035903945565223694, 0.29292693734169006, 0.2857851982116699, 0.4149126410484314, 0.006335423327982426, 0.9957003593444824, 0.072479747235775, 0.03204682841897011, 0.5271254181861877, 0.05630657821893692, 0.07427676767110825, 0.2375059425830841, 0.3694712221622467, 0.04512036219239235, 0.30796197056770325, 0.0007040778873488307, 4.2414329072926193e-05, 0.10006389021873474, 0.14461590349674225, 0.02909623086452484, 0.0023667195346206427, 0.0005598691641353071, 0.3137110769748688, 0.09743146598339081, 0.17287002503871918, 0.0002881167165469378, 0.17320616543293, 0.19318224489688873, 0.046674907207489014, 0.0004321750602684915, 0.00220889481715858, 0.7069599628448486, 0.07069599628448486, 0.20675915479660034, 0.015276019461452961, 0.14040976762771606, 0.09964563697576523, 0.12255880236625671, 0.04209628701210022, 0.07167025655508041, 0.4284229576587677, 0.07007166743278503, 0.0253110583871603, 0.030521167442202568, 0.04047372192144394, 0.00597153278067708, 0.06900437921285629, 0.7271999716758728, 0.0013270073104649782, 0.025876641273498535, 0.061042334884405136, 0.03781970590353012, 0.21501947939395905, 0.4348568916320801, 0.07825247943401337, 0.04242885112762451, 0.03500768914818764, 0.08878198266029358, 0.025255270302295685, 0.07347340881824493, 0.006916058249771595, 0.9990335702896118, 0.998776376247406, 0.07984903454780579, 0.09856364876031876, 0.8209478855133057, 0.11462793499231339, 0.8842726349830627, 0.9978539347648621, 0.9985815286636353, 0.07964928448200226, 0.15108361840248108, 0.024287672713398933, 0.0010715150274336338, 0.743988573551178, 0.12646332383155823, 0.22315700352191925, 0.016846392303705215, 0.5887006521224976, 0.0133848050609231, 0.0133848050609231, 0.018000254407525063, 0.9987553954124451, 0.05119602382183075, 0.57380610704422, 0.0005132433725520968, 0.16167165338993073, 0.0651819035410881, 0.11086056381464005, 0.016295475885272026, 0.02027311362326145, 0.2759804129600525, 0.5969468355178833, 0.004298999905586243, 0.0051434459164738655, 0.11384672671556473, 0.0037616246845573187, 0.997040331363678, 0.28522568941116333, 0.028192387893795967, 0.00025398549041710794, 0.6349636912345886, 0.012953259982168674, 0.03835180774331093, 0.2287294864654541, 0.19304534792900085, 0.13981163501739502, 0.1035425066947937, 0.08891785889863968, 0.24627907574176788, 0.10976136475801468, 0.8896447420120239, 0.9408957362174988, 0.014294215478003025, 0.04372348263859749, 0.049868445843458176, 0.8427767157554626, 0.014960533939301968, 0.09225662797689438, 0.1971670687198639, 0.10042048990726471, 0.19145207107067108, 0.05796629935503006, 0.03265707194805145, 0.40943801403045654, 0.010613547638058662, 0.2219708263874054, 0.07651369273662567, 0.0405549630522728, 0.002162931254133582, 0.5528993010520935, 0.05299181491136551, 0.04190679267048836, 0.010814656503498554, 0.9990121722221375, 0.9963353276252747, 0.9990687370300293, 0.32969266176223755, 0.30109816789627075, 0.04192644730210304, 0.1405683308839798, 0.16424530744552612, 0.016355320811271667, 0.00018213051953352988, 0.0032419231720268726, 0.0026226795744150877, 0.13731375336647034, 0.10789713263511658, 0.00021709683642257005, 0.33552315831184387, 0.24119459092617035, 0.11017664521932602, 0.009443712420761585, 0.05785631015896797, 0.0003256452619098127, 0.19656413793563843, 0.8021941781044006, 0.9974846243858337, 0.8679410219192505, 0.13196763396263123, 0.995816707611084, 0.1139761209487915, 0.11361042410135269, 0.003413188736885786, 0.6244916319847107, 0.056683313101530075, 0.08776770532131195, 0.2769630253314972, 0.314736932516098, 0.07863281667232513, 0.17611831426620483, 0.13745318353176117, 0.014739368110895157, 0.0008912176126614213, 0.00041133121703751385, 0.9985088109970093, 0.18232131004333496, 0.0880785882472992, 0.06931222975254059, 0.3188910484313965, 0.17286963760852814, 0.14917197823524475, 0.01917729713022709, 0.36969536542892456, 0.4064616858959198, 0.015057557262480259, 0.004595788195729256, 0.012632643803954124, 0.06942180544137955, 0.052609074860811234, 0.004665071610361338, 0.06461817026138306, 0.00020784972002729774, 0.05087787285447121, 0.11124788969755173, 0.837586522102356, 0.02582237683236599, 0.9738724827766418, 0.08662240207195282, 0.26500582695007324, 0.647465705871582, 0.9990971684455872, 0.005659446585923433, 0.5522738099098206, 0.10172304511070251, 0.00029399723280221224, 0.046855807304382324, 0.1916494518518448, 0.028444232419133186, 0.07309506088495255, 0.9944738745689392, 0.9970408082008362, 0.9975978136062622, 0.8260751962661743, 0.06935332715511322, 0.05445520579814911, 0.019521677866578102, 0.03082370012998581, 0.8238856792449951, 0.11979681998491287, 0.056507933884859085, 0.337417334318161, 0.18324407935142517, 0.1008223444223404, 0.32478731870651245, 0.0010887942044064403, 0.03919659182429314, 0.013392169028520584, 0.2634710967540741, 0.25484058260917664, 0.41642341017723083, 0.013425279408693314, 0.05178321897983551, 0.9984967708587646, 0.9940202832221985, 0.09145206958055496, 0.9073007702827454, 0.04610784351825714, 0.29331254959106445, 0.659953236579895, 0.08173388987779617, 0.9175786375999451, 0.15518221259117126, 0.09725379943847656, 0.03438027948141098, 0.1653079092502594, 0.4073827564716339, 0.11232461035251617, 0.0028257763478904963, 0.013186955824494362, 0.012480512261390686, 0.9961825013160706, 0.9978509545326233, 0.005494296550750732, 0.14608365297317505, 0.5604182481765747, 0.03975285217165947, 0.037490494549274445, 0.14673003554344177, 0.05623574182391167, 0.008079848252236843, 0.3090744912624359, 0.20955431461334229, 0.08429303765296936, 0.05238857865333557, 0.03299211338162422, 0.027916405349969864, 0.03009170852601528, 0.034986142069101334, 0.024109622463583946, 0.19450844824314117, 0.9970496296882629, 0.03063090518116951, 0.9689038991928101, 0.9976735711097717, 0.09727933257818222, 0.07741060853004456, 0.253132700920105, 0.006192849017679691, 0.5640653371810913, 0.0015482122544199228, 0.9976252913475037, 0.9982370138168335, 0.9975764155387878, 0.2199779599905014, 0.1112399473786354, 0.2297932505607605, 0.340070903301239, 0.08198653906583786, 0.014049336314201355, 0.002886849921196699, 0.9985034465789795, 0.09670424461364746, 0.13817548751831055, 0.05374523997306824, 0.5964047908782959, 0.0608120895922184, 0.018968908116221428, 0.035148270428180695, 0.999252200126648, 0.9989422559738159, 0.9033486247062683, 0.06788478046655655, 0.028597131371498108, 0.1404104232788086, 0.0014738672180101275, 0.8580855131149292, 0.02553166262805462, 0.08661125600337982, 0.3888668715953827, 0.0504741333425045, 0.44857168197631836, 0.06594191491603851, 0.002637676429003477, 0.9310997724533081, 0.7510441541671753, 0.24879933893680573, 0.9254072308540344, 0.07382891327142715, 0.9864524602890015, 0.013308506458997726, 0.997511625289917, 0.072468601167202, 0.14073613286018372, 0.6396141648292542, 0.01050269603729248, 0.007701977156102657, 0.017154403030872345, 0.10992822051048279, 0.0017504493007436395, 0.9979313015937805, 0.9078635573387146, 0.0016688668401911855, 0.010013201273977757, 0.07843673974275589, 0.013367651030421257, 0.985386848449707, 0.9974194169044495, 0.9964780807495117, 0.9712634682655334, 0.027402417734265327, 0.9884228110313416, 0.010269328020513058, 0.9961854219436646, 0.9973797798156738, 0.030584191903471947, 0.0451480932533741, 0.9248077273368835, 0.0030818935483694077, 0.03150380030274391, 0.06917139142751694, 0.06026814132928848, 0.03184623271226883, 0.8040317893028259, 0.2807330787181854, 0.7169702053070068, 0.0021307715214788914, 0.023793615400791168, 0.08381035178899765, 0.0071025718934834, 0.262440025806427, 0.11754756420850754, 0.4961146414279938, 0.0071025718934834, 0.3513145446777344, 0.6483111381530762, 0.9974492192268372, 0.9977177977561951, 0.9985169172286987, 0.10648516565561295, 0.8928371667861938, 0.1270541101694107, 0.3147153854370117, 0.11916254460811615, 0.17029985785484314, 0.057924047112464905, 0.054925255477428436, 0.15593720972537994, 0.009941505268216133, 0.005193323828279972, 0.05089457333087921, 0.06721644848585129, 0.3475075662136078, 0.08991869539022446, 0.37451282143592834, 0.054752469062805176, 0.041043564677238464, 0.043457891792058945, 0.1339951753616333, 0.018107455223798752, 0.7641346454620361, 0.07020161300897598, 0.9289469122886658, 0.997865617275238, 0.9989351034164429, 0.9990922212600708, 0.07319444417953491, 0.9252882599830627, 0.9976275563240051, 0.7792607545852661, 0.0006054862169548869, 0.13623440265655518, 0.07891503721475601, 0.005045718513429165, 0.0691508799791336, 0.03238017484545708, 0.06393712759017944, 0.5581464171409607, 0.2763291299343109, 0.07035059481859207, 0.9293556213378906, 0.03901106119155884, 0.029258297756314278, 0.9306924939155579, 0.41950029134750366, 0.5618546009063721, 0.01823914237320423, 0.9981078505516052, 0.22003087401390076, 0.1505899429321289, 0.19378866255283356, 0.4247201681137085, 0.010496886447072029, 0.36324623227119446, 0.40266233682632446, 0.12299583852291107, 0.03196346014738083, 0.07750725001096725, 0.0016009332612156868, 5.520459671970457e-05, 0.0013272151118144393, 0.996738612651825, 0.9974057078361511, 0.9982349276542664, 0.9964446425437927, 0.9959922432899475, 0.9977461695671082, 0.1946612000465393, 0.32511359453201294, 0.05742618441581726, 0.05222617834806442, 0.054713137447834015, 0.11112193018198013, 0.06816533207893372, 0.12220020592212677, 0.014469590038061142, 0.17351576685905457, 0.0019050220726057887, 0.2782919704914093, 0.11636509746313095, 0.1196988895535469, 0.11731760948896408, 0.19081971049308777, 0.0019050220726057887, 0.00015875183453317732, 0.9941588044166565, 0.06549345701932907, 0.9345412254333496, 0.5928054451942444, 0.19133049249649048, 0.08328762650489807, 0.06128297001123428, 0.037848010659217834, 0.016613515093922615, 0.016613515093922615, 0.09175596386194229, 0.36108940839767456, 0.5468838214874268, 0.9968103766441345, 0.9937472939491272, 0.995652437210083, 0.9938338994979858, 0.9982496500015259, 0.994484543800354, 0.09540674090385437, 0.1080087199807167, 0.5449347496032715, 0.24893218278884888, 0.0005178894498385489, 0.002244187518954277, 0.3803086578845978, 0.3676129877567291, 0.09959372133016586, 0.010588206350803375, 0.10741166025400162, 0.0167915690690279, 0.009364528581500053, 0.00023793721629772335, 0.005846457555890083, 0.002243408001959324, 0.10822747647762299, 0.7999719381332397, 0.05946815386414528, 0.0013670839834958315, 0.02779737301170826, 0.0018227786058560014, 0.0011392366141080856, 0.21946488320827484, 0.2801966667175293, 0.1119125559926033, 0.1434723138809204, 0.05834402143955231, 0.12603139877319336, 0.05668298155069351, 0.0018686697585508227, 0.002076299861073494, 5.190749288885854e-05, 0.026821283623576164, 0.6152132153511047, 0.3240106999874115, 0.029694993048906326, 0.004071088042110205, 0.9970265626907349, 0.9972915053367615, 0.9941739439964294, 0.32928967475891113, 0.20782236754894257, 0.13947153091430664, 0.013360590673983097, 0.16871818900108337, 0.0276173185557127, 0.1006932333111763, 0.0032586806919425726, 0.005784158129245043, 0.00391041673719883, 0.3763718903064728, 0.05189302936196327, 0.06265493482351303, 0.2773033678531647, 0.04363732039928436, 0.15803787112236023, 0.028452713042497635, 0.0016216571675613523, 0.9962917566299438, 0.2787611186504364, 0.46268847584724426, 0.09717832505702972, 0.0030721931252628565, 0.04292985796928406, 0.09014461934566498, 0.02279890701174736, 0.002344568492844701, 0.006790580693632364, 0.41400811076164246, 0.24831795692443848, 0.24831795692443848, 0.06953554600477219, 0.013037914410233498, 0.8536102175712585, 0.14012235403060913, 0.006059344857931137, 0.9979073405265808, 0.9982705116271973, 0.9982916116714478, 0.06694231182336807, 0.8690959215164185, 0.06347978115081787, 0.9956237077713013, 0.4200472831726074, 0.31152716279029846, 0.08152997493743896, 0.0005165574257262051, 0.022513294592499733, 0.010288101620972157, 0.07360943406820297, 0.07752665877342224, 0.0022814618423581123, 0.00021523225586861372, 0.4911147952079773, 0.21737040579319, 0.1467660665512085, 0.05515305697917938, 0.056331899017095566, 0.00799929816275835, 0.01768265850841999, 0.0025681958068162203, 0.004715375602245331, 0.0002947109751403332, 0.018564393743872643, 0.0140620656311512, 0.23998025059700012, 0.34353381395339966, 0.16115868091583252, 0.10287511348724365, 0.058345239609479904, 0.016035689041018486, 0.04545501247048378, 0.19423632323741913, 0.30269289016723633, 0.3369343876838684, 5.028121086070314e-05, 0.005732058081775904, 0.03826400265097618, 0.0720529779791832, 0.04691237211227417, 0.003117435146123171, 5.028121086070314e-05, 0.9928684830665588, 0.9991934299468994, 0.9955063462257385, 0.9997130632400513, 0.9981895685195923, 0.9993364810943604, 0.9987929463386536, 0.0006468594656325877, 0.01811206340789795, 0.024580659344792366, 0.2251070886850357, 0.7044299244880676, 0.027168096974492073, 0.9972354769706726, 0.011016287840902805, 0.7147626876831055, 0.047305237501859665, 0.05767350643873215, 0.15098793804645538, 0.01749645732343197, 0.03023955225944519, 0.9676656723022461, 0.06583607196807861, 0.0626632496714592, 0.03648746386170387, 0.09835750609636307, 0.05473119392991066, 0.6821569204330444, 0.13544470071792603, 0.07495483756065369, 0.7889982461929321, 0.18045584857463837, 0.1915639340877533, 0.3956497013568878, 0.008886465802788734, 0.006664849352091551, 0.11542307585477829, 0.07775657624006271, 0.020600443705916405, 0.003029477084055543, 0.0393935963511467, 0.8880442380905151, 0.07090847939252853, 0.029602525755763054, 0.020831406116485596, 0.7488342523574829, 0.0608496367931366, 0.09538591653108597, 0.0356326699256897, 0.008771118707954884, 0.9970695376396179, 0.9983066320419312, 0.9943600296974182, 0.8700689077377319, 0.07717038691043854, 0.03695483133196831, 0.0157601498067379, 0.08530812710523605, 0.08354008197784424, 0.8309807181358337, 0.27322113513946533, 0.7259876132011414, 0.9978927969932556, 0.06608825922012329, 0.03811601921916008, 0.0470302514731884, 0.00030738726491108537, 0.2379177361726761, 0.12418445199728012, 0.03350520879030228, 0.43925637006759644, 0.013525038957595825, 0.9975720643997192, 0.9988674521446228, 0.26570987701416016, 0.7335203289985657, 0.8027026653289795, 0.0839114859700203, 0.07971007376909256, 0.033611275255680084, 0.0001167058217106387, 0.9977887868881226, 0.16503851115703583, 0.8343613743782043, 0.6735571622848511, 0.0637921541929245, 0.24490320682525635, 0.017597835510969162, 0.9059721827507019, 0.09397052228450775, 0.9086153507232666, 0.09063773602247238, 0.11753259599208832, 0.06296388804912567, 0.6917632818222046, 0.12676729261875153, 0.033992525190114975, 0.03981981426477432, 0.8478706479072571, 0.07769719511270523, 0.9976951479911804, 0.009132787585258484, 0.10108335316181183, 0.8896165490150452, 0.9979749321937561, 0.36318299174308777, 0.5248003602027893, 0.040304239839315414, 0.049067962914705276, 0.013123344630002975, 0.007429147604852915, 0.0015570069663226604, 0.000533830956555903, 0.05079391226172447, 0.07875621318817139, 0.08260423690080643, 0.1364765763282776, 0.0192401185631752, 0.28552335500717163, 0.017957443371415138, 0.30194157361984253, 0.026423094794154167, 0.998171329498291, 0.3652794063091278, 0.27219241857528687, 0.18385297060012817, 0.007701568771153688, 0.024300383403897285, 0.06906794756650925, 0.057498011738061905, 0.015965808182954788, 0.0033408631570637226, 0.0008440075325779617, 0.08713861554861069, 0.8862249851226807, 0.02652044966816902, 0.9993206858634949, 0.999177873134613, 0.9993413686752319, 0.9977916479110718, 0.19790393114089966, 0.3020404279232025, 0.12409412860870361, 0.0002229903475381434, 0.003790835849940777, 0.33192113041877747, 0.04013826325535774, 0.9994001388549805, 0.995078980922699, 0.9954798221588135, 0.9967396259307861, 0.010687797330319881, 0.08692742139101028, 0.009975277818739414, 0.1296786069869995, 0.7303327918052673, 0.020663075149059296, 0.011400316841900349, 0.14998751878738403, 0.1032848134636879, 0.012573803775012493, 0.6704531908035278, 0.06376714259386063, 0.8869820833206177, 0.11223038285970688, 0.9978309273719788, 0.00718745868653059, 0.8559065461158752, 0.024557150900363922, 0.05330698564648628, 0.05270802974700928, 0.005989548750221729, 0.9981862902641296, 0.998465895652771, 0.3252314627170563, 0.2791479229927063, 0.18391898274421692, 0.08059429377317429, 0.031397007405757904, 0.02568845823407173, 0.056618403643369675, 0.014894116669893265, 0.00217962684109807, 0.00036327113048173487, 0.03407720848917961, 0.2841895520687103, 0.17677052319049835, 0.22856469452381134, 0.057141173630952835, 0.06528139859437943, 0.05961516126990318, 0.06033341959118843, 0.032401278614997864, 0.0015163160860538483, 0.996392011642456, 0.09253299981355667, 0.3406538665294647, 0.23078078031539917, 0.05217789486050606, 0.2682984173297882, 0.0001576371432747692, 0.015448439866304398, 0.9972083568572998, 0.9987277984619141, 0.9618828892707825, 0.037647079676389694, 0.9975969195365906, 0.9981254935264587, 0.9963899850845337, 0.9973117709159851, 0.024636464193463326, 0.06528662890195847, 0.9090855121612549, 0.7093136310577393, 0.11020075529813766, 0.07117625325918198, 0.02630598656833172, 0.05371792986989021, 0.028596896678209305, 7.899695629021153e-05, 0.0006319756503216922, 0.08594844490289688, 0.1722722053527832, 0.027023091912269592, 0.004503848496824503, 0.09608210623264313, 0.5295775532722473, 0.08482248336076736, 0.9967882633209229, 0.994729220867157, 0.07959402352571487, 0.36520618200302124, 0.008062325417995453, 0.30310913920402527, 0.08113787323236465, 0.06758630275726318, 0.05514973774552345, 0.03422199934720993, 0.0008576941909268498, 0.0050603956915438175, 0.9954546689987183, 0.2554955780506134, 0.27241095900535583, 0.1294998824596405, 0.04503895342350006, 0.08511795848608017, 0.08234810829162598, 0.0875270739197731, 0.02776285633444786, 0.014274359680712223, 0.0005410858429968357, 0.9961841702461243, 0.997356653213501, 0.9964677095413208, 0.9984257221221924, 0.9956757426261902, 0.996343731880188, 0.9971169233322144, 0.2254052460193634, 0.07329611480236053, 0.7010418772697449, 0.05731363222002983, 0.9412661790847778, 0.9701336622238159, 0.02909739688038826, 0.001452373806387186, 0.09634079039096832, 0.027595100924372673, 0.20139582455158234, 0.06826156377792358, 0.023722104728221893, 0.5794971585273743, 0.0019364983309060335, 0.034453339874744415, 0.9646934866905212, 0.27510711550712585, 0.26049870252609253, 0.3796384334564209, 0.08462059497833252, 0.00014428063877858222, 0.06078868359327316, 0.00012109299132134765, 0.09505800157785416, 0.014773344621062279, 0.06236289069056511, 0.4861883521080017, 0.001210929942317307, 0.2794826328754425, 0.07967941462993622, 0.013371488079428673, 0.1751115471124649, 0.0005495132063515484, 0.1190611943602562, 0.45151668787002563, 0.0018317106878384948, 0.15880931913852692, 0.009264115244150162, 0.10705199837684631, 0.736497163772583, 0.015954865142703056, 0.04940861463546753, 0.03139505907893181, 0.025218980386853218, 0.02418963424861431, 0.9951235055923462, 0.9979557394981384, 0.9961913228034973, 0.0684480294585228, 0.8859681487083435, 0.01735302247107029, 0.026993589475750923, 0.05513196438550949, 0.018076054751873016, 0.858612596988678, 0.06688140332698822, 0.1424170285463333, 0.21186256408691406, 0.0850822925567627, 0.09749970585107803, 0.01349052507430315, 0.00045990425860509276, 0.38248705863952637, 0.00199291855096817, 0.06469319760799408, 0.03397831320762634, 0.1851627230644226, 0.018961681053042412, 0.2327578216791153, 0.005853941664099693, 0.04874042794108391, 0.2157050222158432, 0.049376726150512695, 0.20361536741256714, 0.005726682022213936, 0.019186493009328842, 0.012449251487851143, 0.0008787707192823291, 0.2551364302635193, 0.0013181560207158327, 0.14543655514717102, 0.2857469320297241, 0.002489850390702486, 0.24781332910060883, 0.029585279524326324, 0.6314432621002197, 0.10840744525194168, 0.00235554575920105, 0.054177552461624146, 0.16195684671401978, 0.04145760461688042, 0.00020938183297403157, 0.9725214242935181, 0.026388736441731453, 0.17752309143543243, 0.13531272113323212, 0.6301621198654175, 0.008681568317115307, 0.0481976717710495, 0.4828929305076599, 0.5166244506835938, 0.7974284291267395, 0.00502430135384202, 0.013457950204610825, 0.10263930261135101, 0.052216846495866776, 0.029248612001538277, 0.99128258228302, 0.007882962934672832, 0.9959645867347717, 0.9992781281471252, 0.7842493057250977, 0.012269231490790844, 0.0323907695710659, 0.003926154226064682, 0.0991353914141655, 0.06772615760564804, 0.9979575276374817, 0.1992708146572113, 0.7999573349952698, 0.9967108368873596, 0.9958821535110474, 0.9962896108627319, 0.9953566193580627, 0.9972841143608093, 0.997907280921936, 0.9977859258651733, 0.23739463090896606, 0.19354984164237976, 0.1641693264245987, 0.005062490236014128, 0.007684136740863323, 0.21976631879806519, 0.15142269432544708, 0.010757791809737682, 0.01003457885235548, 0.9953923225402832, 0.2230309396982193, 0.22409671545028687, 0.3615815341472626, 0.07971988618373871, 0.0341758169233799, 0.02941535785794258, 0.023589128628373146, 0.01534714363515377, 0.008668294176459312, 0.0004263095324859023, 0.06627025455236435, 0.14828427135944366, 0.7439842820167542, 0.04100700840353966, 0.4039587080478668, 0.4620499014854431, 0.05734202265739441, 0.004005297087132931, 0.015646591782569885, 0.05667927488684654, 0.00034578103804960847, 0.9993939399719238, 0.9966180920600891, 0.8853011727333069, 0.11345042288303375, 0.9981173872947693, 0.9179916977882385, 0.03739011287689209, 0.023207655176520348, 0.020629027858376503, 0.07259265333414078, 0.7857087254524231, 0.059568677097558975, 0.01195643749088049, 0.03266669437289238, 0.004270156379789114, 0.033093709498643875, 0.9688407778739929, 0.030067473649978638, 0.9943757653236389, 0.9984031319618225, 0.07532090693712234, 0.001260600984096527, 0.11282378435134888, 0.37124696373939514, 0.1140843853354454, 0.3252350389957428, 0.12523959577083588, 0.06189166009426117, 0.8118729591369629, 0.04098626226186752, 0.9554922580718994, 0.08615804463624954, 0.913572371006012, 0.998070478439331, 0.10239151865243912, 0.0065356288105249405, 0.10293615609407425, 0.0005446357536129653, 0.044115494936704636, 0.7434277534484863, 0.9956368207931519, 0.012164514511823654, 0.021234547719359398, 0.03852096199989319, 0.07021272927522659, 0.058795154094696045, 0.4682271182537079, 0.01974065974354744, 0.3098750114440918, 0.0012804752914234996, 0.061104800552129745, 0.07615070790052414, 0.060490682721138, 0.034697700291872025, 0.5333619713783264, 0.006141185760498047, 0.22753094136714935, 0.0006141185876913369, 0.9947831034660339, 0.9984436631202698, 0.9343322515487671, 0.0650649219751358, 0.9982420206069946, 0.032892804592847824, 0.0524229072034359, 0.9138032793998718, 0.9953508973121643, 0.9959026575088501, 0.999444842338562, 0.19249264895915985, 0.7209423780441284, 0.06919805705547333, 0.01727365516126156, 0.47554153203964233, 0.3944227993488312, 0.11476125568151474, 0.013119283132255077, 0.002143249148502946, 0.9982677102088928, 0.1817224770784378, 0.31051281094551086, 0.007080096285790205, 0.5003268122673035, 0.3328847885131836, 0.06894681602716446, 0.07220076024532318, 0.09215420484542847, 0.18670281767845154, 0.10228440910577774, 0.11996623128652573, 0.00018418561376165599, 0.023944130167365074, 0.0007367424550466239, 0.4223262071609497, 0.09738407284021378, 0.09994681179523468, 0.00013142250827513635, 0.07293949276208878, 0.1545528620481491, 0.10724076628684998, 0.014193630777299404, 0.022801805287599564, 0.00847675185650587, 0.23197339475154877, 0.19077348709106445, 0.03448253124952316, 0.2729493975639343, 0.26959070563316345, 0.21314600110054016, 0.7866232395172119, 0.013217751868069172, 0.1627870500087738, 0.8236746191978455, 0.9553657174110413, 0.044410426169633865, 0.998818039894104, 0.15011873841285706, 0.055223383009433746, 0.0028563819359987974, 0.25358325242996216, 0.48971080780029297, 0.04887586832046509, 0.05280629172921181, 0.3448707163333893, 0.02477332204580307, 0.21513675153255463, 0.36182090640068054, 0.2782420516014099, 0.08239348232746124, 0.1429125964641571, 0.029603322967886925, 0.008749750442802906, 0.37434348464012146, 0.026103422045707703, 0.057310864329338074, 0.749914824962616, 0.019099706783890724, 0.2031717449426651, 0.00067016517277807, 0.027029994875192642, 0.998589277267456, 0.9994997978210449, 0.9807776808738708, 0.01604544185101986, 0.26190006732940674, 0.4727121591567993, 0.18132683634757996, 0.015972407534718513, 0.04871732369065285, 0.004978412762284279, 0.01137922890484333, 0.0010668027680367231, 0.0018372713821008801, 8.890022581908852e-05, 0.40171924233436584, 0.44634315371513367, 0.10299425572156906, 0.009216371923685074, 0.018432743847370148, 0.01280919462442398, 0.008539462462067604, 0.9977651238441467, 0.997300386428833, 0.9988915920257568, 0.7504672408103943, 0.09255723655223846, 0.15555737912654877, 0.001295330235734582, 0.12763752043247223, 0.10484509915113449, 0.09952687472105026, 0.05485374107956886, 0.13493108749389648, 0.1612183302640915, 0.1344752460718155, 0.1765652298927307, 0.0001519494253443554, 0.005622128490358591, 0.9980806112289429, 0.9993219971656799, 0.9955486059188843, 0.9979576468467712, 0.03343170881271362, 0.9667335748672485, 0.0127206239849329, 0.8634123802185059, 0.07090570032596588, 0.018727585673332214, 0.0001766753412084654, 0.010718303732573986, 0.0017078615492209792, 0.021377716213464737, 0.0002944589068647474, 0.049787528812885284, 0.04779602587223053, 0.04912369325757027, 0.6698082089424133, 0.1832181066274643, 0.9971489310264587, 0.03274821490049362, 0.8074129223823547, 0.07565966993570328, 0.08356441557407379, 0.18113355338573456, 0.01430001761764288, 0.8039787411689758, 0.13673308491706848, 0.7669144868850708, 0.08882999420166016, 0.007441256195306778, 0.9988790154457092, 0.08255569636821747, 0.024486860260367393, 0.2665569484233856, 0.000699624593835324, 0.030783481895923615, 0.5715932846069336, 0.02308761142194271, 0.9985051155090332, 0.9217597842216492, 0.06447090953588486, 0.01335883792489767, 0.9994857907295227, 0.18643693625926971, 0.8131647109985352, 0.4743339717388153, 0.2548689544200897, 0.17530877888202667, 0.006712273228913546, 0.04836785048246384, 0.04040525481104851, 0.008754383772611618, 0.05046645179390907, 0.8381035327911377, 0.008239420130848885, 0.035532500594854355, 0.05870587006211281, 0.3556840419769287, 0.6039846539497375, 0.035040903836488724, 0.005274975206702948, 0.29241979122161865, 0.7054571509361267, 0.001780333579517901, 0.9962673783302307, 0.9962332844734192, 0.01622537150979042, 0.10880543291568756, 0.04867611452937126, 0.7917026877403259, 0.034359607845544815, 0.9972187876701355, 0.22207753360271454, 0.3137624263763428, 0.025422465056180954, 0.3923260271549225, 0.013285288587212563, 0.015581510961055756, 0.01508946344256401, 0.0022962226066738367, 0.9970192909240723, 0.9973740577697754, 0.10799798369407654, 0.060163259506225586, 0.022684507071971893, 0.8023411631584167, 0.005917697679251432, 0.9964196085929871, 0.9978742599487305, 0.09580057114362717, 0.1871795803308487, 0.003131941892206669, 0.14388509094715118, 0.3275642693042755, 0.13172578811645508, 0.08658897876739502, 0.024134375154972076, 0.008465289138257504, 0.13920697569847107, 0.06207878515124321, 0.05361349508166313, 0.7355395555496216, 0.99263596534729, 0.04766214266419411, 0.002344039734452963, 0.09532428532838821, 0.025003090500831604, 0.8290086984634399, 0.13533970713615417, 0.4446876049041748, 0.11158259958028793, 0.021419309079647064, 0.04523960500955582, 0.10431645810604095, 0.07651558518409729, 0.05339031293988228, 0.006760667078197002, 0.0006318380474112928, 0.22585834562778473, 0.2361774891614914, 0.3518099784851074, 0.03886391222476959, 0.07918122410774231, 0.039939429610967636, 0.026800693944096565, 0.0013371278764680028, 0.9964156150817871, 0.9940781593322754, 0.7481405735015869, 0.14054138958454132, 0.11126193404197693, 0.0980532169342041, 0.8467247486114502, 0.05471477657556534, 0.9947907328605652, 0.9984283447265625, 0.01940285973250866, 0.003167813876643777, 0.01940285973250866, 0.013859185390174389, 0.618911623954773, 0.32509690523147583, 0.8025241494178772, 0.060096852481365204, 0.012481654062867165, 0.026119017973542213, 0.09777295589447021, 0.00023114174837246537, 0.00046228349674493074, 0.16954436898231506, 0.0628647655248642, 0.7672358751296997, 0.21995055675506592, 0.15786300599575043, 0.2232801765203476, 0.1487555205821991, 0.1956639438867569, 0.03437339514493942, 0.01263295765966177, 0.0032316867727786303, 0.0031337568070739508, 0.0008813691092655063, 0.995247483253479, 0.996242105960846, 0.9965136051177979, 0.9971861839294434, 0.9428964853286743, 0.016287194564938545, 0.04049788787961006, 0.03438691422343254, 0.8292158842086792, 0.10610819607973099, 0.029474498704075813, 0.538017213344574, 0.3085044026374817, 0.0364808551967144, 0.005728398449718952, 0.10793508589267731, 0.0033164413180202246, 0.3413969874382019, 0.49602946639060974, 0.14979131519794464, 0.01271992176771164, 0.3837529420852661, 0.5302289128303528, 0.08422116190195084, 0.0018220925703644753, 0.004177351016551256, 0.9942095279693604, 0.07332568615674973, 0.9262635707855225, 0.9978102445602417, 0.3167094588279724, 0.3272664248943329, 0.11794305592775345, 0.04390228912234306, 0.05738230422139168, 0.01563682034611702, 0.04693884029984474, 0.0675135999917984, 0.0065555451437830925, 0.0001702738954918459, 0.9964033961296082, 0.09962393343448639, 0.13767771422863007, 0.061142586171627045, 0.03591592609882355, 0.017957963049411774, 0.647341787815094, 0.20023755729198456, 0.7992240786552429, 0.0005221798201091588, 0.6226994395256042, 0.1543041467666626, 0.1300227791070938, 0.017493024468421936, 0.03603040799498558, 0.038902398198843, 0.9961034059524536, 0.9963122606277466, 0.9978416562080383, 0.9957295656204224, 0.9966611862182617, 0.9978944063186646, 0.06167495623230934, 0.26466184854507446, 0.534092366695404, 0.01716725528240204, 0.020505333319306374, 0.040374841541051865, 0.047209952026605606, 0.014147089794278145, 0.11501441895961761, 0.7733585834503174, 0.00013741268776357174, 0.0037101423367857933, 0.0072828722186386585, 0.000549650751054287, 0.0864325761795044, 0.013329030014574528, 0.00013741268776357174, 0.20323604345321655, 0.12256865203380585, 0.16292722523212433, 0.17327812314033508, 0.09251121431589127, 0.08161290735006332, 0.06573820114135742, 0.05479012802243233, 0.041403621435165405, 0.0018910306971520185, 0.06118692085146904, 0.025698507204651833, 0.044054582715034485, 0.868854284286499, 0.07105373591184616, 0.9282826781272888, 0.9538115859031677, 0.006596207153052092, 0.03430027887225151, 0.0039577241986989975, 0.9978343844413757, 0.049191564321517944, 0.9501739144325256, 0.9984650015830994, 0.9946287870407104, 0.9974197745323181, 0.1072397530078888, 0.13335752487182617, 0.6059691905975342, 0.0007383115589618683, 0.06469454616308212, 0.007013959810137749, 0.0026763794012367725, 0.07835331559181213, 0.998881459236145, 0.05283081904053688, 0.3701891005039215, 0.01866813376545906, 0.005973802879452705, 0.35096094012260437, 0.20105580985546112, 0.00018668133998289704, 0.9961950778961182, 0.07764288038015366, 0.16905494034290314, 0.6754547953605652, 0.0007649544859305024, 0.0022948633413761854, 0.009179453365504742, 0.06540361046791077, 0.20392777025699615, 0.6425542831420898, 0.1407073587179184, 0.004196044523268938, 0.008112353272736073, 0.04367569461464882, 0.04220016673207283, 0.33907684683799744, 0.023313378915190697, 0.5512580871582031, 0.9975243806838989, 0.9982602596282959, 0.9954618215560913, 0.9924083948135376, 0.0035317023284733295, 0.05683274194598198, 0.05022428557276726, 0.11710188537836075, 0.6026914119720459, 0.11128643900156021, 0.020882729440927505, 0.03885773569345474, 0.001850368338637054, 0.9998725056648254, 0.0396822988986969, 0.9592169523239136, 0.9990123510360718, 0.0009215981117449701, 0.9657864570617676, 0.03361734375357628, 0.9788297414779663, 0.0022605767007917166, 0.018084613606333733, 0.3900105655193329, 0.541521430015564, 0.05841322988271713, 0.0004038944316562265, 0.005351601168513298, 0.0016660644905641675, 0.002574827056378126, 0.28710949420928955, 0.6949910521507263, 0.017816239967942238, 0.00015766583965159953, 0.48854687809944153, 0.4532448351383209, 0.05446816608309746, 0.0037120168562978506, 0.0265804436057806, 0.09643137454986572, 0.15144671499729156, 0.6861463189125061, 0.03894343972206116, 0.9987747073173523, 0.29949504137039185, 0.23327739536762238, 0.2582104206085205, 0.07279245555400848, 0.06655919551849365, 0.04798751696944237, 0.013448459096252918, 0.0008111768984235823, 0.007385978940874338, 4.269352211849764e-05, 0.8769306540489197, 0.01370204146951437, 0.1084744930267334, 0.8987632989883423, 0.05936785042285919, 0.04122767597436905, 0.17186696827411652, 0.3466617166996002, 0.1814170777797699, 0.007458838168531656, 0.026419622823596, 0.1027158722281456, 0.11561199277639389, 0.024363214150071144, 0.023491855710744858, 0.9976253509521484, 0.9976245164871216, 0.9128895998001099, 0.0866784080862999, 0.9168955683708191, 0.08183246850967407, 0.9968851804733276, 0.0023264531046152115, 0.03179958835244179, 0.0009636239265091717, 0.0070665753446519375, 0.795310914516449, 0.10599862784147263, 0.015257378108799458, 0.004818119574338198, 0.03870556131005287, 0.0016594456974416971, 0.9956674575805664, 0.2986246347427368, 0.13611631095409393, 0.39286699891090393, 0.04519892856478691, 0.05558948591351509, 0.04021146148443222, 0.023378755897283554, 0.007896823808550835, 0.00010390557872597128, 0.04831131547689438, 0.09243303537368774, 0.6973064541816711, 0.036789920181035995, 0.12516428530216217, 0.12020137906074524, 0.2168891876935959, 0.5968685150146484, 0.054551489651203156, 0.01109840627759695, 0.00018810857727658004, 0.3610723614692688, 0.21006207168102264, 0.020735740661621094, 0.3887951672077179, 0.00022538848861586303, 0.019158022478222847, 0.9987177848815918, 0.9993604421615601, 0.3152351379394531, 0.10276896506547928, 0.2943679094314575, 0.04470367357134819, 0.00016495821182616055, 0.06977731734514236, 0.13493581116199493, 0.013279135338962078, 0.024743730202317238, 0.004393553826957941, 0.1265164166688919, 0.5864945650100708, 0.005379861686378717, 0.12804070115089417, 0.0008966436143964529, 0.14489760994911194, 0.003407245734706521, 0.9296275973320007, 0.017255833372473717, 0.05273965746164322, 0.9676737189292908, 0.0035098320804536343, 0.02859863080084324, 0.44172757863998413, 0.31860509514808655, 0.13818080723285675, 0.013930830173194408, 0.03565390408039093, 7.516634650528431e-05, 0.009972069412469864, 0.04149182513356209, 0.0001753881515469402, 0.00022549905406776816, 0.997059166431427, 0.01316716056317091, 0.46521416306495667, 0.06353920698165894, 0.15747006237506866, 0.004516642540693283, 0.02794194035232067, 0.05473558232188225, 0.21174632012844086, 0.001224852167069912, 0.0003827663022093475, 0.16130907833576202, 0.1665959656238556, 0.41237789392471313, 0.023732289671897888, 0.04946184903383255, 0.08846739679574966, 0.06356023997068405, 0.018797852098941803, 0.0149207953363657, 0.0008224060875363648, 0.06949838250875473, 0.1752115786075592, 0.5497463941574097, 0.09862151741981506, 0.09209717810153961, 0.013237787410616875, 0.0016074456507340074, 0.9953583478927612, 0.9991516470909119, 0.9986919164657593], \"Term\": [\"1/10\", \"13th\", \"13th\", \"19th\", \"9/11\", \"abbott\", \"abc\", \"abysmal\", \"accessible\", \"acting\", \"acting\", \"acting\", \"acting\", \"acting\", \"acting\", \"acting\", \"action\", \"action\", \"action\", \"action\", \"action\", \"action\", \"action\", \"action\", \"action\", \"actor\", \"actor\", \"actor\", \"actor\", \"actor\", \"actor\", \"actor\", \"actor\", \"actors\", \"actors\", \"actors\", \"actors\", \"actors\", \"actors\", \"actually\", \"actually\", \"actually\", \"actually\", \"actually\", \"actually\", \"actually\", \"actually\", \"actually\", \"addiction\", \"aids\", \"aircraft\", \"aired\", \"aired\", \"akira\", \"al\", \"al\", \"al\", \"al\", \"al\", \"aladdin\", \"alien\", \"aliens\", \"allen\", \"also\", \"also\", \"also\", \"also\", \"also\", \"also\", \"also\", \"also\", \"also\", \"also\", \"amateurish\", \"america\", \"america\", \"america\", \"america\", \"america\", \"america\", \"american\", \"american\", \"american\", \"american\", \"american\", \"american\", \"american\", \"american\", \"american\", \"americans\", \"americans\", \"americans\", \"americans\", \"andre\", \"angela\", \"animated\", \"animated\", \"animated\", \"animated\", \"animation\", \"animation\", \"animation\", \"anime\", \"anime\", \"annoying\", \"annoying\", \"annoying\", \"ants\", \"anything\", \"anything\", \"anything\", \"anything\", \"anything\", \"anything\", \"appalling\", \"argento\", \"army\", \"army\", \"around\", \"around\", \"around\", \"around\", \"around\", \"around\", \"around\", \"around\", \"around\", \"around\", \"art\", \"art\", \"art\", \"art\", \"art\", \"art\", \"arthur\", \"arthur\", \"arts\", \"artsy\", \"asoka\", \"atmosphere\", \"atmosphere\", \"atmosphere\", \"atmosphere\", \"atmosphere\", \"atmosphere\", \"atmospheric\", \"attitudes\", \"attitudes\", \"audience\", \"audience\", \"audience\", \"audience\", \"audience\", \"aunt\", \"aunt\", \"aunt\", \"avery\", \"avoid\", \"avoid\", \"avoid\", \"avoid\", \"avoid\", \"avoid\", \"avoid\", \"awesome\", \"awesome\", \"awesome\", \"awesome\", \"awesome\", \"awful\", \"awful\", \"awful\", \"baby\", \"baby\", \"baby\", \"baby\", \"baby\", \"bachchan\", \"back\", \"back\", \"back\", \"back\", \"back\", \"back\", \"back\", \"back\", \"back\", \"back\", \"bad\", \"bad\", \"bad\", \"badly\", \"badly\", \"badly\", \"bambi\", \"band\", \"band\", \"band\", \"band\", \"band\", \"bands\", \"batman\", \"batman\", \"battle\", \"battle\", \"battle\", \"beatles\", \"beatles\", \"bela\", \"beliefs\", \"beowulf\", \"berry\", \"best\", \"best\", \"best\", \"best\", \"best\", \"best\", \"best\", \"best\", \"best\", \"best\", \"bette\", \"better\", \"better\", \"better\", \"better\", \"better\", \"better\", \"better\", \"better\", \"better\", \"better\", \"bible\", \"big\", \"big\", \"big\", \"big\", \"big\", \"big\", \"big\", \"big\", \"big\", \"big\", \"bit\", \"bit\", \"bit\", \"bit\", \"bit\", \"bit\", \"bit\", \"bit\", \"bit\", \"bit\", \"black\", \"black\", \"black\", \"black\", \"black\", \"black\", \"black\", \"black\", \"black\", \"black\", \"blah\", \"blood\", \"blood\", \"bogart\", \"boring\", \"boring\", \"boring\", \"boris\", \"boy\", \"boy\", \"boy\", \"boy\", \"boy\", \"boy\", \"boyfriend\", \"boyfriend\", \"boyfriend\", \"boys\", \"boys\", \"boys\", \"boys\", \"boys\", \"boys\", \"boys\", \"british\", \"british\", \"british\", \"british\", \"british\", \"british\", \"broadway\", \"brooks\", \"brother\", \"brother\", \"brother\", \"brother\", \"brother\", \"brother\", \"bruce\", \"bruce\", \"bruce\", \"bruce\", \"bruce\", \"buffy\", \"bugs\", \"bullets\", \"bunuel\", \"burt\", \"buscemi\", \"buster\", \"cagney\", \"canceled\", \"candyman\", \"car\", \"car\", \"car\", \"car\", \"car\", \"carpenter\", \"carrey\", \"cartoon\", \"cartoon\", \"cartoon\", \"cartoons\", \"cary\", \"cassavetes\", \"cast\", \"cast\", \"cast\", \"cast\", \"cast\", \"cast\", \"cast\", \"cast\", \"cave\", \"cave\", \"cells\", \"century\", \"century\", \"century\", \"chainsaw\", \"chan\", \"chaney\", \"chaplin\", \"character\", \"character\", \"character\", \"character\", \"character\", \"character\", \"character\", \"characters\", \"characters\", \"characters\", \"characters\", \"characters\", \"characters\", \"characters\", \"che\", \"che\", \"chinese\", \"chinese\", \"chinese\", \"chinese\", \"chow\", \"chris\", \"chris\", \"chris\", \"chuck\", \"chuck\", \"civil\", \"classic\", \"classic\", \"classic\", \"classic\", \"classic\", \"classic\", \"classic\", \"clint\", \"colbert\", \"columbo\", \"comedy\", \"comedy\", \"comedy\", \"comedy\", \"comedy\", \"comedy\", \"comes\", \"comes\", \"comes\", \"comes\", \"comes\", \"comes\", \"comes\", \"comes\", \"comes\", \"comes\", \"communist\", \"compassion\", \"complex\", \"complex\", \"complex\", \"complex\", \"computer\", \"computer\", \"computer\", \"computer\", \"connery\", \"contestants\", \"cop\", \"cop\", \"cop\", \"cop\", \"costello\", \"costner\", \"could\", \"could\", \"could\", \"could\", \"could\", \"could\", \"could\", \"could\", \"could\", \"could\", \"country\", \"country\", \"country\", \"country\", \"country\", \"crap\", \"crap\", \"crap\", \"crawford\", \"crawford\", \"creature\", \"creature\", \"creatures\", \"creatures\", \"creatures\", \"creepy\", \"creepy\", \"crime\", \"crime\", \"crime\", \"crime\", \"crime\", \"cronenberg\", \"crow\", \"crowe\", \"cultural\", \"culture\", \"culture\", \"culture\", \"daffy\", \"dakota\", \"damme\", \"dance\", \"dance\", \"dance\", \"daniels\", \"dark\", \"dark\", \"dark\", \"dark\", \"dark\", \"dark\", \"daughter\", \"daughter\", \"daughters\", \"david\", \"david\", \"david\", \"david\", \"david\", \"david\", \"david\", \"david\", \"davis\", \"dead\", \"dead\", \"dead\", \"dead\", \"dead\", \"death\", \"death\", \"death\", \"death\", \"death\", \"death\", \"death\", \"decides\", \"decides\", \"decides\", \"deeply\", \"deeply\", \"dennis\", \"dennis\", \"dennis\", \"dennis\", \"detective\", \"detective\", \"detective\", \"dialogue\", \"dialogue\", \"dialogue\", \"dialogue\", \"dialogue\", \"dialogue\", \"dialogue\", \"dialogue\", \"different\", \"different\", \"different\", \"different\", \"different\", \"different\", \"different\", \"different\", \"dinosaur\", \"dinosaurs\", \"director\", \"director\", \"director\", \"director\", \"director\", \"director\", \"director\", \"dirty\", \"dirty\", \"dirty\", \"dirty\", \"dirty\", \"dirty\", \"dirty\", \"disjointed\", \"disney\", \"divorce\", \"documentaries\", \"documentary\", \"documentary\", \"donnie\", \"douglas\", \"douglas\", \"douglas\", \"dracula\", \"dracula\", \"dreck\", \"drivel\", \"dunne\", \"dvd\", \"dvd\", \"dvd\", \"dvd\", \"dvd\", \"dvd\", \"earth\", \"earth\", \"earth\", \"earth\", \"earth\", \"eastwood\", \"ebay\", \"eddie\", \"eddie\", \"eerie\", \"effects\", \"effects\", \"effects\", \"effects\", \"effects\", \"elvis\", \"embarrassingly\", \"emily\", \"emily\", \"emily\", \"emmanuelle\", \"emotional\", \"emotional\", \"emotional\", \"emotionally\", \"emotionally\", \"emotions\", \"emotions\", \"emotions\", \"emotions\", \"ending\", \"ending\", \"ending\", \"ending\", \"ending\", \"ending\", \"english\", \"english\", \"english\", \"english\", \"english\", \"english\", \"english\", \"enjoyed\", \"enjoyed\", \"enjoyed\", \"enjoyed\", \"enjoyed\", \"enjoyed\", \"enterprise\", \"episode\", \"episode\", \"episode\", \"episodes\", \"episodes\", \"errol\", \"europe\", \"europe\", \"even\", \"even\", \"even\", \"even\", \"even\", \"even\", \"even\", \"even\", \"even\", \"even\", \"ever\", \"ever\", \"ever\", \"ever\", \"ever\", \"ever\", \"ever\", \"ever\", \"ever\", \"everett\", \"evil\", \"evil\", \"evil\", \"evil\", \"evil\", \"evil\", \"evil\", \"excellent\", \"excellent\", \"excellent\", \"excellent\", \"excellent\", \"excellent\", \"excuse\", \"excuse\", \"excuse\", \"excuse\", \"experience\", \"experience\", \"experience\", \"experience\", \"experience\", \"experience\", \"explores\", \"expressed\", \"fairbanks\", \"family\", \"family\", \"family\", \"family\", \"family\", \"fan\", \"fan\", \"fan\", \"fan\", \"fan\", \"fan\", \"fan\", \"father\", \"father\", \"father\", \"father\", \"favorite\", \"favorite\", \"favorite\", \"favorite\", \"favorite\", \"favorite\", \"feel\", \"feel\", \"feel\", \"feel\", \"fellini\", \"fight\", \"fight\", \"fight\", \"fight\", \"fight\", \"fight\", \"film\", \"film\", \"film\", \"film\", \"film\", \"film\", \"film\", \"film\", \"film\", \"film\", \"films\", \"films\", \"films\", \"films\", \"films\", \"films\", \"films\", \"films\", \"films\", \"finds\", \"finds\", \"finds\", \"finds\", \"fine\", \"fine\", \"fine\", \"fine\", \"fine\", \"fine\", \"fine\", \"fine\", \"fire\", \"fire\", \"fire\", \"fire\", \"fire\", \"fire\", \"fire\", \"fire\", \"fire\", \"first\", \"first\", \"first\", \"first\", \"first\", \"first\", \"first\", \"first\", \"first\", \"flynn\", \"football\", \"ford\", \"ford\", \"ford\", \"france\", \"france\", \"francisco\", \"freddy\", \"french\", \"french\", \"french\", \"french\", \"french\", \"friend\", \"friend\", \"friend\", \"friend\", \"friend\", \"friend\", \"friend\", \"fu\", \"fun\", \"fun\", \"fun\", \"fun\", \"fun\", \"fun\", \"fun\", \"fun\", \"funny\", \"funny\", \"funny\", \"funny\", \"funny\", \"funny\", \"gable\", \"game\", \"game\", \"game\", \"game\", \"game\", \"game\", \"gang\", \"gang\", \"gang\", \"gang\", \"gang\", \"gang\", \"gangster\", \"gangster\", \"garbage\", \"garbage\", \"garbage\", \"gene\", \"gene\", \"gene\", \"gene\", \"general\", \"general\", \"general\", \"general\", \"general\", \"general\", \"general\", \"genre\", \"genre\", \"genre\", \"genre\", \"genre\", \"genre\", \"genre\", \"genre\", \"german\", \"germans\", \"germany\", \"get\", \"get\", \"get\", \"get\", \"get\", \"get\", \"get\", \"get\", \"get\", \"gets\", \"gets\", \"gets\", \"gets\", \"gets\", \"gets\", \"gets\", \"gets\", \"gets\", \"ghost\", \"ghost\", \"giallo\", \"giant\", \"giant\", \"ginger\", \"girl\", \"girl\", \"girl\", \"girl\", \"girl\", \"girl\", \"go\", \"go\", \"go\", \"go\", \"go\", \"go\", \"go\", \"go\", \"godzilla\", \"goes\", \"goes\", \"goes\", \"goes\", \"goes\", \"goes\", \"goes\", \"good\", \"good\", \"good\", \"good\", \"good\", \"good\", \"good\", \"good\", \"good\", \"good\", \"gore\", \"gore\", \"gore\", \"gory\", \"gory\", \"government\", \"government\", \"government\", \"grant\", \"great\", \"great\", \"great\", \"great\", \"great\", \"great\", \"great\", \"great\", \"gremlins\", \"grief\", \"gruesome\", \"gun\", \"gun\", \"gun\", \"gun\", \"gun\", \"guns\", \"guns\", \"guns\", \"guy\", \"guy\", \"guy\", \"guy\", \"guy\", \"guy\", \"guy\", \"guys\", \"guys\", \"guys\", \"guys\", \"guys\", \"ha\", \"hangs\", \"hardy\", \"hardy\", \"harry\", \"harry\", \"harry\", \"haunted\", \"haunted\", \"head\", \"head\", \"head\", \"head\", \"head\", \"head\", \"head\", \"head\", \"head\", \"helicopter\", \"hellraiser\", \"hero\", \"hero\", \"hero\", \"hero\", \"hero\", \"hero\", \"hero\", \"hero\", \"high\", \"high\", \"high\", \"high\", \"high\", \"high\", \"high\", \"high\", \"high\", \"high\", \"highlander\", \"historical\", \"historical\", \"historically\", \"history\", \"history\", \"history\", \"history\", \"history\", \"history\", \"hitler\", \"hoffman\", \"hogan\", \"hollywood\", \"hollywood\", \"hollywood\", \"hollywood\", \"hollywood\", \"hollywood\", \"hollywood\", \"holmes\", \"home\", \"home\", \"home\", \"home\", \"home\", \"home\", \"home\", \"hong\", \"hopkins\", \"horrible\", \"horrible\", \"horrible\", \"horror\", \"horror\", \"horror\", \"house\", \"house\", \"house\", \"house\", \"house\", \"hudson\", \"hudson\", \"hudson\", \"human\", \"human\", \"humanity\", \"humanity\", \"husband\", \"husband\", \"im\", \"important\", \"important\", \"important\", \"important\", \"important\", \"important\", \"important\", \"important\", \"indians\", \"inner\", \"inner\", \"inner\", \"inner\", \"insight\", \"insight\", \"insights\", \"inspiring\", \"insult\", \"insult\", \"insulting\", \"insulting\", \"investigating\", \"irene\", \"issues\", \"issues\", \"issues\", \"jack\", \"jack\", \"jack\", \"jack\", \"jack\", \"jack\", \"jackie\", \"jackie\", \"james\", \"james\", \"james\", \"james\", \"james\", \"james\", \"james\", \"james\", \"jane\", \"jane\", \"jaws\", \"jazz\", \"jenny\", \"jet\", \"jet\", \"job\", \"job\", \"job\", \"job\", \"job\", \"job\", \"job\", \"john\", \"john\", \"john\", \"john\", \"john\", \"john\", \"john\", \"john\", \"jr\", \"jr\", \"jr\", \"jr\", \"jr\", \"jungle\", \"jungle\", \"karloff\", \"keaton\", \"kelly\", \"khan\", \"khan\", \"kidman\", \"kids\", \"kids\", \"kids\", \"kids\", \"kids\", \"kill\", \"kill\", \"kill\", \"kill\", \"kill\", \"killer\", \"killer\", \"killers\", \"killers\", \"killers\", \"killing\", \"killing\", \"killing\", \"killings\", \"king\", \"king\", \"king\", \"king\", \"king\", \"know\", \"know\", \"know\", \"know\", \"know\", \"know\", \"know\", \"kong\", \"kong\", \"kung\", \"kurosawa\", \"lab\", \"lana\", \"lane\", \"last\", \"last\", \"last\", \"last\", \"last\", \"last\", \"last\", \"last\", \"last\", \"later\", \"later\", \"later\", \"later\", \"later\", \"later\", \"later\", \"later\", \"later\", \"laughton\", \"le\", \"le\", \"least\", \"least\", \"least\", \"least\", \"least\", \"least\", \"least\", \"lee\", \"lee\", \"lee\", \"lenny\", \"leopard\", \"lesbians\", \"lewton\", \"li\", \"liam\", \"life\", \"life\", \"life\", \"life\", \"life\", \"life\", \"like\", \"like\", \"like\", \"like\", \"like\", \"like\", \"like\", \"like\", \"like\", \"like\", \"liked\", \"liked\", \"liked\", \"liked\", \"liked\", \"liked\", \"liked\", \"little\", \"little\", \"little\", \"little\", \"little\", \"little\", \"little\", \"little\", \"little\", \"little\", \"lives\", \"lives\", \"lives\", \"lives\", \"lives\", \"lohan\", \"lol\", \"loneliness\", \"look\", \"look\", \"look\", \"look\", \"look\", \"look\", \"look\", \"look\", \"look\", \"look\", \"looks\", \"looks\", \"looks\", \"looks\", \"looks\", \"looks\", \"looks\", \"looks\", \"looney\", \"lot\", \"lot\", \"lot\", \"lot\", \"lot\", \"lot\", \"lot\", \"lot\", \"love\", \"love\", \"love\", \"love\", \"love\", \"love\", \"loved\", \"loved\", \"loved\", \"lowest\", \"loy\", \"lugosi\", \"machine\", \"machine\", \"machine\", \"madame\", \"made\", \"made\", \"made\", \"made\", \"made\", \"made\", \"made\", \"made\", \"made\", \"made\", \"make\", \"make\", \"make\", \"make\", \"make\", \"make\", \"make\", \"make\", \"make\", \"make\", \"man\", \"man\", \"man\", \"man\", \"man\", \"man\", \"man\", \"man\", \"man\", \"many\", \"many\", \"many\", \"many\", \"many\", \"many\", \"many\", \"many\", \"many\", \"many\", \"marcus\", \"marie\", \"maris\", \"married\", \"marries\", \"marry\", \"martial\", \"martin\", \"martin\", \"martin\", \"martin\", \"martin\", \"martin\", \"marx\", \"mary\", \"mary\", \"mary\", \"mary\", \"mary\", \"mary\", \"massacre\", \"massacre\", \"match\", \"match\", \"match\", \"match\", \"match\", \"match\", \"matt\", \"matt\", \"matt\", \"may\", \"may\", \"may\", \"may\", \"may\", \"may\", \"may\", \"may\", \"may\", \"media\", \"media\", \"media\", \"meets\", \"meets\", \"meets\", \"meets\", \"meets\", \"meets\", \"meets\", \"meg\", \"melanie\", \"meryl\", \"mess\", \"mess\", \"mess\", \"mess\", \"message\", \"message\", \"message\", \"metal\", \"metal\", \"mgm\", \"michael\", \"michael\", \"michael\", \"michael\", \"michael\", \"michael\", \"michael\", \"michael\", \"michael\", \"mickey\", \"miike\", \"military\", \"military\", \"minutes\", \"minutes\", \"minutes\", \"minutes\", \"minutes\", \"miserably\", \"mom\", \"mom\", \"money\", \"money\", \"money\", \"money\", \"monster\", \"monster\", \"monsters\", \"monsters\", \"moore\", \"moore\", \"moore\", \"moore\", \"moral\", \"moral\", \"moral\", \"moral\", \"morris\", \"mother\", \"mother\", \"mother\", \"movie.i\", \"movies\", \"movies\", \"movies\", \"movies\", \"movies\", \"movies\", \"movies\", \"movies\", \"mr\", \"mr\", \"mr\", \"mr\", \"mr\", \"mr\", \"mr\", \"mr\", \"mr\", \"mst3k\", \"much\", \"much\", \"much\", \"much\", \"much\", \"much\", \"much\", \"much\", \"much\", \"much\", \"murder\", \"murder\", \"murder\", \"murderer\", \"murders\", \"murphy\", \"murray\", \"music\", \"music\", \"music\", \"music\", \"music\", \"music\", \"music\", \"musical\", \"musicals\", \"muslim\", \"mutant\", \"mysterious\", \"mysterious\", \"mysterious\", \"mysterious\", \"mysterious\", \"mysterious\", \"mysterious\", \"mystery\", \"mystery\", \"mystery\", \"mystery\", \"mystery\", \"na\", \"na\", \"nathan\", \"nature\", \"nature\", \"nature\", \"nature\", \"nature\", \"nature\", \"nazi\", \"nazis\", \"never\", \"never\", \"never\", \"never\", \"never\", \"never\", \"never\", \"never\", \"never\", \"never\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new\", \"new\", \"nicholson\", \"night\", \"night\", \"night\", \"night\", \"night\", \"night\", \"night\", \"nikita\", \"nikki\", \"ninja\", \"ninja\", \"niro\", \"noah\", \"non-existent\", \"norris\", \"north\", \"north\", \"north\", \"nothing\", \"nothing\", \"nothing\", \"nothing\", \"nothing\", \"nothing\", \"nothing\", \"nothing\", \"novel\", \"novel\", \"novel\", \"novel\", \"novel\", \"novel\", \"novel\", \"o'hara\", \"o'toole\", \"old\", \"old\", \"old\", \"old\", \"old\", \"old\", \"old\", \"old\", \"old\", \"old\", \"olivia\", \"one\", \"one\", \"one\", \"one\", \"one\", \"one\", \"one\", \"one\", \"one\", \"one\", \"orphanage\", \"orson\", \"p.s\", \"pacino\", \"paltrow\", \"pammy\", \"panther\", \"parents\", \"parents\", \"parents\", \"parker\", \"parker\", \"pathetic\", \"pathetic\", \"paul\", \"paul\", \"paul\", \"paul\", \"paul\", \"paul\", \"paul\", \"paul\", \"penn\", \"penn\", \"people\", \"people\", \"people\", \"people\", \"people\", \"performance\", \"performance\", \"performance\", \"performance\", \"performance\", \"performance\", \"performance\", \"performance\", \"performances\", \"performances\", \"performances\", \"performances\", \"performances\", \"performances\", \"performances\", \"performances\", \"personal\", \"personal\", \"personal\", \"personal\", \"personal\", \"personal\", \"personal\", \"personal\", \"phantasm\", \"philosophy\", \"pitiful\", \"plane\", \"plane\", \"plane\", \"plane\", \"planet\", \"planet\", \"planet\", \"planet\", \"play\", \"play\", \"play\", \"play\", \"play\", \"play\", \"play\", \"play\", \"play\", \"played\", \"played\", \"played\", \"played\", \"played\", \"played\", \"played\", \"played\", \"played\", \"played\", \"plays\", \"plays\", \"plays\", \"plays\", \"plays\", \"plays\", \"plays\", \"plays\", \"plays\", \"plays\", \"plot\", \"plot\", \"plot\", \"plot\", \"plot\", \"plot\", \"plot\", \"pointless\", \"pointless\", \"police\", \"police\", \"police\", \"police\", \"police\", \"political\", \"political\", \"poor\", \"poor\", \"poor\", \"poor\", \"poor\", \"poor\", \"poorly\", \"poorly\", \"porky\", \"powell\", \"powerful\", \"powerful\", \"powerful\", \"powerful\", \"powerful\", \"powerful\", \"predator\", \"president\", \"president\", \"previews\", \"programme\", \"prostitution\", \"pryor\", \"punk\", \"puppet\", \"quaid\", \"quite\", \"quite\", \"quite\", \"quite\", \"quite\", \"quite\", \"quite\", \"quite\", \"quite\", \"randolph\", \"real\", \"real\", \"real\", \"real\", \"real\", \"real\", \"real\", \"real\", \"real\", \"real\", \"reality\", \"reality\", \"reality\", \"reality\", \"really\", \"really\", \"really\", \"really\", \"really\", \"really\", \"really\", \"redeeming\", \"reflect\", \"relationships\", \"relationships\", \"religion\", \"religious\", \"religious\", \"religious\", \"religious\", \"remember\", \"remember\", \"remember\", \"remember\", \"remember\", \"remember\", \"remember\", \"remotely\", \"remotely\", \"reruns\", \"revolution\", \"robert\", \"robert\", \"robert\", \"robert\", \"robert\", \"robert\", \"robin\", \"robin\", \"robin\", \"robinson\", \"robinson\", \"robot\", \"robot\", \"robots\", \"rock\", \"rock\", \"rock\", \"rock\", \"rock\", \"rock\", \"rodman\", \"role\", \"role\", \"role\", \"role\", \"role\", \"role\", \"role\", \"role\", \"role\", \"roles\", \"roles\", \"roles\", \"roles\", \"roles\", \"roles\", \"roles\", \"roles\", \"roman\", \"rubber\", \"rubbish\", \"rubbish\", \"russia\", \"russian\", \"russian\", \"russian\", \"russians\", \"sadako\", \"sandler\", \"saw\", \"saw\", \"saw\", \"saw\", \"say\", \"say\", \"say\", \"say\", \"say\", \"scares\", \"scary\", \"scary\", \"scary\", \"scary\", \"scene\", \"scene\", \"scene\", \"scene\", \"scene\", \"scene\", \"scene\", \"scene\", \"scene\", \"scene\", \"scenes\", \"scenes\", \"scenes\", \"scenes\", \"scenes\", \"scenes\", \"scenes\", \"scenes\", \"scenes\", \"scenes\", \"school\", \"school\", \"school\", \"school\", \"school\", \"sci-fi\", \"sci-fi\", \"science\", \"science\", \"science\", \"scientist\", \"scientist\", \"scientists\", \"score\", \"score\", \"score\", \"score\", \"score\", \"score\", \"scott\", \"scott\", \"scott\", \"scott\", \"scott\", \"screen\", \"screen\", \"screen\", \"screen\", \"screen\", \"screen\", \"screen\", \"screen\", \"script\", \"script\", \"script\", \"script\", \"script\", \"seagal\", \"season\", \"seasons\", \"seasons\", \"see\", \"see\", \"see\", \"see\", \"see\", \"see\", \"see\", \"see\", \"see\", \"see\", \"seen\", \"seen\", \"seen\", \"seen\", \"seen\", \"seen\", \"seen\", \"self-indulgent\", \"sellers\", \"serial\", \"series\", \"series\", \"series\", \"series\", \"set\", \"set\", \"set\", \"set\", \"set\", \"set\", \"set\", \"set\", \"set\", \"set\", \"seth\", \"shark\", \"sharks\", \"sharpe\", \"sherlock\", \"sherlock\", \"show\", \"show\", \"show\", \"show\", \"show\", \"show\", \"show\", \"show\", \"show\", \"silent\", \"silent\", \"silent\", \"silent\", \"silent\", \"sinbad\", \"singer\", \"singer\", \"singer\", \"singer\", \"singing\", \"singing\", \"singing\", \"sister\", \"sister\", \"sister\", \"sister\", \"slasher\", \"smith\", \"smith\", \"smith\", \"smith\", \"smith\", \"smith\", \"smith\", \"snoopy\", \"social\", \"social\", \"social\", \"society\", \"soldiers\", \"soldiers\", \"something\", \"something\", \"something\", \"something\", \"something\", \"something\", \"son\", \"son\", \"son\", \"son\", \"son\", \"son\", \"song\", \"song\", \"song\", \"song\", \"songs\", \"songs\", \"songs\", \"sophie\", \"soviet\", \"space\", \"space\", \"space\", \"space\", \"space\", \"spaghetti\", \"special\", \"special\", \"special\", \"special\", \"special\", \"special\", \"special\", \"special\", \"spiritual\", \"spooky\", \"stage\", \"stage\", \"stage\", \"stage\", \"stage\", \"stan\", \"stanwyck\", \"star\", \"star\", \"star\", \"star\", \"star\", \"star\", \"star\", \"star\", \"states\", \"states\", \"states\", \"states\", \"states\", \"steaming\", \"steve\", \"steve\", \"steve\", \"steve\", \"steve\", \"still\", \"still\", \"still\", \"still\", \"still\", \"still\", \"still\", \"still\", \"still\", \"still\", \"story\", \"story\", \"story\", \"story\", \"story\", \"story\", \"story\", \"story\", \"streep\", \"studied\", \"stupid\", \"stupid\", \"stupid\", \"subject\", \"subject\", \"subject\", \"subtly\", \"supernatural\", \"supporting\", \"supporting\", \"supporting\", \"supporting\", \"supporting\", \"supporting\", \"supposed\", \"supposed\", \"supposed\", \"supposed\", \"supposed\", \"supposed\", \"supposed\", \"suspense\", \"suspense\", \"suspense\", \"take\", \"take\", \"take\", \"take\", \"take\", \"take\", \"take\", \"take\", \"take\", \"take\", \"taped\", \"tarzan\", \"tempted\", \"terminator\", \"terrible\", \"terrible\", \"terrible\", \"themes\", \"themes\", \"themes\", \"themes\", \"thing\", \"thing\", \"thing\", \"thing\", \"thing\", \"thing\", \"think\", \"think\", \"think\", \"think\", \"thought\", \"thought\", \"thought\", \"thought\", \"thought-provoking\", \"thought-provoking\", \"thriller\", \"thriller\", \"thrillers\", \"time\", \"time\", \"time\", \"time\", \"time\", \"time\", \"time\", \"time\", \"time\", \"time\", \"timothy\", \"tom\", \"tom\", \"tom\", \"tom\", \"tom\", \"tom\", \"tony\", \"tony\", \"town\", \"town\", \"town\", \"town\", \"town\", \"town\", \"town\", \"tribe\", \"tripe\", \"troll\", \"troma\", \"troops\", \"troy\", \"true\", \"true\", \"true\", \"true\", \"true\", \"true\", \"true\", \"true\", \"tv\", \"tv\", \"tv\", \"tv\", \"tv\", \"tv\", \"tv\", \"tv\", \"tv\", \"two\", \"two\", \"two\", \"two\", \"two\", \"two\", \"two\", \"two\", \"two\", \"two\", \"u.s\", \"u.s\", \"u.s\", \"u.s\", \"understanding\", \"understanding\", \"unfunny\", \"unfunny\", \"unfunny\", \"unfunny\", \"uninteresting\", \"union\", \"union\", \"united\", \"unlikeable\", \"unwatchable\", \"us\", \"us\", \"us\", \"us\", \"us\", \"us\", \"us\", \"us\", \"vampires\", \"version\", \"version\", \"version\", \"version\", \"version\", \"version\", \"version\", \"vienna\", \"view\", \"view\", \"view\", \"view\", \"view\", \"view\", \"view\", \"viewer\", \"viewer\", \"viewer\", \"viewer\", \"viewer\", \"violence\", \"violence\", \"violence\", \"violence\", \"violence\", \"violin\", \"virus\", \"visitor\", \"waitress\", \"waitress\", \"wants\", \"wants\", \"wants\", \"wants\", \"wants\", \"wants\", \"wants\", \"wants\", \"war\", \"warner\", \"warner\", \"waste\", \"waste\", \"wasted\", \"wasted\", \"wasting\", \"wasting\", \"wasting\", \"watch\", \"watch\", \"watch\", \"watch\", \"watch\", \"watch\", \"watch\", \"watched\", \"watched\", \"watched\", \"watched\", \"watching\", \"watching\", \"watching\", \"watching\", \"water\", \"water\", \"water\", \"water\", \"water\", \"watson\", \"way\", \"way\", \"way\", \"way\", \"way\", \"way\", \"way\", \"way\", \"way\", \"way\", \"wayne\", \"wayne\", \"wayne\", \"wealthy\", \"wealthy\", \"wealthy\", \"well\", \"well\", \"well\", \"well\", \"well\", \"well\", \"well\", \"well\", \"well\", \"welles\", \"werewolf\", \"western\", \"western\", \"westerns\", \"westerns\", \"whatsoever\", \"whatsoever\", \"wife\", \"wife\", \"wife\", \"wife\", \"wife\", \"wife\", \"wife\", \"wife\", \"wilson\", \"wilson\", \"without\", \"without\", \"without\", \"without\", \"without\", \"without\", \"without\", \"without\", \"without\", \"woman\", \"woman\", \"woman\", \"woman\", \"woman\", \"women\", \"women\", \"women\", \"women\", \"women\", \"women\", \"wonderful\", \"wonderful\", \"wonderful\", \"wonderful\", \"wonderful\", \"wonderful\", \"woo\", \"woody\", \"work\", \"work\", \"work\", \"work\", \"work\", \"work\", \"work\", \"work\", \"work\", \"world\", \"world\", \"world\", \"world\", \"world\", \"world\", \"world\", \"world\", \"worse\", \"worse\", \"worse\", \"worst\", \"worst\", \"worst\", \"would\", \"would\", \"would\", \"would\", \"would\", \"would\", \"would\", \"would\", \"would\", \"would\", \"wwii\", \"years\", \"years\", \"years\", \"years\", \"years\", \"years\", \"years\", \"years\", \"years\", \"years\", \"yet\", \"yet\", \"yet\", \"yet\", \"yet\", \"yet\", \"yet\", \"yet\", \"yet\", \"yet\", \"young\", \"young\", \"young\", \"young\", \"young\", \"young\", \"young\", \"zohan\", \"zombie\", \"zombies\"]}, \"R\": 30, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [7, 9, 3, 6, 4, 2, 5, 10, 1, 8]};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el1437551404698944397044237069456\", ldavis_el1437551404698944397044237069456_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el1437551404698944397044237069456\", ldavis_el1437551404698944397044237069456_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.rawgit.com/bmabey/pyLDAvis/files/ldavis.v1.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el1437551404698944397044237069456\", ldavis_el1437551404698944397044237069456_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "PreparedData(topic_coordinates=            Freq  cluster  topics         x         y\n",
       "topic                                                \n",
       "6      21.775694        1       1  0.165085  0.049737\n",
       "8      16.944525        1       2  0.138518  0.016776\n",
       "2      16.244984        1       3  0.153824 -0.000688\n",
       "5       8.819422        1       4  0.016536  0.090156\n",
       "3       8.718466        1       5  0.051643  0.179948\n",
       "1       8.077996        1       6  0.063781  0.000383\n",
       "4       7.723183        1       7 -0.012106 -0.208874\n",
       "9       5.885868        1       8  0.006707 -0.051551\n",
       "0       3.719499        1       9 -0.181318 -0.243855\n",
       "7       2.090364        1      10 -0.402670  0.167967, topic_info=      Category           Freq         Term          Total  loglift  logprob\n",
       "term                                                                       \n",
       "51     Default  117884.000000         film  117884.000000  30.0000  30.0000\n",
       "8      Default   27038.000000          bad   27038.000000  29.0000  29.0000\n",
       "450    Default   43300.000000         good   43300.000000  28.0000  28.0000\n",
       "1135   Default   10177.000000       horror   10177.000000  27.0000  27.0000\n",
       "186    Default   27211.000000        great   27211.000000  26.0000  26.0000\n",
       "2148   Default   16980.000000         show   16980.000000  25.0000  25.0000\n",
       "704    Default   10312.000000       action   10312.000000  24.0000  24.0000\n",
       "118    Default   34704.000000       really   34704.000000  23.0000  23.0000\n",
       "214    Default    6023.000000          war    6023.000000  22.0000  22.0000\n",
       "81     Default   58839.000000         like   58839.000000  21.0000  21.0000\n",
       "940    Default   18912.000000         best   18912.000000  20.0000  20.0000\n",
       "1021   Default    9371.000000         role    9371.000000  19.0000  19.0000\n",
       "604    Default   17378.000000         life   17378.000000  18.0000  18.0000\n",
       "816    Default   13026.000000        funny   13026.000000  17.0000  17.0000\n",
       "20     Default   20484.000000    character   20484.000000  16.0000  16.0000\n",
       "875    Default   11272.000000         cast   11272.000000  15.0000  15.0000\n",
       "696    Default   19807.000000        watch   19807.000000  14.0000  14.0000\n",
       "996    Default    8258.000000  performance    8258.000000  13.0000  13.0000\n",
       "109    Default   27723.000000       people   27723.000000  12.0000  12.0000\n",
       "140    Default   21069.000000        think   21069.000000  11.0000  11.0000\n",
       "985    Default   18407.000000         love   18407.000000  10.0000  10.0000\n",
       "2035   Default    9363.000000       comedy    9363.000000   9.0000   9.0000\n",
       "917    Default   22479.000000       movies   22479.000000   8.0000   8.0000\n",
       "637    Default   19103.000000         plot   19103.000000   7.0000   7.0000\n",
       "125    Default   33745.000000          see   33745.000000   6.0000   6.0000\n",
       "56     Default   27452.000000          get   27452.000000   5.0000   5.0000\n",
       "854    Default   10575.000000        young   10575.000000   4.0000   4.0000\n",
       "750    Default    7637.000000        woman    7637.000000   3.0000   3.0000\n",
       "1127   Default    7980.000000       family    7980.000000   2.0000   2.0000\n",
       "304    Default    6739.000000         john    6739.000000   1.0000   1.0000\n",
       "...        ...            ...          ...            ...      ...      ...\n",
       "11827  Topic10     236.240265       abbott     237.133896   3.8641  -6.6281\n",
       "21903  Topic10     229.653168       kidman     230.546967   3.8639  -6.6564\n",
       "11662  Topic10     214.989212        elvis     215.882889   3.8637  -6.7224\n",
       "20888  Topic10     210.273285     gremlins     211.166931   3.8636  -6.7445\n",
       "15211  Topic10     208.268814        avery     209.162628   3.8635  -6.7541\n",
       "10734  Topic10     244.776810         chow     246.068649   3.8626  -6.5926\n",
       "8491   Topic10     880.195251        chuck     964.342957   3.7765  -5.3128\n",
       "3107   Topic10     854.855103        eddie     986.127930   3.7250  -5.3420\n",
       "23165  Topic10     353.108490       hudson     379.121552   3.7968  -6.2262\n",
       "1799   Topic10    1365.116333         rock    1836.089600   3.5714  -4.8739\n",
       "7639   Topic10     376.626099        hardy     415.518219   3.7696  -6.1617\n",
       "712    Topic10    1188.493164        harry    1800.127563   3.4527  -5.0125\n",
       "3010   Topic10    1198.136597          lee    2190.593262   3.2644  -5.0044\n",
       "4412   Topic10     797.709656         band    1247.509277   3.4207  -5.4112\n",
       "3111   Topic10     557.767761        metal     768.608154   3.5472  -5.7690\n",
       "2654   Topic10     521.394775       jackie     726.668945   3.5359  -5.8364\n",
       "5328   Topic10     326.906799          jet     366.248199   3.7542  -6.3033\n",
       "4234   Topic10     817.311462        bruce    1559.576050   3.2217  -5.3869\n",
       "5274   Topic10     329.124115      beatles     370.890991   3.7484  -6.2965\n",
       "2932   Topic10     644.153076      chinese    1101.371948   3.3315  -5.6250\n",
       "1023   Topic10    1204.058228       school    4466.029297   2.5570  -4.9995\n",
       "704    Topic10    1694.326294       action   10312.356445   2.0618  -4.6579\n",
       "299    Topic10    1073.200439         high    5516.469727   2.2307  -5.1145\n",
       "281    Topic10     793.160583        fight    3338.863770   2.4305  -5.4169\n",
       "3968   Topic10     554.642578        scott    1533.908081   2.8506  -5.7746\n",
       "707    Topic10     480.602203        dirty    1001.221191   3.1339  -5.9179\n",
       "546    Topic10     646.293884        black    6047.805664   1.6316  -5.6217\n",
       "706    Topic10     433.185120          cop    2167.926514   2.2575  -6.0218\n",
       "3590   Topic10     421.390778         gang    1709.442871   2.4675  -6.0494\n",
       "7878   Topic10     412.223999         boys    1932.367188   2.3229  -6.0714\n",
       "\n",
       "[824 rows x 6 columns], token_table=       Topic      Freq        Term\n",
       "term                              \n",
       "8745       1  0.998418        1/10\n",
       "5801       5  0.004689        13th\n",
       "5801       6  0.991748        13th\n",
       "5292       8  0.996792        19th\n",
       "6810       3  0.994564        9/11\n",
       "11827     10  0.995218      abbott\n",
       "4084       2  0.993156         abc\n",
       "5350       1  0.996650     abysmal\n",
       "9201       3  0.995932  accessible\n",
       "722        1  0.698299      acting\n",
       "722        2  0.171862      acting\n",
       "722        3  0.023542      acting\n",
       "722        5  0.001280      acting\n",
       "722        6  0.019813      acting\n",
       "722        7  0.070459      acting\n",
       "722        9  0.014693      acting\n",
       "704        1  0.132656      action\n",
       "704        2  0.134984      action\n",
       "704        3  0.001939      action\n",
       "704        5  0.440055      action\n",
       "704        6  0.050425      action\n",
       "704        7  0.000291      action\n",
       "704        8  0.067492      action\n",
       "704        9  0.007952      action\n",
       "704       10  0.164269      action\n",
       "1876       1  0.302455       actor\n",
       "1876       2  0.100611       actor\n",
       "1876       3  0.004368       actor\n",
       "1876       4  0.000468       actor\n",
       "1876       6  0.000312       actor\n",
       "...      ...       ...         ...\n",
       "702        1  0.013167       years\n",
       "702        2  0.465214       years\n",
       "702        3  0.063539       years\n",
       "702        4  0.157470       years\n",
       "702        5  0.004517       years\n",
       "702        6  0.027942       years\n",
       "702        7  0.054736       years\n",
       "702        8  0.211746       years\n",
       "702        9  0.001225       years\n",
       "702       10  0.000383       years\n",
       "2255       1  0.161309         yet\n",
       "2255       2  0.166596         yet\n",
       "2255       3  0.412378         yet\n",
       "2255       4  0.023732         yet\n",
       "2255       5  0.049462         yet\n",
       "2255       6  0.088467         yet\n",
       "2255       7  0.063560         yet\n",
       "2255       8  0.018798         yet\n",
       "2255       9  0.014921         yet\n",
       "2255      10  0.000822         yet\n",
       "854        2  0.069498       young\n",
       "854        3  0.175212       young\n",
       "854        4  0.549746       young\n",
       "854        6  0.098622       young\n",
       "854        7  0.092097       young\n",
       "854        8  0.013238       young\n",
       "854        9  0.001607       young\n",
       "51577      2  0.995358       zohan\n",
       "4336       5  0.999152      zombie\n",
       "4445       5  0.998692     zombies\n",
       "\n",
       "[2359 rows x 3 columns], R=30, lambda_step=0.01, plot_opts={'xlab': 'PC1', 'ylab': 'PC2'}, topic_order=[7, 9, 3, 6, 4, 2, 5, 10, 1, 8])"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyLDAvis\n",
    "import pyLDAvis.gensim\n",
    "pyLDAvis.enable_notebook()\n",
    "pyLDAvis.gensim.prepare(lda_tf, corpus_tf, dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making predictions with LDA vectors (we'll have to use only the labeled sentences):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Corpora for predictions\n",
      "corpus tf done\n",
      "corpus lda tf done\n",
      "Creating lda vectors\n",
      "lda vectors done\n"
     ]
    }
   ],
   "source": [
    "#For predictions we need only the labeled set:\n",
    "print(\"Creating Corpora for predictions\")\n",
    "corpus_tf2 = [dictionary.doc2bow(sentence) for sentence in labeled_sentences]\n",
    "print('corpus tf done')\n",
    "\n",
    "lda_tf2 = models.LdaModel(corpus_tf2, id2word=dictionary, num_topics=10, passes=10)\n",
    "corpus_lda_tf2 = lda_tf2[corpus_tf2]\n",
    "print('corpus lda tf done')\n",
    "\n",
    "print('Creating lda vectors')\n",
    "X = gensim.matutils.corpus2csc(corpus_lda_tf2)\n",
    "X = X.T\n",
    "print('lda vectors done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[0.30562708, 0.        , 0.        , 0.05587129, 0.        ,\n",
       "         0.        , 0.04004983, 0.38296849, 0.21312414, 0.        ]])"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0].todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving the Sentences, Model and the Corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_tf2.save(os.path.join(outputs, 'model_tf2.lda'))\n",
    "corpora.MmCorpus.serialize(os.path.join(outputs, 'corpus_lda_tf2.mm'), corpus_lda_tf2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the Sentences, Model and the Corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_tf2 = models.LdaModel.load(os.path.join(outputs, 'model_tf2.lda'))\n",
    "corpus_lda_tf2 = corpora.MmCorpus(os.path.join(outputs, 'corpus_lda_tf2.mm'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_traincv_tf, X_testcv_tf, y_traincv_tf, y_testcv_tf = model_selection.train_test_split(X,\n",
    "                                                                                        train[\"sentiment\"],\n",
    "                                                                                        test_size=0.2,\n",
    "                                                                                        random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression as LR\n",
    "\n",
    "clf_LR_tf_lda = LR(penalty='l2',\n",
    "                   dual=False,\n",
    "                   tol=0.0001,\n",
    "                   C=1.0,\n",
    "                   fit_intercept=True,\n",
    "                   intercept_scaling=1,\n",
    "                   class_weight=None,\n",
    "                   random_state=0,\n",
    "                   solver='liblinear',\n",
    "                   max_iter=100,\n",
    "                   multi_class='ovr',\n",
    "                   verbose=0).fit(X_traincv_tf, y_traincv_tf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7424\n"
     ]
    }
   ],
   "source": [
    "eval_LR_tf_lda = clf_LR_tf_lda.score(X_testcv_tf, y_testcv_tf)\n",
    "print(eval_LR_tf_lda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distributed Word Vectors\n",
    "\n",
    "https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors  \n",
    "\n",
    "Introducing Distributed Word Vectors: This part of the tutorial will focus on using distributed word vectors created by the Word2Vec algorithm, using the Gensim implementation.  \n",
    "\n",
    "https://radimrehurek.com/gensim/models/word2vec.html  \n",
    "\n",
    "Word2vec, published by Google in 2013, is a neural network implementation that learns distributed representations for words. Other deep or recurrent neural network architectures had been proposed for learning word representations prior to this, but the major problem with these was the long time required to train the models. Word2vec learns quickly relative to other models.\n",
    "\n",
    "Word2Vec does not need labels in order to create meaningful representations. This is useful, since most data in the real world is unlabeled. If the network is given enough training data (tens of billions of words), it produces word vectors with intriguing characteristics. Words with similar meanings appear in clusters, and clusters are spaced such that some word relationships, such as analogies, can be reproduced using vector math. The famous example is that, with highly trained word vectors, \"king - man + woman = queen.\"\n",
    "\n",
    "Distributed word vectors are powerful and can be used for many applications, particularly word prediction and translation. Here, we will try to apply them to sentiment analysis.\n",
    "\n",
    "Using word2vec in Python: In Python, we will use the excellent implementation of word2vec from the gensim package. If you don't already have gensim installed, you'll need to install it. There is an excellent tutorial that accompanies the Python Word2Vec implementation, here.\n",
    "\n",
    "Although Word2Vec does not require graphics processing units (GPUs) like many deep learning algorithms, it is compute intensive. Both Google's version and the Python version rely on multi-threading (running multiple processes in parallel on your computer to save time). ln order to train your model in a reasonable amount of time, you will need to install cython (instructions here). Word2Vec will run without cython installed, but it will take days to run instead of minutes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preparing to Train a Model\n",
    "\n",
    "We are going to use the whole set of sentences for training, the labeled and the unlabeled, which contains 50,000 additional reviews with no labels. When we built the Bag of Words model in Part 1, extra unlabeled training reviews were not useful. However, since Word2Vec can learn from unlabeled data, these extra 50,000 reviews can now be used.  \n",
    "\n",
    "To train Word2Vec it is better not to remove stop words because the algorithm relies on the broader context of the sentence in order to produce high-quality word vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training and Saving Your Model  \n",
    "\n",
    "With the list of nicely parsed sentences, we're ready to train the model. There are a number of parameter choices that affect the run time and the quality of the final model that is produced. For details on the algorithms below, see the word2vec API documentation as well as the Google documentation. \n",
    "\n",
    "Architecture: Architecture options are skip-gram (default) or continuous bag of words. We found that skip-gram was very slightly slower but produced better results.\n",
    "\n",
    "Training algorithm: Hierarchical softmax (default) or negative sampling. For us, the default worked well.\n",
    "\n",
    "Downsampling of frequent words: The Google documentation recommends values between .00001 and .001. For us, values closer 0.001 seemed to improve the accuracy of the final model.\n",
    "\n",
    "Word vector dimensionality: More features result in longer runtimes, and often, but not always, result in better models. Reasonable values can be in the tens to hundreds; we used 300.\n",
    "\n",
    "Context / window size: How many words of context should the training algorithm take into account? 10 seems to work well for hierarchical softmax (more is better, up to a point).\n",
    "\n",
    "Worker threads: Number of parallel processes to run. This is computer-specific, but between 4 and 6 should work on most systems.\n",
    "\n",
    "Minimum word count: This helps limit the size of the vocabulary to meaningful words. Any word that does not occur at least this many times across all documents is ignored. Reasonable values could be between 10 and 100. In this case, since each movie occurs 30 times, we set the minimum word count to 40, to avoid attaching too much importance to individual movie titles. This resulted in an overall vocabulary size of around 15,000 words. Higher values also help limit run time.\n",
    "\n",
    "Choosing parameters is not easy, but once we have chosen our parameters, creating a Word2Vec model is straightforward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the built-in logging module and configure it so that Word2Vec creates nice output messages\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_features = 300    # Word vector dimensionality\n",
    "min_word_count = 5   # Minimum word count\n",
    "num_workers = -1       # Number of threads to run in parallel\n",
    "context = 10          # Context window size\n",
    "downsampling = 1e-5   # Downsample setting for frequent words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-04-25 22:51:25,123 : INFO : collecting all words and their counts\n",
      "2018-04-25 22:51:25,124 : INFO : PROGRESS: at sentence #0, processed 0 words, keeping 0 word types\n",
      "2018-04-25 22:51:25,767 : INFO : PROGRESS: at sentence #10000, processed 2228586 words, keeping 74969 word types\n",
      "2018-04-25 22:51:26,407 : INFO : PROGRESS: at sentence #20000, processed 4434587 words, keeping 110864 word types\n",
      "2018-04-25 22:51:26,978 : INFO : PROGRESS: at sentence #30000, processed 6632633 words, keeping 142823 word types\n",
      "2018-04-25 22:51:27,551 : INFO : PROGRESS: at sentence #40000, processed 8843751 words, keeping 171836 word types\n",
      "2018-04-25 22:51:28,155 : INFO : PROGRESS: at sentence #50000, processed 11083869 words, keeping 198432 word types\n",
      "2018-04-25 22:51:28,744 : INFO : PROGRESS: at sentence #60000, processed 13311069 words, keeping 222847 word types\n",
      "2018-04-25 22:51:29,329 : INFO : PROGRESS: at sentence #70000, processed 15517410 words, keeping 245574 word types\n",
      "2018-04-25 22:51:29,625 : INFO : collected 256645 word types from a corpus of 16627625 raw words and 75000 sentences\n",
      "2018-04-25 22:51:29,627 : INFO : Loading a fresh vocabulary\n",
      "2018-04-25 22:51:29,871 : INFO : min_count=5 retains 52830 unique words (20% of original 256645, drops 203815)\n",
      "2018-04-25 22:51:29,872 : INFO : min_count=5 leaves 16352274 word corpus (98% of original 16627625, drops 275351)\n",
      "2018-04-25 22:51:30,170 : INFO : deleting the raw counts dictionary of 256645 items\n",
      "2018-04-25 22:51:30,179 : INFO : sample=1e-05 downsamples 3075 most-common words\n",
      "2018-04-25 22:51:30,180 : INFO : downsampling leaves estimated 4510471 word corpus (27.6% of prior 16352274)\n",
      "2018-04-25 22:51:30,413 : INFO : estimated required memory for 52830 words and 300 dimensions: 153207000 bytes\n",
      "2018-04-25 22:51:30,415 : INFO : resetting layer weights\n",
      "2018-04-25 22:51:32,040 : INFO : training model with -1 workers on 52830 vocabulary and 300 features, using sg=0 hs=0 sample=1e-05 negative=5 window=10\n",
      "2018-04-25 22:51:32,047 : INFO : EPOCH - 1 : training on 0 raw words (0 effective words) took 0.0s, 0 effective words/s\n",
      "2018-04-25 22:51:32,070 : WARNING : EPOCH - 1 : supplied example count (0) did not equal expected count (75000)\n",
      "2018-04-25 22:51:32,184 : INFO : EPOCH - 2 : training on 0 raw words (0 effective words) took 0.0s, 0 effective words/s\n",
      "2018-04-25 22:51:32,252 : WARNING : EPOCH - 2 : supplied example count (0) did not equal expected count (75000)\n",
      "2018-04-25 22:51:32,333 : INFO : EPOCH - 3 : training on 0 raw words (0 effective words) took 0.0s, 0 effective words/s\n",
      "2018-04-25 22:51:32,408 : WARNING : EPOCH - 3 : supplied example count (0) did not equal expected count (75000)\n",
      "2018-04-25 22:51:32,540 : INFO : EPOCH - 4 : training on 0 raw words (0 effective words) took 0.0s, 0 effective words/s\n",
      "2018-04-25 22:51:32,637 : WARNING : EPOCH - 4 : supplied example count (0) did not equal expected count (75000)\n",
      "2018-04-25 22:51:32,768 : INFO : EPOCH - 5 : training on 0 raw words (0 effective words) took 0.0s, 0 effective words/s\n",
      "2018-04-25 22:51:32,825 : WARNING : EPOCH - 5 : supplied example count (0) did not equal expected count (75000)\n",
      "2018-04-25 22:51:32,989 : INFO : training on a 0 raw words (0 effective words) took 0.9s, 0 effective words/s\n"
     ]
    }
   ],
   "source": [
    "# Initialize and train the model (this will take some time)\n",
    "model = Word2Vec(all_sentences_sw,\n",
    "                 workers = num_workers,\n",
    "                 size = num_features,\n",
    "                 min_count = min_word_count, \n",
    "                 window = context,\n",
    "                 sample = downsampling,\n",
    "                 seed=1,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-04-25 22:51:32,999 : INFO : collecting all words and their counts\n",
      "2018-04-25 22:51:33,000 : INFO : PROGRESS: at sentence #0, processed 0 words and 0 word types\n",
      "2018-04-25 22:51:38,112 : INFO : PROGRESS: at sentence #10000, processed 2228586 words and 835332 word types\n",
      "2018-04-25 22:51:43,017 : INFO : PROGRESS: at sentence #20000, processed 4434587 words and 1392007 word types\n",
      "2018-04-25 22:51:48,078 : INFO : PROGRESS: at sentence #30000, processed 6632633 words and 1884563 word types\n",
      "2018-04-25 22:51:53,088 : INFO : PROGRESS: at sentence #40000, processed 8843751 words and 2341023 word types\n",
      "2018-04-25 22:51:58,192 : INFO : PROGRESS: at sentence #50000, processed 11083869 words and 2770663 word types\n",
      "2018-04-25 22:52:03,570 : INFO : PROGRESS: at sentence #60000, processed 13311069 words and 3169526 word types\n",
      "2018-04-25 22:52:08,564 : INFO : PROGRESS: at sentence #70000, processed 15517410 words and 3541604 word types\n",
      "2018-04-25 22:52:11,090 : INFO : collected 3722419 word types from a corpus of 16627625 words (unigram + bigrams) and 75000 sentences\n",
      "2018-04-25 22:52:11,091 : INFO : using 3722419 counts as vocab in Phrases<0 vocab, min_count=5, threshold=10.0, max_vocab_size=40000000>\n",
      "2018-04-25 22:52:11,092 : INFO : collecting all words and their counts\n",
      "2018-04-25 22:52:11,096 : INFO : PROGRESS: at sentence #0, processed 0 words, keeping 0 word types\n",
      "2018-04-25 22:52:25,233 : INFO : PROGRESS: at sentence #10000, processed 2088197 words, keeping 92630 word types\n",
      "2018-04-25 22:52:39,346 : INFO : PROGRESS: at sentence #20000, processed 4155139 words, keeping 132085 word types\n",
      "2018-04-25 22:52:53,440 : INFO : PROGRESS: at sentence #30000, processed 6214916 words, keeping 166343 word types\n",
      "2018-04-25 22:53:07,689 : INFO : PROGRESS: at sentence #40000, processed 8285847 words, keeping 196659 word types\n",
      "2018-04-25 22:53:22,174 : INFO : PROGRESS: at sentence #50000, processed 10384784 words, keeping 223833 word types\n",
      "2018-04-25 22:53:36,612 : INFO : PROGRESS: at sentence #60000, processed 12471603 words, keeping 248535 word types\n",
      "2018-04-25 22:53:50,871 : INFO : PROGRESS: at sentence #70000, processed 14538558 words, keeping 271408 word types\n",
      "2018-04-25 22:53:58,061 : INFO : collected 282543 word types from a corpus of 15578244 raw words and 75000 sentences\n",
      "2018-04-25 22:53:58,062 : INFO : Loading a fresh vocabulary\n",
      "2018-04-25 22:54:00,311 : INFO : min_count=5 retains 76350 unique words (27% of original 282543, drops 206193)\n",
      "2018-04-25 22:54:00,313 : INFO : min_count=5 leaves 15297357 word corpus (98% of original 15578244, drops 280887)\n",
      "2018-04-25 22:54:00,731 : INFO : deleting the raw counts dictionary of 282543 items\n",
      "2018-04-25 22:54:00,741 : INFO : sample=1e-05 downsamples 2930 most-common words\n",
      "2018-04-25 22:54:00,742 : INFO : downsampling leaves estimated 4600288 word corpus (30.1% of prior 15297357)\n",
      "2018-04-25 22:54:01,076 : INFO : estimated required memory for 76350 words and 300 dimensions: 221415000 bytes\n",
      "2018-04-25 22:54:01,077 : INFO : resetting layer weights\n",
      "2018-04-25 22:54:02,949 : INFO : training model with -1 workers on 76350 vocabulary and 300 features, using sg=0 hs=0 sample=1e-05 negative=5 window=10\n",
      "2018-04-25 22:54:02,956 : INFO : EPOCH - 1 : training on 0 raw words (0 effective words) took 0.0s, 0 effective words/s\n",
      "2018-04-25 22:54:02,983 : WARNING : EPOCH - 1 : supplied example count (0) did not equal expected count (75000)\n",
      "2018-04-25 22:54:03,046 : INFO : EPOCH - 2 : training on 0 raw words (0 effective words) took 0.0s, 0 effective words/s\n",
      "2018-04-25 22:54:03,323 : WARNING : EPOCH - 2 : supplied example count (0) did not equal expected count (75000)\n",
      "2018-04-25 22:54:04,940 : INFO : EPOCH - 3 : training on 0 raw words (0 effective words) took 0.0s, 0 effective words/s\n",
      "2018-04-25 22:54:05,267 : WARNING : EPOCH - 3 : supplied example count (0) did not equal expected count (75000)\n",
      "2018-04-25 22:54:05,365 : INFO : EPOCH - 4 : training on 0 raw words (0 effective words) took 0.0s, 0 effective words/s\n",
      "2018-04-25 22:54:05,561 : WARNING : EPOCH - 4 : supplied example count (0) did not equal expected count (75000)\n",
      "2018-04-25 22:54:06,115 : INFO : EPOCH - 5 : training on 0 raw words (0 effective words) took 0.0s, 0 effective words/s\n",
      "2018-04-25 22:54:06,178 : WARNING : EPOCH - 5 : supplied example count (0) did not equal expected count (75000)\n",
      "2018-04-25 22:54:06,196 : INFO : training on a 0 raw words (0 effective words) took 3.2s, 0 effective words/s\n"
     ]
    }
   ],
   "source": [
    "##Optionally converting the model for Bigrams (to capture more context):\n",
    "bigram_transformer = gensim.models.Phrases(all_sentences_sw)\n",
    "model = Word2Vec(bigram_transformer[all_sentences_sw],\n",
    "                 workers = num_workers,\n",
    "                 size = num_features,\n",
    "                 min_count = min_word_count, \n",
    "                 window = context,\n",
    "                 sample = downsampling,\n",
    "                 seed=1,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.getLogger('').setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you don't plan to train the model any further, calling init_sims will make the model much more memory-efficient.\n",
    "model.init_sims(replace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/gensim/utils.py:598: DeprecationWarning: Call to deprecated `cum_table` (Attribute will be removed in 4.0.0, use self.vocabulary.cum_table instead).\n",
      "  if hasattr(self, attrib):\n",
      "/usr/local/lib/python3.6/dist-packages/gensim/utils.py:599: DeprecationWarning: Call to deprecated `cum_table` (Attribute will be removed in 4.0.0, use self.vocabulary.cum_table instead).\n",
      "  asides[attrib] = getattr(self, attrib)\n",
      "/usr/local/lib/python3.6/dist-packages/gensim/utils.py:600: DeprecationWarning: Call to deprecated `cum_table` (Attribute will be removed in 4.0.0, use self.vocabulary.cum_table instead).\n",
      "  delattr(self, attrib)\n",
      "/usr/local/lib/python3.6/dist-packages/gensim/utils.py:555: DeprecationWarning: Call to deprecated `cum_table` (Attribute will be removed in 4.0.0, use self.vocabulary.cum_table instead).\n",
      "  setattr(obj, attrib, val)\n",
      "/usr/local/lib/python3.6/dist-packages/gensim/utils.py:491: DeprecationWarning: Call to deprecated `cum_table` (Attribute will be removed in 4.0.0, use self.vocabulary.cum_table instead).\n",
      "  setattr(self, attrib, None)\n"
     ]
    }
   ],
   "source": [
    "model_name = \"{}features_{}minwords_{}context\".format(num_features, min_word_count, context)\n",
    "model.save(os.path.join(outputs,model_name))\n",
    "model = Word2Vec.load(os.path.join(outputs,model_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exploring the Model Results\n",
    "\n",
    "Congratulations on making it successfully through everything so far! Let's take a look at the model we created out of our 75,000 training reviews.\n",
    "\n",
    "The \"doesnt_match\" function will try to deduce which word in a set is most dissimilar from the others:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "onion\n",
      "daughter\n",
      "germany\n",
      "[('ass', 0.2521023750305176), ('dreamlike', 0.23195180296897888), ('t.v_shows', 0.2257888913154602), ('pemberton', 0.2245239019393921), ('blissful', 0.2237330824136734), ('rhyming', 0.21968628466129303), ('swashbuckling', 0.21907924115657806), ('inhospitable', 0.21688570082187653), ('reb_brown', 0.2118222415447235), ('gowcaizer', 0.21170194447040558)]\n",
      "[('day-glo', 0.24879208207130432), ('starkly', 0.24478764832019806), ('futureworld', 0.22612175345420837), ('conscious', 0.219954714179039), ('audience.if', 0.21391691267490387), ('rescue_mission', 0.211456298828125), ('tennant', 0.21145150065422058), ('deeply_unsettling', 0.2096026986837387), ('two_seasons', 0.209297314286232), ('director_stuart', 0.20913293957710266)]\n",
      "[('miss', 0.24158237874507904), ('crime_wave', 0.23791907727718353), ('redemptive', 0.23433518409729004), ('summoning', 0.22931572794914246), ('fitter', 0.22097429633140564), ('8/10', 0.21520406007766724), ('18_months', 0.20744585990905762), ('tacked-on', 0.2070545256137848), ('wittiest', 0.20671504735946655), ('predicting', 0.2065007984638214)]\n"
     ]
    }
   ],
   "source": [
    "print(model.wv.doesnt_match(\"captain onion starship alien\".split()))\n",
    "print(model.wv.doesnt_match(\"father mother son daughter film\".split()))\n",
    "print(model.wv.doesnt_match(\"france england germany berlin\".split()))\n",
    "print(model.wv.most_similar(\"man\"))\n",
    "print(model.wv.most_similar(\"queen\"))\n",
    "print(model.wv.most_similar(\"awful\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So it seems we have a reasonably good model for semantic meaning - at least as good as Bag of Words. But how can we use these fancy distributed word vectors for supervised learning? The next section takes a stab at that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part 3: More Fun With Word Vectors\n",
    "--\n",
    "\n",
    "https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-3-more-fun-with-word-vectors  \n",
    "\n",
    "Numeric Representations of Words\n",
    "\n",
    "Now that we have a trained model with some semantic understanding of words, how should we use it? If you look beneath the hood, the Word2Vec model trained in Part 2 consists of a feature vector for each word in the vocabulary, stored in a numpy array called \"syn0\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/gensim/utils.py:491: DeprecationWarning: Call to deprecated `cum_table` (Attribute will be removed in 4.0.0, use self.vocabulary.cum_table instead).\n",
      "  setattr(self, attrib, None)\n"
     ]
    }
   ],
   "source": [
    "model = Word2Vec.load(os.path.join(outputs,model_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(76350, 300)"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.vectors.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of rows in syn0 is the number of words in the model's vocabulary, and the number of columns corresponds to the size of the feature vector, which we set in Part 2.  Setting the minimum word count to 40 gave us a total vocabulary of 16,492 words with 300 features apiece. Individual word vectors can be accessed in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4.2098176e-04,  7.0480525e-02,  7.3045544e-02,  2.8861310e-02,\n",
       "        9.1160800e-05,  5.9849393e-02,  1.1083634e-02, -6.0755208e-02,\n",
       "        1.5395473e-02, -5.8088291e-02], dtype=float32)"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv[\"flower\"][0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the', 'and', 'of', 'to', 'is', 'it', 'in', 'this', 'that', 'was']"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.index2word[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From Words To Paragraphs, \n",
    "--\n",
    "Attempt 1:  Vector Averaging  \n",
    "--\n",
    "\n",
    "One challenge with the IMDB dataset is the variable-length reviews. We need to find a way to take individual word vectors and transform them into a feature set that is the same length for every review.\n",
    "\n",
    "Since each word is a vector in 300-dimensional space, we can use vector operations to combine the words in each review. One method we tried was to simply average the word vectors in a given review (for this purpose, we removed stop words, which would just add noise).\n",
    "\n",
    "The following code averages the feature vectors, building on our code from Part 2. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "def makeFeatureVec(words, model, num_features):\n",
    "    # Function to average all of the word vectors in a given paragraph\n",
    "    # Pre-initialize an empty numpy array (for speed)\n",
    "    featureVec = np.zeros((num_features,),dtype=\"float32\")\n",
    "    nwords = 0.\n",
    "    # Index2word is a list that contains the names of the words in the model's vocabulary. \n",
    "    #Convert it to a set, for speed\n",
    "    index2word_set = set(model.wv.index2word)\n",
    "    # Loop over each word in the review and, if it is in the model's\n",
    "    # vocabulary, add its feature vector to the total\n",
    "    for word in words:\n",
    "        if word in index2word_set:\n",
    "            nwords = nwords + 1.\n",
    "            featureVec = np.add(featureVec,model.wv[word])\n",
    "    # Divide the result by the number of words to get the average\n",
    "    featureVec = np.divide(featureVec,nwords)\n",
    "    return featureVec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getAvgFeatureVecs(reviews, model, num_features):\n",
    "    # Given a set of reviews (each one a list of words), calculate\n",
    "    # the average feature vector for each one and return a 2D numpy array\n",
    "    # Initialize a counter\n",
    "    counter = 0.\n",
    "    # Preallocate a 2D numpy array, for speed\n",
    "    reviewFeatureVecs = np.zeros((len(reviews),num_features),dtype=\"float32\")\n",
    "    # Loop through the reviews\n",
    "    for review in reviews:\n",
    "        # Print a status message every 2000th review\n",
    "        if counter%2000. == 0.:\n",
    "            print(\"Review {} of {}\".format(counter, len(reviews)))\n",
    "        #Call the function (defined above) that makes average feature vectors\n",
    "        reviewFeatureVecs[int(counter)] = makeFeatureVec(review, model, num_features)\n",
    "        counter = counter + 1.\n",
    "    return reviewFeatureVecs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can call these functions to create average vectors for each paragraph. The following operations will take a few minutes:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review 0.0 of 25000\n",
      "Review 2000.0 of 25000\n",
      "Review 4000.0 of 25000\n",
      "Review 6000.0 of 25000\n",
      "Review 8000.0 of 25000\n",
      "Review 10000.0 of 25000\n",
      "Review 12000.0 of 25000\n",
      "Review 14000.0 of 25000\n",
      "Review 16000.0 of 25000\n",
      "Review 18000.0 of 25000\n",
      "Review 20000.0 of 25000\n",
      "Review 22000.0 of 25000\n",
      "Review 24000.0 of 25000\n"
     ]
    }
   ],
   "source": [
    "# Calculate average feature vectors for training and testing sets, using the functions \n",
    "# we defined above. Notice that we now use stop word removal.\n",
    "trainDataVecs = getAvgFeatureVecs(labeled_sentences_sw, model, num_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.00728554, -0.00903158,  0.00625665,  0.00996365, -0.00603685,\n",
       "        0.00619752, -0.00757262,  0.00559287,  0.00052192,  0.01074613],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainDataVecs[100][0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check for Nan values:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print(np.isnan(trainDataVecs).any()) #testando se não há valores que inviabilizam o treinamento\n",
    "print(np.isfinite(trainDataVecs).all())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If there are Nan values we can use inputer  \n",
    "from sklearn.preprocessing import Imputer\n",
    "trainDataVecs = Imputer().fit_transform(trainDataVecs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print(np.isnan(trainDataVecs).any()) #testando se não há valores que inviabilizam o treinamento\n",
    "print(np.isfinite(trainDataVecs).all())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, use the average paragraph vectors with the classifiers from Part 1.  \n",
    "Note that, as in Part 1, we can only use the labeled training reviews to train the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_traincvWV, X_testcvWV, y_traincvWV, y_testcvWV = model_selection.train_test_split(trainDataVecs,\n",
    "                                                                                    train[\"sentiment\"],\n",
    "                                                                                    test_size=0.2,\n",
    "                                                                                    random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.698\n"
     ]
    }
   ],
   "source": [
    "# Initialize a Random Forest classifier with 300 trees\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "clf_RF_WV = RandomForestClassifier(n_estimators=300, \n",
    "                                   criterion='gini', \n",
    "                                   max_depth=None, \n",
    "                                   min_samples_split=3, \n",
    "                                   min_samples_leaf=1, \n",
    "                                   min_weight_fraction_leaf=0.0, \n",
    "                                   max_features='auto', \n",
    "                                   max_leaf_nodes=None, \n",
    "                                   bootstrap=False, \n",
    "                                   oob_score=False, \n",
    "                                   n_jobs=-1, \n",
    "                                   random_state=None, \n",
    "                                   verbose=0, \n",
    "                                   warm_start=False, \n",
    "                                   class_weight=None).fit(X_traincvWV, y_traincvWV)\n",
    "\n",
    "eval_RF_WV_tts = clf_RF_WV.score(X_testcvWV, y_testcvWV)\n",
    "print(eval_RF_WV_tts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7032\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression as LR\n",
    "\n",
    "clf_LR_WV = LR(penalty='l2',\n",
    "               dual=False,\n",
    "               tol=0.0001,\n",
    "               C=1.0,\n",
    "               fit_intercept=True,\n",
    "               intercept_scaling=1,\n",
    "               class_weight=None,\n",
    "               random_state=None,\n",
    "               solver='liblinear',\n",
    "               max_iter=100,\n",
    "               multi_class='ovr',\n",
    "               verbose=0).fit(X_traincvWV, y_traincvWV)\n",
    "\n",
    "eval_LR_WV_tts = clf_LR_WV.score(X_testcvWV, y_testcvWV)\n",
    "print(eval_LR_WV_tts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a Submission  \n",
    "--"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review 0.0 of 25000\n",
      "Review 2000.0 of 25000\n",
      "Review 4000.0 of 25000\n",
      "Review 6000.0 of 25000\n",
      "Review 8000.0 of 25000\n",
      "Review 10000.0 of 25000\n",
      "Review 12000.0 of 25000\n",
      "Review 14000.0 of 25000\n",
      "Review 16000.0 of 25000\n",
      "Review 18000.0 of 25000\n",
      "Review 20000.0 of 25000\n",
      "Review 22000.0 of 25000\n",
      "Review 24000.0 of 25000\n"
     ]
    }
   ],
   "source": [
    "testDataVecs = getAvgFeatureVecs(test_labeled_sentences, model, num_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print(np.isnan(testDataVecs).any()) #testando se não há valores que inviabilizam o treinamento\n",
    "print(np.isfinite(testDataVecs).all())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import Imputer\n",
    "testDataVecs = Imputer().fit_transform(testDataVecs)\n",
    "\n",
    "print(np.isnan(testDataVecs).any()) #testando se não há valores que inviabilizam o treinamento\n",
    "print(np.isfinite(testDataVecs).all())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.00404975, -0.0137976 ,  0.00157094,  0.01829989,  0.00228919,\n",
       "        0.0006032 ,  0.00190888,  0.00233696,  0.00812894,  0.01035845],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "testDataVecs[0][0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 1 1 1 0 0 0 1 1 0]\n",
      "[[0.46333333 0.53666667]\n",
      " [0.47       0.53      ]\n",
      " [0.47833333 0.52166667]\n",
      " [0.465      0.535     ]\n",
      " [0.53       0.47      ]\n",
      " [0.59       0.41      ]\n",
      " [0.515      0.485     ]\n",
      " [0.435      0.565     ]\n",
      " [0.48833333 0.51166667]\n",
      " [0.54166667 0.45833333]]\n"
     ]
    }
   ],
   "source": [
    "# Use the random forest to make sentiment label predictions\n",
    "result = clf_RF_WV.predict(testDataVecs)\n",
    "result_prob = clf_RF_WV.predict_proba(testDataVecs)\n",
    "print(result[0:10])\n",
    "print(result_prob[0:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = pd.DataFrame(data={\"id\":test[\"id\"], \"sentiment\":result,})# \"probs\":result_prob[:,1]})\n",
    "# Use pandas to write the comma-separated output file\n",
    "output.to_csv(os.path.join(outputs,'Word2Vec_AverageVectors.csv'), index=False, quoting=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"12311_10\"</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"8348_2\"</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"5828_4\"</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"7186_2\"</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"12128_7\"</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id  sentiment\n",
       "0  \"12311_10\"          1\n",
       "1    \"8348_2\"          1\n",
       "2    \"5828_4\"          1\n",
       "3    \"7186_2\"          1\n",
       "4   \"12128_7\"          0"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We found that this produced results much better than chance, but underperformed Bag of Words by a few percentage points.\n",
    "\n",
    "Since the element-wise average of the vectors didn't produce spectacular results, perhaps we could do it in a more intelligent way? A standard way of weighting word vectors is to apply \"tf-idf\" weights, which measure how important a given word is within a given set of documents. One way to extract tf-idf weights in Python is by using scikit-learn's TfidfVectorizer, which has an interface similar to the CountVectorizer that we used in Part 1. However, when we tried weighting our word vectors in this way, we found no substantial improvement in performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From Words to Paragraphs, Attempt 2: Clustering \n",
    "--\n",
    "\n",
    "Word2Vec creates clusters of semantically related words, so another possible approach is to exploit the similarity of words within a cluster. Grouping vectors in this way is known as \"vector quantization.\" To accomplish this, we first need to find the centers of the word clusters, which we can do by using a clustering algorithm such as K-Means.\n",
    "\n",
    "In K-Means, the one parameter we need to set is \"K,\" or the number of clusters. How should we decide how many clusters to create? Trial and error suggested that small clusters, with an average of only 5 words or so per cluster, gave better results than large clusters with many words. Clustering code is given below. We use scikit-learn to perform our K-Means.\n",
    "\n",
    "K-Means clustering with large K can be very slow; the following code took more than 40 minutes on my computer. Below, we set a timer around the K-Means function to see how long it takes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(76350, 300)\n",
      "Initialization complete\n",
      "Initialization complete\n",
      "Initialization complete\n",
      "Initialization complete\n",
      "Initialization complete\n",
      "start iteration\n",
      "start iteration\n",
      "done sorting\n",
      "end inner loop\n",
      "start iteration\n",
      "done sorting\n",
      "done sorting\n",
      "end inner loop\n",
      "start iteration\n",
      "start iteration\n",
      "end inner loop\n",
      "done sorting\n",
      "done sorting\n",
      "end inner loop\n",
      "end inner loop\n",
      "Iteration 0, inertia 53462.824\n",
      "start iteration\n",
      "Iteration 0, inertia 53468.188\n",
      "start iteration\n",
      "Iteration 0, inertia 53467.52\n",
      "start iteration\n",
      "Iteration 0, inertia 53462.258\n",
      "start iteration\n",
      "Iteration 0, inertia 53454.434\n",
      "start iteration\n",
      "done sorting\n",
      "done sorting\n",
      "done sorting\n",
      "done sorting\n",
      "done sorting\n",
      "end inner loop\n",
      "end inner loop\n",
      "end inner loop\n",
      "end inner loop\n",
      "end inner loop\n",
      "Iteration 1, inertia 53462.824\n",
      "center shift 0.000000e+00 within tolerance 3.333265e-07\n",
      "Iteration 1, inertia 53462.258\n",
      "center shift 0.000000e+00 within tolerance 3.333265e-07\n",
      "Iteration 1, inertia 53468.188\n",
      "center shift 0.000000e+00 within tolerance 3.333265e-07\n",
      "Iteration 1, inertia 53454.434\n",
      "center shift 0.000000e+00 within tolerance 3.333265e-07\n",
      "Iteration 1, inertia 53467.52\n",
      "center shift 0.000000e+00 within tolerance 3.333265e-07\n",
      "Time taken for K Means clustering:  2511.2730469703674 seconds.\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "\n",
    "# Set \"k\" (num_clusters) to be 1/5th of the vocabulary size, or an average of 5 words per cluster\n",
    "word_vectors = model.wv.syn0\n",
    "print(word_vectors.shape)\n",
    "\n",
    "num_clusters = int(word_vectors.shape[0] / 5)\n",
    "\n",
    "# Initalize a k-means object and use it to extract centroids\n",
    "kmeans_clustering = KMeans(n_clusters = num_clusters,\n",
    "                           n_init=5,\n",
    "                           verbose=2,\n",
    "                           n_jobs=-2,\n",
    "                           random_state=0)\n",
    "\n",
    "idx = kmeans_clustering.fit_predict(word_vectors) # trimmed last line because of an error\n",
    "\n",
    "elapsed = time.time() - start\n",
    "print(\"Time taken for K Means clustering: \", elapsed, \"seconds.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cluster assignment for each word is now stored in idx, and the vocabulary from our original Word2Vec model is still stored in model.index2word. For convenience, we zip these into one dictionary as follows:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Word / Index dictionary, mapping each vocabulary word to a cluster number\n",
    "word_centroid_map = dict(zip(model.wv.index2word, idx))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a little abstract, so let's take a closer look at what our clusters contain. Your clusters may differ, as Word2Vec relies on a random number seed. Here is a loop that prints out the words for clusters 0 through 9:\n",
    "\n",
    "Run k-means on the word vectors and print a few clusters  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Cluster 0\n",
      "['closely', 'set-up', 'door_open', 'burglar', 'piano_tuner', 'oil_wells', 're-introduced', 'seel']\n",
      "\n",
      "Cluster 1\n",
      "['boom_mike', 'movingly', 'charlie_hall']\n",
      "\n",
      "Cluster 2\n",
      "['stranger', 'la_moustache', 'room_mate', 'sexual_shenanigans']\n",
      "\n",
      "Cluster 3\n",
      "['series', 'tenement', 'courtney_cox', 'st_bernard', 'bailed_out', 'preordained']\n",
      "\n",
      "Cluster 4\n",
      "['death_penalty', 'hobbit', 'dillinger', 'veteran_actor', 'faultless', 'glitzy', 'transfusion', 'trading_cards']\n",
      "\n",
      "Cluster 5\n",
      "['elmo', 'unimpressed', 'rural_america', 'immigrant_workers', 'parrish', 'baazigar']\n",
      "\n",
      "Cluster 6\n",
      "['lacy', 'unconditionally', 'mctiernan', 'david_attenborough']\n",
      "\n",
      "Cluster 7\n",
      "['fleet', 'bookends']\n",
      "\n",
      "Cluster 8\n",
      "['radio_station', 'rely', 'obligated', 'marquez', 'bruce_joel']\n",
      "\n",
      "Cluster 9\n",
      "['terrorist_group', 'welding', 'mannix', 'happening.the', 'hinged']\n"
     ]
    }
   ],
   "source": [
    "# Print the first ten clusters\n",
    "for cluster in range(0,10):\n",
    "    # Print the cluster number\n",
    "    print(\"\\nCluster {}\".format(cluster))\n",
    "    # Find all of the words for that cluster number, and print them out\n",
    "    words = []\n",
    "    for i in range(0,len(word_centroid_map.values())):\n",
    "        #print(len(word_centroid_map.values()))\n",
    "        #print(cluster)\n",
    "        #print(word_centroid_map.keys())\n",
    "        if(list(word_centroid_map.values())[i] == cluster):\n",
    "            words.append(list(word_centroid_map.keys())[i])\n",
    "    print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the clusters are of varying quality. Some make sense, some cointain mostly names, and some contain related adjectives. On the other hand, some are a little mystifying. Perhaps our algorithm works best on adjectives.\n",
    "\n",
    "At any rate, now we have a cluster (or \"centroid\") assignment for each word, and we can define a function to convert reviews into bags-of-centroids. This works just like Bag of Words but uses semantically related clusters instead of individual words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_bag_of_centroids(wordlist, word_centroid_map):\n",
    "    # The number of clusters is equal to the highest cluster index in the word / centroid map\n",
    "    num_centroids = max(word_centroid_map.values()) + 1\n",
    "    # Pre-allocate the bag of centroids vector (for speed)\n",
    "    bag_of_centroids = np.zeros(num_centroids, dtype=\"float32\")\n",
    "    # Loop over the words in the review. If the word is in the vocabulary,\n",
    "    # find which cluster it belongs to, and increment that cluster count by one\n",
    "    for word in wordlist:\n",
    "        if word in word_centroid_map:\n",
    "            index = word_centroid_map[word]\n",
    "            bag_of_centroids[index] += 1\n",
    "    return bag_of_centroids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function above will give us a numpy array for each review, each with a number of features equal to the number of clusters. Finally, we create bags of centroids for our training and test set, then train a random forest and extract results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ****** Create bags of centroids\n",
    "\n",
    "# Pre-allocate an array for the training set bags of centroids (for speed)\n",
    "train_centroids = np.zeros((train[\"review\"].size, num_clusters), dtype=\"float32\")\n",
    "\n",
    "# Transform the training set reviews into bags of centroids\n",
    "counter = 0\n",
    "for review in clean_train_reviews:\n",
    "    train_centroids[counter] = create_bag_of_centroids(review, word_centroid_map)\n",
    "    counter += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_traincvCT, X_testcvCT, y_traincvCT, y_testcvCT = model_selection.train_test_split(train_centroids,\n",
    "                                                                                    train[\"sentiment\"],\n",
    "                                                                                    test_size=0.2,\n",
    "                                                                                    random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a Random Forest classifier with 100 trees\n",
    "clf_RF_CT = RandomForestClassifier(n_estimators=100, \n",
    "                                   criterion='gini', \n",
    "                                   max_depth=None, \n",
    "                                   min_samples_split=2, \n",
    "                                   min_samples_leaf=1, \n",
    "                                   min_weight_fraction_leaf=0.0, \n",
    "                                   max_features='auto', \n",
    "                                   max_leaf_nodes=None, \n",
    "                                   bootstrap=True, \n",
    "                                   oob_score=False, \n",
    "                                   n_jobs=1, \n",
    "                                   random_state=None, \n",
    "                                   verbose=0, \n",
    "                                   warm_start=False, \n",
    "                                   class_weight=None).fit(X_traincvCT, y_traincvCT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4904\n"
     ]
    }
   ],
   "source": [
    "eval_RF_CT_tts = clf_RF_CT.score(X_testcvCT, y_testcvCT)\n",
    "print(eval_RF_CT_tts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Repeat for test reviews\n",
    "test_centroids = np.zeros((test[\"review\"].size, num_clusters), dtype=\"float32\")\n",
    "\n",
    "counter = 0\n",
    "for review in clean_test_reviews:\n",
    "    test_centroids[counter] = create_bag_of_centroids(review, word_centroid_map)\n",
    "    counter += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wrote BagOfCentroids.csv\n"
     ]
    }
   ],
   "source": [
    "result = clf_RF_CT.predict(test_centroids)\n",
    "\n",
    "# Write the test results\n",
    "output = pd.DataFrame(data={\"id\":test[\"id\"], \"sentiment\":result})\n",
    "output.to_csv(os.path.join(outputs,\"BagOfCentroids.csv\"), index=False, quoting=3)\n",
    "print(\"Wrote BagOfCentroids.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We found that the code above gives about the same (or slightly worse) results compared to the Bag of Words in Part 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Part 4: Comparing deep and non-deep learning methods\n",
    "--\n",
    "\n",
    "You may ask: Why is Bag of Words better?\n",
    "\n",
    "The biggest reason is, in our tutorial, averaging the vectors and using the centroids lose the order of words, making it very similar to the concept of Bag of Words. The fact that the performance is similar (within range of standard error) makes all three methods practically equivalent.  \n",
    "\n",
    "A few things to try:\n",
    "\n",
    "First, training Word2Vec on a lot more text should greatly improve performance. Google's results are based on word vectors that were learned out of more than a billion-word corpus; our labeled and unlabeled training sets together are only a measly 18 million words or so. Conveniently, Word2Vec provides functions to load any pre-trained model that is output by Google's original C tool, so it's also possible to train a model in C and then import it into Python.\n",
    "\n",
    "Second, in published literature, distributed word vector techniques have been shown to outperform Bag of Words models. In this paper, an algorithm called Paragraph Vector is used on the IMDB dataset to produce some of the most state-of-the-art results to date. In part, it does better than the approaches we try here because vector averaging and clustering lose the word order, whereas Paragraph Vectors preserves word order information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is Deep Learning?\n",
    "\n",
    "The term \"deep learning\" was coined in 2006, and refers to machine learning algorithms that have multiple non-linear layers and can learn feature hierarchies [1].\n",
    "\n",
    "Most modern machine learning relies on feature engineering or some level of domain knowledge to obtain good results. In deep learning systems, this is not the case -- instead, algorithms can automatically learn feature hierarchies, which represent objects in increasing levels of abstraction. Although the basic ingredients of many deep learning algorithms have been around for many years, they are currently increasing in popularity for many reasons, including advances in compute power, the falling cost of computing hardware, and advances in machine learning research.\n",
    "\n",
    "Deep learning algorithms can be categorized by their architecture (feed-forward, feed-back, or bi-directional) and training protocols (purely supervised, hybrid, or unsupervised) [2]. \n",
    "\n",
    "Some good background materials include:\n",
    "\n",
    "[1] \"Deep Learning for Signal and Information Processing\", by Li Deng and Dong Yu (out of Microsoft)\n",
    "\n",
    "[2] \"Deep Learning Tutorial\" (2013 Presentation by Yann LeCun and Marc'Aurelio Ranzato)\n",
    "\n",
    "Where Does Word2Vec Fit In?\n",
    "\n",
    "Word2Vec works in a way that is similar to deep approaches such as recurrent neural nets or deep neural nets, but it implements certain algorithms, such as hierarchical softmax, that make it computationally more efficient.  \n",
    "\n",
    "See Part 2 of this tutorial for more on Word2Vec, as well as this paper: Efficient Estimation of Word Representations in Vector Space\n",
    "\n",
    "In this tutorial, we use a hybrid approach to training -- consisting of an unsupervised piece (Word2Vec) followed by supervised learning (the Random Forest). \n",
    "\n",
    "Libraries and Packages \n",
    "\n",
    "The lists below should in no way be considered exhaustive.\n",
    "\n",
    "In Python:\n",
    "\n",
    "Theano offers very low-level, nuts and bolts functionality for building deep learning systems. You can also find some good tutorials on their site.  \n",
    "Caffe is a deep learning framework out of the Berkeley Vision and Learning Center.  \n",
    "Pylearn2 wraps Theano and seems slightly more user friendly.  \n",
    "OverFeat was used to win the Kaggle Cats and Dogs competition.  \n",
    "\n",
    "\n",
    "More Tutorials  \n",
    "The O'Reilly Blog has a series of deep learning articles and tutorials:  \n",
    "\n",
    "http://radar.oreilly.com/2014/07/what-is-deep-learning-and-why-should-you-care.html  \n",
    "http://radar.oreilly.com/2014/07/how-to-build-and-run-your-first-deep-learning-network.html  \n",
    "Webcast: How to Get Started with Deep Learning in Computer Vision  \n",
    "There are several tutorials using Theano as well.  \n",
    "\n",
    "If you want to dive into the weeds of creating a neural network from scratch, check out Geoffrey Hinton's Coursera course.\n",
    "\n",
    "For NLP, check out this recent lecture at Stanford: http://techtalks.tv/talks/deep-learning-for-nlp-without-magic-part-1/58414/  \n",
    "\n",
    "This free, online book also introduces neural nets for deep learning: http://neuralnetworksanddeeplearning.com/  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}