Notebooks/Python_Tasks/08_NLP_TopicModeling.ipynb

{"cells":[{"cell_type":"markdown","source":["### Topic Modeling"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"d5e0f22c-924f-4efc-8374-fb1edb74c2ee","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["!pip install -U -q gensim \n#!pip install -U -q mpld3\n!pip install -U -q pyldavis\n#!pip install -U -q beautifulsoup4"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"34ae6aa6-f4ed-41d1-b628-be5ab57cd37c","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["import re\nimport os\nimport codecs\nimport string\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib as mpl\n\nimport nltk\nfrom nltk.stem.wordnet import WordNetLemmatizer\nfrom nltk.tokenize import word_tokenize\nfrom nltk.tokenize.casual import casual_tokenize\n\n#from bs4 import BeautifulSoup\nfrom sklearn import feature_extraction\n\nimport gensim\nimport pyLDAvis\nimport pyLDAvis.gensim_models\n\nfrom IPython.display import display, Image\nfrom IPython.core.interactiveshell import InteractiveShell\n\n%matplotlib inline"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"95a36d92-33e8-47e0-abd1-eafeb55825f9","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"markdown","source":["### [Topic Modeling](http://www.cs.columbia.edu/~blei/topicmodeling.html)  \n\nAnalytics Industry is all about obtaining the “Information” from the data. With the growing amount of data in recent years, that too mostly unstructured, it’s difficult to obtain the relevant and desired information. But, technology has developed some powerful methods which can be used to mine through the data and fetch the information that we are looking for.  \n\nOne such technique in the field of text mining is Topic Modelling. As the name suggests, it is a process to automatically identify topics present in a text object and to derive hidden patterns exhibited by a text corpus. Thus, assisting better decision making.  \n\nTopic Modelling is different from rule-based text mining approaches that use regular expressions or dictionary based keyword searching techniques. It is an unsupervised approach used for finding and observing the bunch of words (called “topics”) in large clusters of texts.\n\n\n\nTopics can be defined as “a repeating pattern of co-occurring terms in a corpus”. A good topic model should result in – “health”, “doctor”, “patient”, “hospital” for a topic – Healthcare, and “farm”, “crops”, “wheat” for a topic – “Farming”.  \n\nTopic Models are very useful for the purpose for document clustering, organizing large blocks of textual data, information retrieval from unstructured text and feature selection. For Example – New York Times are using topic models to boost their user – article recommendation engines. Various professionals are using topic models for recruitment industries where they aim to extract latent features of job descriptions and map them to right candidates. They are being used to organize large datasets of emails, customer reviews, and user social media profiles."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"66a333e4-79bf-47e2-859b-dbe8542c029c","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["Image(url='https://www.analyticsvidhya.com/wp-content/uploads/2016/08/Modeling1.png')"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"e2708f2a-e90b-4f59-b420-a3fa5d1f2612","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"datasetInfos":[],"data":"<img src=\"https://www.analyticsvidhya.com/wp-content/uploads/2016/08/Modeling1.png\"/>","textData":null,"removedWidgets":[],"addedWidgets":{},"metadata":{},"type":"htmlSandbox","arguments":{}}},"output_type":"display_data","data":{"text/html":["<img src=\"https://www.analyticsvidhya.com/wp-content/uploads/2016/08/Modeling1.png\"/>"]}}],"execution_count":0},{"cell_type":"markdown","source":["#### 1 - Latent Dirichlet Allocation for Topic Modeling  \n\nThere are many approaches for obtaining topics from a text such as – Term Frequency and Inverse Document Frequency. NonNegative Matrix Factorization techniques. Latent Dirichlet Allocation is the most popular topic modeling technique and in this article, we will discuss the same.  \n\nLDA assumes documents are produced from a mixture of topics. Those topics then generate words based on their probability distribution. Given a dataset of documents, LDA backtracks and tries to figure out what topics would create those documents in the first place.  \n\nLDA is a matrix factorization technique. In vector space, any corpus (collection of documents) can be represented as a document-term matrix. The following matrix shows a corpus of N documents D1, D2, D3 … Dn and vocabulary size of M words W1,W2 .. Wn. The value of i,j cell gives the frequency count of word Wj in Document Di."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"06905fa4-b267-4326-be21-db460b892666","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["Image(url='https://www.analyticsvidhya.com/wp-content/uploads/2016/08/Modeling2.png')"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"a531cc59-0234-4c13-845f-b58ca7a23cd0","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"datasetInfos":[],"data":"<img src=\"https://www.analyticsvidhya.com/wp-content/uploads/2016/08/Modeling2.png\"/>","textData":null,"removedWidgets":[],"addedWidgets":{},"metadata":{},"type":"htmlSandbox","arguments":{}}},"output_type":"display_data","data":{"text/html":["<img src=\"https://www.analyticsvidhya.com/wp-content/uploads/2016/08/Modeling2.png\"/>"]}}],"execution_count":0},{"cell_type":"markdown","source":["LDA converts this Document-Term Matrix into two lower dimensional matrices – M1 and M2.\nM1 is a document-topics matrix and M2 is a topic – terms matrix with dimensions (N,  K) and (K, M) respectively, where N is the number of documents, K is the number of topics and M is the vocabulary size."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"3fa3564d-1ef7-49d0-a7e1-58445e594662","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["Image(url='https://www.analyticsvidhya.com/wp-content/uploads/2016/08/modeling3.png')"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"f3015879-03b1-4704-a740-77a2b93e6efd","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"datasetInfos":[],"data":"<img src=\"https://www.analyticsvidhya.com/wp-content/uploads/2016/08/modeling3.png\"/>","textData":null,"removedWidgets":[],"addedWidgets":{},"metadata":{},"type":"htmlSandbox","arguments":{}}},"output_type":"display_data","data":{"text/html":["<img src=\"https://www.analyticsvidhya.com/wp-content/uploads/2016/08/modeling3.png\"/>"]}}],"execution_count":0},{"cell_type":"markdown","source":["Notice that these two matrices already provides topic word and document topic distributions, However, these distribution needs to be improved, which is the main aim of LDA. LDA makes use of sampling techniques in order to improve these matrices.  \n\nIt Iterates through each word “w” for each document “d” and tries to adjust the current topic – word assignment with a new assignment. A new topic “k” is assigned to word “w” with a probability P which is a product of two probabilities p1 and p2.  \n\nFor every topic, two probabilities p1 and p2 are calculated. P1 – p(topic t / document d) = the proportion of words in document d that are currently assigned to topic t. P2 – p(word w / topic t) = the proportion of assignments to topic t over all documents that come from this word w.  \n\nThe current topic – word assignment is updated with a new topic with the probability, product of p1 and p2 . In this step, the model assumes that all the existing word – topic assignments except the current word are correct. This is essentially the probability that topic t generated word w, so it makes sense to adjust the current word’s topic with new probability.  \n\nAfter a number of iterations, a steady state is achieved where the document topic and topic term distributions are fairly good. This is the convergence point of LDA.  \n\n \nParameters of LDA  \n\nAlpha and Beta Hyperparameters – alpha represents document-topic density and Beta represents topic-word density. Higher the value of alpha, documents are composed of more topics and lower the value of alpha, documents contain fewer topics. On the other hand, higher the beta, topics are composed of a large number of words in the corpus, and with the lower value of beta, they are composed of few words.  \n\nNumber of Topics – Number of topics to be extracted from the corpus. Researchers have developed approaches to obtain an optimal number of topics by using Kullback Leibler Divergence Score. I will not discuss this in detail, as it is too mathematical. For understanding, one can refer to this[1] original paper on the use of KL divergence.  \n\nNumber of Topic Terms – Number of terms composed in a single topic. It is generally decided according to the requirement. If the problem statement talks about extracting themes or concepts, it is recommended to choose a higher number, if problem statement talks about extracting features or terms, a low number is recommended.  \n\nNumber of Iterations / passes – Maximum number of iterations allowed to LDA algorithm for convergence."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"1d9b4768-e5d6-4362-abbf-89d884a41fd2","inputWidgets":{},"title":""}}},{"cell_type":"markdown","source":["#### 2 - Topic Modeling with Gensim"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"e8529fb2-7ccf-4ddd-af4a-0d2e0dccd3db","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# Here are the sample documents combining together to form a corpus.\n\ndoc1 = \"Raiffeisen Bank International AG (RBI) regards Austria, where it is a leading corporate and investment bank, as well as Central and Eastern Europe (CEE) as its home market. 12 markets of the region are covered by subsidiary banks.\"\ndoc2 = \"RBI operates representative offices and service branches at selected Asian and Western European locations.\"\ndoc3 = \"RBI was already active in CEE even before the process of political transition started in the region and the 'Iron Curtain' fell: already back in 1986, its first subsidiary bank was founded in Hungary. Therefore, the bank looks back on more than 30 years of experience in the region's banking business.\"\ndoc4 = \"Meet Microsoft Windows 11: Learn how to use the new features of Windows 11 and see what makes it the best Windows yet.\"\ndoc5 = \"From the moment you start up, Windows 11 is on guard. It works in combination with your hardware and was designed with multiple layers of protection to help keep your apps, information and privacy secure.\"\ndoc6 = \"Mercedes-Benz and Microsoft collaborate to boost efficiency, resilience and sustainability in car production\"\n\n# compile documents\ndoc_complete = [doc1, doc2, doc3, doc4, doc5, doc6]"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"43a6f1ba-e002-472d-bb53-d5fd2cc821eb","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"markdown","source":["Cleaning and Preprocessing\n\nCleaning is an important step before any text mining task, in this step, we will remove the punctuations, stopwords and normalize the corpus."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"098062c8-e594-406c-991c-4446d0c7868a","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["#nltk.download('stopwords')\n#nltk.download('wordnet')"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"db892790-36a9-4879-8089-4d112ddf53ae","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["stopwords = [\n\"0o\", \"0s\", \"3a\", \"3b\", \"3d\", \"6b\", \"6o\", \"a\", \"a1\", \"a2\", \"a3\", \"a4\", \"ab\", \"able\", \"about\", \"above\", \"abst\", \"ac\", \"accordance\", \"according\", \"accordingly\", \"across\", \"act\", \"actually\", \"ad\", \"added\", \"adj\", \"ae\", \"af\", \"affected\", \"affecting\", \"affects\", \"after\", \"afterwards\", \"ag\", \"again\", \"against\", \"ah\", \"ain\", \"ain't\", \"aj\", \"al\", \"all\", \"allow\", \"allows\", \"almost\", \"alone\", \"along\", \"already\", \"also\", \"although\", \"always\", \"am\", \"among\", \"amongst\", \"amoungst\", \"amount\", \"an\", \"and\", \"announce\", \"another\", \"any\", \"anybody\", \"anyhow\", \"anymore\", \"anyone\", \"anything\", \"anyway\", \"anyways\", \"anywhere\", \"ao\", \"ap\", \"apart\", \"apparently\", \"appear\", \"appreciate\", \"appropriate\", \"approximately\", \"ar\", \"are\", \"aren\", \"arent\", \"aren't\", \"arise\", \"around\", \"as\", \"a's\", \"aside\", \"ask\", \"asking\", \"associated\", \"at\", \"au\", \"auth\", \"av\", \"available\", \"aw\", \"away\", \"awfully\", \"ax\", \"ay\", \"az\", \"b\", \"b1\", \"b2\", \"b3\", \"ba\", \"back\", \"bc\", \"bd\", \"be\", \"became\", \"because\", \"become\", \"becomes\", \"becoming\", \"been\", \"before\", \"beforehand\", \"begin\", \"beginning\", \"beginnings\", \"begins\", \"behind\", \"being\", \"believe\", \"below\", \"beside\", \"besides\", \"best\", \"better\", \"between\", \"beyond\", \"bi\", \"bill\", \"biol\", \"bj\", \"bk\", \"bl\", \"bn\", \"both\", \"bottom\", \"bp\", \"br\", \"brief\", \"briefly\", \"bs\", \"bt\", \"bu\", \"but\", \"bx\", \"by\", \"c\", \"c1\", \"c2\", \"c3\", \"ca\", \"call\", \"came\", \"can\", \"cannot\", \"cant\", \"can't\", \"cause\", \"causes\", \"cc\", \"cd\", \"ce\", \"certain\", \"certainly\", \"cf\", \"cg\", \"ch\", \"changes\", \"ci\", \"cit\", \"cj\", \"cl\", \"clearly\", \"cm\", \"c'mon\", \"cn\", \"co\", \"com\", \"come\", \"comes\", \"con\", \"concerning\", \"consequently\", \"consider\", \"considering\", \"contain\", \"containing\", \"contains\", \"corresponding\", \"could\", \"couldn\", \"couldnt\", \"couldn't\", \"course\", \"cp\", \"cq\", \"cr\", \"cry\", \"cs\", \"c's\", \"ct\", \"cu\", \"currently\", \"cv\", \"cx\", \"cy\", \"cz\", \"d\", \"d2\", \"da\", \"date\", \"dc\", \"dd\", \"de\", \"definitely\", \"describe\", \"described\", \"despite\", \"detail\", \"df\", \"di\", \"did\", \"didn\", \"didn't\", \"different\", \"dj\", \"dk\", \"dl\", \"do\", \"does\", \"doesn\", \"doesn't\", \"doing\", \"don\", \"done\", \"don't\", \"down\", \"downwards\", \"dp\", \"dr\", \"ds\", \"dt\", \"du\", \"due\", \"during\", \"dx\", \"dy\", \"e\", \"e2\", \"e3\", \"ea\", \"each\", \"ec\", \"ed\", \"edu\", \"ee\", \"ef\", \"effect\", \"eg\", \"ei\", \"eight\", \"eighty\", \"either\", \"ej\", \"el\", \"eleven\", \"else\", \"elsewhere\", \"em\", \"empty\", \"en\", \"end\", \"ending\", \"enough\", \"entirely\", \"eo\", \"ep\", \"eq\", \"er\", \"es\", \"especially\", \"est\", \"et\", \"et-al\", \"etc\", \"eu\", \"ev\", \"even\", \"ever\", \"every\", \"everybody\", \"everyone\", \"everything\", \"everywhere\", \"ex\", \"exactly\", \"example\", \"except\", \"ey\", \"f\", \"f2\", \"fa\", \"far\", \"fc\", \"few\", \"ff\", \"fi\", \"fifteen\", \"fifth\", \"fify\", \"fill\", \"find\", \"fire\", \"first\", \"five\", \"fix\", \"fj\", \"fl\", \"fn\", \"fo\", \"followed\", \"following\", \"follows\", \"for\", \"former\", \"formerly\", \"forth\", \"forty\", \"found\", \"four\", \"fr\", \"from\", \"front\", \"fs\", \"ft\", \"fu\", \"full\", \"further\", \"furthermore\", \"fy\", \"g\", \"ga\", \"gave\", \"ge\", \"get\", \"gets\", \"getting\", \"gi\", \"give\", \"given\", \"gives\", \"giving\", \"gj\", \"gl\", \"go\", \"goes\", \"going\", \"gone\", \"got\", \"gotten\", \"gr\", \"greetings\", \"gs\", \"gy\", \"h\", \"h2\", \"h3\", \"had\", \"hadn\", \"hadn't\", \"happens\", \"hardly\", \"has\", \"hasn\", \"hasnt\", \"hasn't\", \"have\", \"haven\", \"haven't\", \"having\", \"he\", \"hed\", \"he'd\", \"he'll\", \"hello\", \"help\", \"hence\", \"her\", \"here\", \"hereafter\", \"hereby\", \"herein\", \"heres\", \"here's\", \"hereupon\", \"hers\", \"herself\", \"hes\", \"he's\", \"hh\", \"hi\", \"hid\", \"him\", \"himself\", \"his\", \"hither\", \"hj\", \"ho\", \"home\", \"hopefully\", \"how\", \"howbeit\", \"however\", \"how's\", \"hr\", \"hs\", \"http\", \"hu\", \"hundred\", \"hy\", \"i\", \"i2\", \"i3\", \"i4\", \"i6\", \"i7\", \"i8\", \"ia\", \"ib\", \"ibid\", \"ic\", \"id\", \"i'd\", \"ie\", \"if\", \"ig\", \"ignored\", \"ih\", \"ii\", \"ij\", \"il\", \"i'll\", \"im\", \"i'm\", \"immediate\", \"immediately\", \"importance\", \"important\", \"in\", \"inasmuch\", \"inc\", \"indeed\", \"index\", \"indicate\", \"indicated\", \"indicates\", \"information\", \"inner\", \"insofar\", \"instead\", \"interest\", \"into\", \"invention\", \"inward\", \"io\", \"ip\", \"iq\", \"ir\", \"is\", \"isn\", \"isn't\", \"it\", \"itd\", \"it'd\", \"it'll\", \"its\", \"it's\", \"itself\", \"iv\", \"i've\", \"ix\", \"iy\", \"iz\", \"j\", \"jj\", \"jr\", \"js\", \"jt\", \"ju\", \"just\", \"k\", \"ke\", \"keep\", \"keeps\", \"kept\", \"kg\", \"kj\", \"km\", \"know\", \"known\", \"knows\", \"ko\", \"l\", \"l2\", \"la\", \"largely\", \"last\", \"lately\", \"later\", \"latter\", \"latterly\", \"lb\", \"lc\", \"le\", \"least\", \"les\", \"less\", \"lest\", \"let\", \"lets\", \"let's\", \"lf\", \"like\", \"liked\", \"likely\", \"line\", \"little\", \"lj\", \"ll\", \"ll\", \"ln\", \"lo\", \"look\", \"looking\", \"looks\", \"los\", \"lr\", \"ls\", \"lt\", \"ltd\", \"m\", \"m2\", \"ma\", \"made\", \"mainly\", \"make\", \"makes\", \"many\", \"may\", \"maybe\", \"me\", \"mean\", \"means\", \"meantime\", \"meanwhile\", \"merely\", \"mg\", \"might\", \"mightn\", \"mightn't\", \"mill\", \"million\", \"mine\", \"miss\", \"ml\", \"mn\", \"mo\", \"more\", \"moreover\", \"most\", \"mostly\", \"move\", \"mr\", \"mrs\", \"ms\", \"mt\", \"mu\", \"much\", \"mug\", \"must\", \"mustn\", \"mustn't\", \"my\", \"myself\", \"n\", \"n2\", \"na\", \"name\", \"namely\", \"nay\", \"nc\", \"nd\", \"ne\", \"near\", \"nearly\", \"necessarily\", \"necessary\", \"need\", \"needn\", \"needn't\", \"needs\", \"neither\", \"never\", \"nevertheless\", \"new\", \"next\", \"ng\", \"ni\", \"nine\", \"ninety\", \"nj\", \"nl\", \"nn\", \"no\", \"nobody\", \"non\", \"none\", \"nonetheless\", \"noone\", \"nor\", \"normally\", \"nos\", \"not\", \"noted\", \"nothing\", \"novel\", \"now\", \"nowhere\", \"nr\", \"ns\", \"nt\", \"ny\", \"o\", \"oa\", \"ob\", \"obtain\", \"obtained\", \"obviously\", \"oc\", \"od\", \"of\", \"off\", \"often\", \"og\", \"oh\", \"oi\", \"oj\", \"ok\", \"okay\", \"ol\", \"old\", \"om\", \"omitted\", \"on\", \"once\", \"one\", \"ones\", \"only\", \"onto\", \"oo\", \"op\", \"oq\", \"or\", \"ord\", \"os\", \"ot\", \"other\", \"others\", \"otherwise\", \"ou\", \"ought\", \"our\", \"ours\", \"ourselves\", \"out\", \"outside\", \"over\", \"overall\", \"ow\", \"owing\", \"own\", \"ox\", \"oz\", \"p\", \"p1\", \"p2\", \"p3\", \"page\", \"pagecount\", \"pages\", \"par\", \"part\", \"particular\", \"particularly\", \"pas\", \"past\", \"pc\", \"pd\", \"pe\", \"per\", \"perhaps\", \"pf\", \"ph\", \"pi\", \"pj\", \"pk\", \"pl\", \"placed\", \"please\", \"plus\", \"pm\", \"pn\", \"po\", \"poorly\", \"possible\", \"possibly\", \"potentially\", \"pp\", \"pq\", \"pr\", \"predominantly\", \"present\", \"presumably\", \"previously\", \"primarily\", \"probably\", \"promptly\", \"proud\", \"provides\", \"ps\", \"pt\", \"pu\", \"put\", \"py\", \"q\", \"qj\", \"qu\", \"que\", \"quickly\", \"quite\", \"qv\", \"r\", \"r2\", \"ra\", \"ran\", \"rather\", \"rc\", \"rd\", \"re\", \"readily\", \"really\", \"reasonably\", \"recent\", \"recently\", \"ref\", \"refs\", \"regarding\", \"regardless\", \"regards\", \"related\", \"relatively\", \"research\", \"research-articl\", \"respectively\", \"resulted\", \"resulting\", \"results\", \"rf\", \"rh\", \"ri\", \"right\", \"rj\", \"rl\", \"rm\", \"rn\", \"ro\", \"rq\", \"rr\", \"rs\", \"rt\", \"ru\", \"run\", \"rv\", \"ry\", \"s\", \"s2\", \"sa\", \"said\", \"same\", \"saw\", \"say\", \"saying\", \"says\", \"sc\", \"sd\", \"se\", \"sec\", \"second\", \"secondly\", \"section\", \"see\", \"seeing\", \"seem\", \"seemed\", \"seeming\", \"seems\", \"seen\", \"self\", \"selves\", \"sensible\", \"sent\", \"serious\", \"seriously\", \"seven\", \"several\", \"sf\", \"shall\", \"shan\", \"shan't\", \"she\", \"shed\", \"she'd\", \"she'll\", \"shes\", \"she's\", \"should\", \"shouldn\", \"shouldn't\", \"should've\", \"show\", \"showed\", \"shown\", \"showns\", \"shows\", \"si\", \"side\", \"significant\", \"significantly\", \"similar\", \"similarly\", \"since\", \"sincere\", \"six\", \"sixty\", \"sj\", \"sl\", \"slightly\", \"sm\", \"sn\", \"so\", \"some\", \"somebody\", \"somehow\", \"someone\", \"somethan\", \"something\", \"sometime\", \"sometimes\", \"somewhat\", \"somewhere\", \"soon\", \"sorry\", \"sp\", \"specifically\", \"specified\", \"specify\", \"specifying\", \"sq\", \"sr\", \"ss\", \"st\", \"still\", \"stop\", \"strongly\", \"sub\", \"substantially\", \"successfully\", \"such\", \"sufficiently\", \"suggest\", \"sup\", \"sure\", \"sy\", \"system\", \"sz\", \"t\", \"t1\", \"t2\", \"t3\", \"take\", \"taken\", \"taking\", \"tb\", \"tc\", \"td\", \"te\", \"tell\", \"ten\", \"tends\", \"tf\", \"th\", \"than\", \"thank\", \"thanks\", \"thanx\", \"that\", \"that'll\", \"thats\", \"that's\", \"that've\", \"the\", \"their\", \"theirs\", \"them\", \"themselves\", \"then\", \"thence\", \"there\", \"thereafter\", \"thereby\", \"thered\", \"therefore\", \"therein\", \"there'll\", \"thereof\", \"therere\", \"theres\", \"there's\", \"thereto\", \"thereupon\", \"there've\", \"these\", \"they\", \"theyd\", \"they'd\", \"they'll\", \"theyre\", \"they're\", \"they've\", \"thickv\", \"thin\", \"think\", \"third\", \"this\", \"thorough\", \"thoroughly\", \"those\", \"thou\", \"though\", \"thoughh\", \"thousand\", \"three\", \"throug\", \"through\", \"throughout\", \"thru\", \"thus\", \"ti\", \"til\", \"tip\", \"tj\", \"tl\", \"tm\", \"tn\", \"to\", \"together\", \"too\", \"took\", \"top\", \"toward\", \"towards\", \"tp\", \"tq\", \"tr\", \"tried\", \"tries\", \"truly\", \"try\", \"trying\", \"ts\", \"t's\", \"tt\", \"tv\", \"twelve\", \"twenty\", \"twice\", \"two\", \"tx\", \"u\", \"u201d\", \"ue\", \"ui\", \"uj\", \"uk\", \"um\", \"un\", \"under\", \"unfortunately\", \"unless\", \"unlike\", \"unlikely\", \"until\", \"unto\", \"uo\", \"up\", \"upon\", \"ups\", \"ur\", \"us\", \"use\", \"used\", \"useful\", \"usefully\", \"usefulness\", \"uses\", \"using\", \"usually\", \"ut\", \"v\", \"va\", \"value\", \"various\", \"vd\", \"ve\", \"ve\", \"very\", \"via\", \"viz\", \"vj\", \"vo\", \"vol\", \"vols\", \"volumtype\", \"vq\", \"vs\", \"vt\", \"vu\", \"w\", \"wa\", \"want\", \"wants\", \"was\", \"wasn\", \"wasnt\", \"wasn't\", \"way\", \"we\", \"wed\", \"we'd\", \"welcome\", \"well\", \"we'll\", \"well-b\", \"went\", \"were\", \"we're\", \"weren\", \"werent\", \"weren't\", \"we've\", \"what\", \"whatever\", \"what'll\", \"whats\", \"what's\", \"when\", \"whence\", \"whenever\", \"when's\", \"where\", \"whereafter\", \"whereas\", \"whereby\", \"wherein\", \"wheres\", \"where's\", \"whereupon\", \"wherever\", \"whether\", \"which\", \"while\", \"whim\", \"whither\", \"who\", \"whod\", \"whoever\", \"whole\", \"who'll\", \"whom\", \"whomever\", \"whos\", \"who's\", \"whose\", \"why\", \"why's\", \"wi\", \"widely\", \"will\", \"willing\", \"wish\", \"with\", \"within\", \"without\", \"wo\", \"won\", \"wonder\", \"wont\", \"won't\", \"words\", \"world\", \"would\", \"wouldn\", \"wouldnt\", \"wouldn't\", \"www\", \"x\", \"x1\", \"x2\", \"x3\", \"xf\", \"xi\", \"xj\", \"xk\", \"xl\", \"xn\", \"xo\", \"xs\", \"xt\", \"xv\", \"xx\", \"y\", \"y2\", \"yes\", \"yet\", \"yj\", \"yl\", \"you\", \"youd\", \"you'd\", \"you'll\", \"your\", \"youre\", \"you're\", \"yours\", \"yourself\", \"yourselves\", \"you've\", \"yr\", \"ys\", \"yt\", \"z\", \"zero\", \"zi\", \"zz\",\n]"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"d15ea6ad-a57c-46b6-ae4b-fbae78aad6c2","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["#stopwords = nltk.corpus.stopwords.words('english')\nexclude = set(string.punctuation)\n#lemma = WordNetLemmatizer()\ndef clean(doc):\n    doc = \" \".join([i for i in doc.lower().split() if i not in stopwords])\n    doc = ''.join(ch for ch in doc if ch not in exclude)\n    #doc = \" \".join(lemma.lemmatize(word) for word in doc.split())\n    return doc\n\ndoc_clean = [clean(doc).split() for doc in doc_complete]   "],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"809e474d-cfb5-419d-8138-3571ef34f4de","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"markdown","source":["Preparing Document-Term Matrix\n\nAll the text documents combined is known as the corpus. To run any mathematical model on text corpus, it is a good practice to convert it into a matrix representation. LDA model looks for repeating term patterns in the entire DT matrix. Python provides many great libraries for text mining practices, “gensim” is one such clean and beautiful library to handle text data. It is scalable, robust and efficient. Following code shows how to convert a corpus into a document-term matrix."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"7d8abbf6-00df-4aec-873f-00ddee09d98b","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["import gensim\nfrom gensim import corpora\n\n# Creating the term dictionary of our courpus, where every unique term is assigned an index. \ndictionary = corpora.Dictionary(doc_clean)\n\n# Converting list of documents (corpus) into Document Term Matrix using dictionary prepared above.\ndoc_term_matrix = [dictionary.doc2bow(doc) for doc in doc_clean]"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"e99fc876-1c9a-4ed6-90dc-eb06ca4b7852","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"markdown","source":["Running LDA Model  \n\nNext step is to create an object for LDA model and train it on Document-Term matrix. The training also requires few parameters as input which are explained in the above section. The gensim module allows both LDA model estimation from a training corpus and inference of topic distribution on new, unseen documents."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"2ca584dc-c9ad-4be0-a2d1-e8c900d4b0c6","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# Creating the object for LDA model using gensim library\nLda = gensim.models.ldamodel.LdaModel\n\n# Running and Trainign LDA model on the document term matrix.\nldamodel = Lda(doc_term_matrix, num_topics=3, id2word = dictionary, passes=1000)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"50f6272e-fc2a-4fe7-9d19-4d296244a68d","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["topics = ldamodel.print_topics(num_topics=3, num_words=7)\nfor topic in topics:\n    print(topic)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"ed4991dd-db52-4f8c-8ef0-0367ccf8795d","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"datasetInfos":[],"data":"(0, '0.040*\"bank\" + 0.031*\"of\" + 0.031*\"as\" + 0.021*\"rbi\" + 0.021*\"was\" + 0.021*\"back\" + 0.021*\"already\"')\n(1, '0.037*\"with\" + 0.021*\"to\" + 0.021*\"of\" + 0.021*\"privacy\" + 0.021*\"secure\" + 0.021*\"protection\" + 0.021*\"moment\"')\n(2, '0.043*\"windows\" + 0.030*\"11\" + 0.030*\"microsoft\" + 0.030*\"to\" + 0.017*\"see\" + 0.017*\"new\" + 0.017*\"use\"')\n","removedWidgets":[],"addedWidgets":{},"metadata":{},"type":"ansi","arguments":{}}},"output_type":"display_data","data":{"text/plain":["(0, '0.040*\"bank\" + 0.031*\"of\" + 0.031*\"as\" + 0.021*\"rbi\" + 0.021*\"was\" + 0.021*\"back\" + 0.021*\"already\"')\n(1, '0.037*\"with\" + 0.021*\"to\" + 0.021*\"of\" + 0.021*\"privacy\" + 0.021*\"secure\" + 0.021*\"protection\" + 0.021*\"moment\"')\n(2, '0.043*\"windows\" + 0.030*\"11\" + 0.030*\"microsoft\" + 0.030*\"to\" + 0.017*\"see\" + 0.017*\"new\" + 0.017*\"use\"')\n"]}}],"execution_count":0},{"cell_type":"code","source":["pyLDAvis.enable_notebook()\npyLDAvis.gensim_models.prepare(ldamodel, doc_term_matrix, dictionary)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"a98adf00-f2ca-4ece-9a23-13b25cb0f80a","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"datasetInfos":[],"data":"/local_disk0/.ephemeral_nfs/envs/pythonEnv-0466b1ed-0153-4f8f-ad28-5f48458fabfd/lib/python3.9/site-packages/pyLDAvis/_prepare.py:246: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only\n  default_term_info = default_term_info.sort_values(\n","removedWidgets":[],"addedWidgets":{},"metadata":{},"type":"ansi","arguments":{}}},"output_type":"display_data","data":{"text/plain":["/local_disk0/.ephemeral_nfs/envs/pythonEnv-0466b1ed-0153-4f8f-ad28-5f48458fabfd/lib/python3.9/site-packages/pyLDAvis/_prepare.py:246: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only\n  default_term_info = default_term_info.sort_values(\n"]}},{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"datasetInfos":[],"data":"\n<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v1.0.0.css\">\n\n\n<div id=\"ldavis_el67301401007344292168336646043\"></div>\n<script type=\"text/javascript\">\n\nvar ldavis_el67301401007344292168336646043_data = {\"mdsDat\": {\"x\": [-0.10874492631132812, 0.08322210023430685, 0.02552282607702126], \"y\": [-0.02025003929606211, -0.04712238241760796, 0.06737242171367007], \"topics\": [1, 2, 3], \"cluster\": [1, 1, 1], \"Freq\": [51.76629595208443, 28.850858374738948, 19.38284567317662]}, \"tinfo\": {\"Term\": [\"windows\", \"with\", \"11\", \"to\", \"microsoft\", \"designed\", \"start\", \"layers\", \"combination\", \"moment\", \"multiple\", \"privacy\", \"protection\", \"information\", \"secure\", \"works\", \"up\", \"you\", \"help\", \"hardware\", \"guard\", \"apps\", \"from\", \"keep\", \"bank\", \"was\", \"yet\", \"use\", \"what\", \"best\", \"bank\", \"as\", \"already\", \"back\", \"cee\", \"its\", \"region\", \"subsidiary\", \"1986\", \"30\", \"active\", \"banking\", \"before\", \"business\", \"curtain\", \"even\", \"experience\", \"fell\", \"first\", \"founded\", \"hungary\", \"iron\", \"looks\", \"more\", \"political\", \"process\", \"regions\", \"started\", \"than\", \"therefore\", \"of\", \"was\", \"rbi\", \"microsoft\", \"windows\", \"best\", \"features\", \"how\", \"learn\", \"makes\", \"meet\", \"new\", \"see\", \"use\", \"what\", \"yet\", \"asian\", \"at\", \"branches\", \"european\", \"locations\", \"offices\", \"operates\", \"representative\", \"selected\", \"service\", \"western\", \"boost\", \"car\", \"collaborate\", \"efficiency\", \"mercedesbenz\", \"production\", \"11\", \"to\", \"with\", \"apps\", \"combination\", \"designed\", \"from\", \"guard\", \"hardware\", \"help\", \"information\", \"keep\", \"layers\", \"moment\", \"multiple\", \"privacy\", \"protection\", \"secure\", \"start\", \"up\", \"works\", \"you\", \"to\", \"11\", \"was\", \"windows\", \"of\", \"boost\", \"car\", \"collaborate\", \"efficiency\", \"mercedesbenz\", \"rbi\"], \"Freq\": [2.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 2.0, 1.0, 1.0, 1.0, 1.0, 2.8603709333189498, 2.2002453829935726, 1.5401451637540295, 1.5401451637540295, 1.5401353797896598, 1.5401353797896598, 1.5401353797896598, 1.5401353797896598, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 2.2040311070701333, 1.5403701949345345, 1.5430601150016794, 1.1996938865059632, 1.7151447024959745, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854025933562556, 0.6854025933562556, 0.6854025933562556, 0.6854025933562556, 0.6854025933562556, 0.6854025933562556, 1.1999556250895118, 1.1993167409271, 0.9976623700774936, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5705039090623786, 0.5699872681432525, 0.569983805469778, 0.5691906023257559, 0.5702602673273269, 0.14265498342890257, 0.14265498342890257, 0.14265498342890257, 0.14265498342890257, 0.14265498342890257, 0.14267661259219963], \"Total\": [2.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 2.0, 1.0, 1.0, 1.0, 1.0, 3.1743973366535805, 2.514279514888329, 1.8541678256374243, 1.8541678256374243, 1.8541648639800468, 1.8541648639800468, 1.8541648639800468, 1.8541648639800468, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 3.4569114266647407, 2.281843977479953, 2.3690150852655014, 1.5624325091521065, 2.5044619046124246, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.9900707327723683, 1.9899708385161876, 1.389181405879746, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 1.9899708385161876, 1.9900707327723683, 2.281843977479953, 2.5044619046124246, 3.4569114266647407, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 2.3690150852655014], \"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\"], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -3.2251, -3.4875, -3.8442, -3.8442, -3.8442, -3.8442, -3.8442, -3.8442, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -3.4858, -3.844, -3.8423, -3.5094, -3.1519, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0692, -4.0692, -4.0692, -4.0692, -4.0692, -4.0692, -3.5092, -3.5097, -3.296, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8549, -3.8558, -3.8558, -3.8572, -3.8554, -5.241, -5.241, -5.241, -5.241, -5.241, -5.2409], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 0.5543, 0.525, 0.4729, 0.4729, 0.4729, 0.4729, 0.4729, 0.4729, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.2083, 0.2655, 0.2297, 0.9789, 0.8645, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8182, 0.8182, 0.8182, 0.8182, 0.8182, 0.8182, 0.7371, 0.7367, 1.3097, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 0.3914, 0.3905, 0.2537, 0.1592, -0.1613, -0.3536, -0.3536, -0.3536, -0.3536, -0.3536, -1.1689]}, \"token.table\": {\"Topic\": [2, 3, 1, 1, 1, 1, 3, 1, 2, 2, 1, 1, 1, 1, 2, 2, 2, 1, 2, 1, 2, 3, 1, 3, 2, 2, 1, 1, 2, 1, 1, 1, 3, 3, 3, 3, 2, 1, 3, 1, 1, 3, 3, 2, 2, 1, 2, 2, 2, 2, 3, 1, 3, 2, 1, 2, 3, 2, 2, 1, 3, 1, 2, 3, 1, 2, 1, 1, 2, 3, 2, 2, 2, 3, 1, 1, 1, 1, 2, 3, 3, 2, 1, 3, 2, 2, 2, 3, 3, 3, 2, 3], \"Freq\": [0.5024947020887542, 0.5024947020887542, 0.8374862532846978, 0.8374862532846978, 0.8374862532846978, 1.0786510111685503, 1.0399455841290934, 0.7954565067873249, 0.9540126735326901, 0.9540126735326901, 1.0786510111685503, 0.945061276784768, 0.8374862532846978, 0.8374862532846978, 0.9540139697638006, 0.9540145715819685, 0.9540126735326901, 0.8374862532846978, 0.9540145715819685, 1.0786527340977174, 0.9540145715819685, 1.0399455841290934, 0.8374862532846978, 1.0399455841290934, 0.9540145715819685, 0.9540126735326901, 0.8374862532846978, 0.8374862532846978, 0.9540139697638006, 0.8374862532846978, 0.8374862532846978, 0.8374862532846978, 1.0399455841290934, 1.0399455841290934, 1.0399455841290934, 1.0399455841290934, 0.9540139697638006, 0.8374862532846978, 1.0399455841290934, 0.8374862532846978, 1.0786527340977174, 1.0399455841290934, 1.0399455841290934, 0.9540139697638006, 0.9540126735326901, 0.8374862532846978, 0.9540139697638006, 0.9540139697638006, 0.9540145715819685, 0.6400276454454185, 1.0399455841290934, 0.8374862532846978, 1.0399455841290934, 0.9540139697638006, 0.5785511264688716, 0.2892755632344358, 0.2892755632344358, 0.9540126735326901, 0.9540126735326901, 0.8374862532846978, 1.0399455841290934, 0.8374862532846978, 0.9540145715819685, 1.0399455841290934, 0.8442326992509864, 0.4221163496254932, 1.0786527340977174, 0.8374862532846978, 0.9540126735326901, 1.0399455841290934, 0.9540139697638006, 0.9540126735326901, 0.9540126735326901, 1.0399455841290934, 0.8374862532846978, 1.0786527340977174, 0.8374862532846978, 0.8374862532846978, 0.5025199267470901, 0.5025199267470901, 1.0399455841290934, 0.9540139697638006, 0.8764841153639178, 0.4382420576819589, 0.9540126735326901, 0.9540139697638006, 0.7985747342838931, 0.39928736714194657, 0.7198483911226239, 1.0399455841290934, 0.9540139697638006, 1.0399455841290934], \"Term\": [\"11\", \"11\", \"1986\", \"30\", \"active\", \"already\", \"apps\", \"as\", \"asian\", \"at\", \"back\", \"bank\", \"banking\", \"before\", \"best\", \"boost\", \"branches\", \"business\", \"car\", \"cee\", \"collaborate\", \"combination\", \"curtain\", \"designed\", \"efficiency\", \"european\", \"even\", \"experience\", \"features\", \"fell\", \"first\", \"founded\", \"from\", \"guard\", \"hardware\", \"help\", \"how\", \"hungary\", \"information\", \"iron\", \"its\", \"keep\", \"layers\", \"learn\", \"locations\", \"looks\", \"makes\", \"meet\", \"mercedesbenz\", \"microsoft\", \"moment\", \"more\", \"multiple\", \"new\", \"of\", \"of\", \"of\", \"offices\", \"operates\", \"political\", \"privacy\", \"process\", \"production\", \"protection\", \"rbi\", \"rbi\", \"region\", \"regions\", \"representative\", \"secure\", \"see\", \"selected\", \"service\", \"start\", \"started\", \"subsidiary\", \"than\", \"therefore\", \"to\", \"to\", \"up\", \"use\", \"was\", \"was\", \"western\", \"what\", \"windows\", \"windows\", \"with\", \"works\", \"yet\", \"you\"]}, \"R\": 30, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [1, 3, 2]};\n\nfunction LDAvis_load_lib(url, callback){\n  var s = document.createElement('script');\n  s.src = url;\n  s.async = true;\n  s.onreadystatechange = s.onload = callback;\n  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n  document.getElementsByTagName(\"head\")[0].appendChild(s);\n}\n\nif(typeof(LDAvis) !== \"undefined\"){\n   // already loaded: just create the visualization\n   !function(LDAvis){\n       new LDAvis(\"#\" + \"ldavis_el67301401007344292168336646043\", ldavis_el67301401007344292168336646043_data);\n   }(LDAvis);\n}else if(typeof define === \"function\" && define.amd){\n   // require.js is available: use it to load d3/LDAvis\n   require.config({paths: {d3: \"https://d3js.org/d3.v5\"}});\n   require([\"d3\"], function(d3){\n      window.d3 = d3;\n      LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n        new LDAvis(\"#\" + \"ldavis_el67301401007344292168336646043\", ldavis_el67301401007344292168336646043_data);\n      });\n    });\n}else{\n    // require.js not available: dynamically load d3 & LDAvis\n    LDAvis_load_lib(\"https://d3js.org/d3.v5.js\", function(){\n         LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n                 new LDAvis(\"#\" + \"ldavis_el67301401007344292168336646043\", ldavis_el67301401007344292168336646043_data);\n            })\n         });\n}\n</script>","textData":null,"removedWidgets":[],"addedWidgets":{},"metadata":{},"type":"htmlSandbox","arguments":{}}},"output_type":"display_data","data":{"text/html":["\n<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v1.0.0.css\">\n\n\n<div id=\"ldavis_el67301401007344292168336646043\"></div>\n<script type=\"text/javascript\">\n\nvar ldavis_el67301401007344292168336646043_data = {\"mdsDat\": {\"x\": [-0.10874492631132812, 0.08322210023430685, 0.02552282607702126], \"y\": [-0.02025003929606211, -0.04712238241760796, 0.06737242171367007], \"topics\": [1, 2, 3], \"cluster\": [1, 1, 1], \"Freq\": [51.76629595208443, 28.850858374738948, 19.38284567317662]}, \"tinfo\": {\"Term\": [\"windows\", \"with\", \"11\", \"to\", \"microsoft\", \"designed\", \"start\", \"layers\", \"combination\", \"moment\", \"multiple\", \"privacy\", \"protection\", \"information\", \"secure\", \"works\", \"up\", \"you\", \"help\", \"hardware\", \"guard\", \"apps\", \"from\", \"keep\", \"bank\", \"was\", \"yet\", \"use\", \"what\", \"best\", \"bank\", \"as\", \"already\", \"back\", \"cee\", \"its\", \"region\", \"subsidiary\", \"1986\", \"30\", \"active\", \"banking\", \"before\", \"business\", \"curtain\", \"even\", \"experience\", \"fell\", \"first\", \"founded\", \"hungary\", \"iron\", \"looks\", \"more\", \"political\", \"process\", \"regions\", \"started\", \"than\", \"therefore\", \"of\", \"was\", \"rbi\", \"microsoft\", \"windows\", \"best\", \"features\", \"how\", \"learn\", \"makes\", \"meet\", \"new\", \"see\", \"use\", \"what\", \"yet\", \"asian\", \"at\", \"branches\", \"european\", \"locations\", \"offices\", \"operates\", \"representative\", \"selected\", \"service\", \"western\", \"boost\", \"car\", \"collaborate\", \"efficiency\", \"mercedesbenz\", \"production\", \"11\", \"to\", \"with\", \"apps\", \"combination\", \"designed\", \"from\", \"guard\", \"hardware\", \"help\", \"information\", \"keep\", \"layers\", \"moment\", \"multiple\", \"privacy\", \"protection\", \"secure\", \"start\", \"up\", \"works\", \"you\", \"to\", \"11\", \"was\", \"windows\", \"of\", \"boost\", \"car\", \"collaborate\", \"efficiency\", \"mercedesbenz\", \"rbi\"], \"Freq\": [2.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 2.0, 1.0, 1.0, 1.0, 1.0, 2.8603709333189498, 2.2002453829935726, 1.5401451637540295, 1.5401451637540295, 1.5401353797896598, 1.5401353797896598, 1.5401353797896598, 1.5401353797896598, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 0.8800220259130175, 2.2040311070701333, 1.5403701949345345, 1.5430601150016794, 1.1996938865059632, 1.7151447024959745, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6855050030588941, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854403153838504, 0.6854025933562556, 0.6854025933562556, 0.6854025933562556, 0.6854025933562556, 0.6854025933562556, 0.6854025933562556, 1.1999556250895118, 1.1993167409271, 0.9976623700774936, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5700656048576532, 0.5705039090623786, 0.5699872681432525, 0.569983805469778, 0.5691906023257559, 0.5702602673273269, 0.14265498342890257, 0.14265498342890257, 0.14265498342890257, 0.14265498342890257, 0.14265498342890257, 0.14267661259219963], \"Total\": [2.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 2.0, 1.0, 1.0, 1.0, 1.0, 3.1743973366535805, 2.514279514888329, 1.8541678256374243, 1.8541678256374243, 1.8541648639800468, 1.8541648639800468, 1.8541648639800468, 1.8541648639800468, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 1.1940494498601122, 3.4569114266647407, 2.281843977479953, 2.3690150852655014, 1.5624325091521065, 2.5044619046124246, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482026801427078, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482041043511714, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.9900707327723683, 1.9899708385161876, 1.389181405879746, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 0.9615887747025282, 1.9899708385161876, 1.9900707327723683, 2.281843977479953, 2.5044619046124246, 3.4569114266647407, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 1.0482020189081362, 2.3690150852655014], \"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\"], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -3.2251, -3.4875, -3.8442, -3.8442, -3.8442, -3.8442, -3.8442, -3.8442, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -4.4039, -3.4858, -3.844, -3.8423, -3.5094, -3.1519, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.069, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0691, -4.0692, -4.0692, -4.0692, -4.0692, -4.0692, -4.0692, -3.5092, -3.5097, -3.296, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8557, -3.8549, -3.8558, -3.8558, -3.8572, -3.8554, -5.241, -5.241, -5.241, -5.241, -5.241, -5.2409], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 0.5543, 0.525, 0.4729, 0.4729, 0.4729, 0.4729, 0.4729, 0.4729, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.3533, 0.2083, 0.2655, 0.2297, 0.9789, 0.8645, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8184, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8183, 0.8182, 0.8182, 0.8182, 0.8182, 0.8182, 0.8182, 0.7371, 0.7367, 1.3097, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 1.1179, 0.3914, 0.3905, 0.2537, 0.1592, -0.1613, -0.3536, -0.3536, -0.3536, -0.3536, -0.3536, -1.1689]}, \"token.table\": {\"Topic\": [2, 3, 1, 1, 1, 1, 3, 1, 2, 2, 1, 1, 1, 1, 2, 2, 2, 1, 2, 1, 2, 3, 1, 3, 2, 2, 1, 1, 2, 1, 1, 1, 3, 3, 3, 3, 2, 1, 3, 1, 1, 3, 3, 2, 2, 1, 2, 2, 2, 2, 3, 1, 3, 2, 1, 2, 3, 2, 2, 1, 3, 1, 2, 3, 1, 2, 1, 1, 2, 3, 2, 2, 2, 3, 1, 1, 1, 1, 2, 3, 3, 2, 1, 3, 2, 2, 2, 3, 3, 3, 2, 3], \"Freq\": [0.5024947020887542, 0.5024947020887542, 0.8374862532846978, 0.8374862532846978, 0.8374862532846978, 1.0786510111685503, 1.0399455841290934, 0.7954565067873249, 0.9540126735326901, 0.9540126735326901, 1.0786510111685503, 0.945061276784768, 0.8374862532846978, 0.8374862532846978, 0.9540139697638006, 0.9540145715819685, 0.9540126735326901, 0.8374862532846978, 0.9540145715819685, 1.0786527340977174, 0.9540145715819685, 1.0399455841290934, 0.8374862532846978, 1.0399455841290934, 0.9540145715819685, 0.9540126735326901, 0.8374862532846978, 0.8374862532846978, 0.9540139697638006, 0.8374862532846978, 0.8374862532846978, 0.8374862532846978, 1.0399455841290934, 1.0399455841290934, 1.0399455841290934, 1.0399455841290934, 0.9540139697638006, 0.8374862532846978, 1.0399455841290934, 0.8374862532846978, 1.0786527340977174, 1.0399455841290934, 1.0399455841290934, 0.9540139697638006, 0.9540126735326901, 0.8374862532846978, 0.9540139697638006, 0.9540139697638006, 0.9540145715819685, 0.6400276454454185, 1.0399455841290934, 0.8374862532846978, 1.0399455841290934, 0.9540139697638006, 0.5785511264688716, 0.2892755632344358, 0.2892755632344358, 0.9540126735326901, 0.9540126735326901, 0.8374862532846978, 1.0399455841290934, 0.8374862532846978, 0.9540145715819685, 1.0399455841290934, 0.8442326992509864, 0.4221163496254932, 1.0786527340977174, 0.8374862532846978, 0.9540126735326901, 1.0399455841290934, 0.9540139697638006, 0.9540126735326901, 0.9540126735326901, 1.0399455841290934, 0.8374862532846978, 1.0786527340977174, 0.8374862532846978, 0.8374862532846978, 0.5025199267470901, 0.5025199267470901, 1.0399455841290934, 0.9540139697638006, 0.8764841153639178, 0.4382420576819589, 0.9540126735326901, 0.9540139697638006, 0.7985747342838931, 0.39928736714194657, 0.7198483911226239, 1.0399455841290934, 0.9540139697638006, 1.0399455841290934], \"Term\": [\"11\", \"11\", \"1986\", \"30\", \"active\", \"already\", \"apps\", \"as\", \"asian\", \"at\", \"back\", \"bank\", \"banking\", \"before\", \"best\", \"boost\", \"branches\", \"business\", \"car\", \"cee\", \"collaborate\", \"combination\", \"curtain\", \"designed\", \"efficiency\", \"european\", \"even\", \"experience\", \"features\", \"fell\", \"first\", \"founded\", \"from\", \"guard\", \"hardware\", \"help\", \"how\", \"hungary\", \"information\", \"iron\", \"its\", \"keep\", \"layers\", \"learn\", \"locations\", \"looks\", \"makes\", \"meet\", \"mercedesbenz\", \"microsoft\", \"moment\", \"more\", \"multiple\", \"new\", \"of\", \"of\", \"of\", \"offices\", \"operates\", \"political\", \"privacy\", \"process\", \"production\", \"protection\", \"rbi\", \"rbi\", \"region\", \"regions\", \"representative\", \"secure\", \"see\", \"selected\", \"service\", \"start\", \"started\", \"subsidiary\", \"than\", \"therefore\", \"to\", \"to\", \"up\", \"use\", \"was\", \"was\", \"western\", \"what\", \"windows\", \"windows\", \"with\", \"works\", \"yet\", \"you\"]}, \"R\": 30, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [1, 3, 2]};\n\nfunction LDAvis_load_lib(url, callback){\n  var s = document.createElement('script');\n  s.src = url;\n  s.async = true;\n  s.onreadystatechange = s.onload = callback;\n  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n  document.getElementsByTagName(\"head\")[0].appendChild(s);\n}\n\nif(typeof(LDAvis) !== \"undefined\"){\n   // already loaded: just create the visualization\n   !function(LDAvis){\n       new LDAvis(\"#\" + \"ldavis_el67301401007344292168336646043\", ldavis_el67301401007344292168336646043_data);\n   }(LDAvis);\n}else if(typeof define === \"function\" && define.amd){\n   // require.js is available: use it to load d3/LDAvis\n   require.config({paths: {d3: \"https://d3js.org/d3.v5\"}});\n   require([\"d3\"], function(d3){\n      window.d3 = d3;\n      LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n        new LDAvis(\"#\" + \"ldavis_el67301401007344292168336646043\", ldavis_el67301401007344292168336646043_data);\n      });\n    });\n}else{\n    // require.js not available: dynamically load d3 & LDAvis\n    LDAvis_load_lib(\"https://d3js.org/d3.v5.js\", function(){\n         LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n                 new LDAvis(\"#\" + \"ldavis_el67301401007344292168336646043\", ldavis_el67301401007344292168336646043_data);\n            })\n         });\n}\n</script>"]}}],"execution_count":0},{"cell_type":"markdown","source":["#### 3 - Tips to improve results of topic modeling\n\nThe results of topic models are completely dependent on the features (terms) present in the corpus. The corpus is represented as document term matrix, which in general is very sparse in nature. Reducing the dimensionality of the matrix can improve the results of topic modelling. Based on my practical experience, there are few approaches which do the trick.  \n\n1. Frequency Filter – Arrange every term according to its frequency. Terms with higher frequencies are more likely to appear in the results as compared ones with low frequency. The low frequency terms are essentially weak features of the corpus, hence it is a good practice to get rid of all those weak features. An exploratory analysis of terms and their frequency can help to decide what frequency value should be considered as the threshold.  \n\n2. Part of Speech Tag Filter – POS tag filter is more about the context of the features than frequencies of features. Topic Modelling tries to map out the recurring patterns of terms into topics. However, every term might not be equally important contextually. For example, POS tag IN contain terms such as – “within”, “upon”, “except”. “CD” contains – “one”,”two”, “hundred” etc. “MD” contains “may”, “must” etc. These terms are the supporting words of a language and can be removed by studying their post tags.  \n\n3. Batch Wise LDA –In order to retrieve most important topic terms, a corpus can be divided into batches of fixed sizes. Running LDA multiple times on these batches will provide different results, however, the best topic terms will be the intersection of all batches.  \n\n#### 4 - Topic Modelling for Feature Selection\n\nSometimes LDA can also be used as feature selection technique. Take an example of text classification problem where the training data contain category wise documents. If LDA is running on sets of category wise documents. Followed by removing common topic terms across the results of different categories will give the best features for a category."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"13da306e-41e6-456c-93a5-5b84947eb0ab","inputWidgets":{},"title":""}}},{"cell_type":"code","source":[""],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"0e4b79f6-d3d4-4af3-90f8-3cebd4c2d8ad","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0}],"metadata":{"application/vnd.databricks.v1+notebook":{"notebookName":"03_NLP_TopicModeling","dashboards":[],"notebookMetadata":{"pythonIndentUnit":4},"language":"python","widgets":{},"notebookOrigID":866167628391024}},"nbformat":4,"nbformat_minor":0}