Exercises/Exercicios_Web_Scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Capturando páginas da web\n",
    "==============\n",
    "\n",
    "Neste exercício vamos desenvolver algumas técnicas de captura de páginas da web de forma eficiente, alimentando dois bancos de dados com os resultados da nossa captura: O primeiro com metadados da captura e o segundo com o conteúdo das páginas.\n",
    "\n",
    "Lista de Links\n",
    "----------------\n",
    "Baixe a lista de links (https://copy.com/uT4fSbbmqNrq8B8s) e salve-a no mesmo diretório deste notebook.\n",
    "\n",
    "**Exercício 1:** Determine o número de links a serem capturados\n",
    "\n",
    " \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import gzip\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>link</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Bonde. O seu portal</td>\n",
       "      <td>http://www.bonde.com.br/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>UOL Dieta e Boa Forma</td>\n",
       "      <td>http://boaforma.uol.com.br/ultnot/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Migalhas</td>\n",
       "      <td>http://www.migalhas.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Car And Driver</td>\n",
       "      <td>http://caranddriverbrasil.uol.com.br//</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Jornal dos Concursos</td>\n",
       "      <td>http://www.jcconcursos.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Experiências Discovery em discoverybrasil.uol....</td>\n",
       "      <td>http://discoverybrasil.uol.com.br/experiencia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>UOL Política</td>\n",
       "      <td>http://noticias.uol.com.br/politica/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>UOL Tabloide</td>\n",
       "      <td>http://noticias.uol.com.br/tabloide/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>UOL Ciência</td>\n",
       "      <td>http://noticias.uol.com.br/ciencia/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>UOL Viagem</td>\n",
       "      <td>http://viagem.uol.com.br/ultnot/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>UOL Carros</td>\n",
       "      <td>http://carros.uol.com.br/ultimas/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>TenisBrasil</td>\n",
       "      <td>http://www.tenisbrasil.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Go Outside.com.br</td>\n",
       "      <td>http://gooutside.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Folha de S.Paulo - F5 - Tudo</td>\n",
       "      <td>http://redir.folha.com.br/redir/online/tudo/rs...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>UOL Internacional</td>\n",
       "      <td>http://noticias.uol.com.br/internacional/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Boteco do Tulípio</td>\n",
       "      <td>http://tulipio.uol.com.br/cartum</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>UOL Olimpiadas</td>\n",
       "      <td>http://olimpiadas.uol.com.br/ultimas/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>UOL News</td>\n",
       "      <td>http://noticias.uol.com.br/uolnews/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Click Jogos - Jogos Online</td>\n",
       "      <td>http://www.clickjogos.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Olhares.com</td>\n",
       "      <td>http://olhares.sapo.pt/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>AP470</td>\n",
       "      <td>http://noticias.pgr.mpf.mp.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Notícias do Tribunal de Justiça do Estado de S...</td>\n",
       "      <td>http://www.tjsp.jus.br/Institucional/CanaisCom...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>CBMMT</td>\n",
       "      <td>http://www.bombeiros.mt.gov.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>TCE MT | Últimas Notícias</td>\n",
       "      <td>http://www.tce.mt.gov.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Conversa Afiada</td>\n",
       "      <td>http://www.conversaafiada.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Mobile Xpert - Você agora é Xpert!</td>\n",
       "      <td>http://mobilexpert.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>MobileXpert | Apps</td>\n",
       "      <td>http://mobilexpert.com.br/apps</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>MobileXpert | Reviews</td>\n",
       "      <td>http://mobilexpert.com.br/reviews</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>MobileXpert | Smartphones</td>\n",
       "      <td>http://mobilexpert.com.br/smartphones</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>MobileXpert | Tablets</td>\n",
       "      <td>http://mobilexpert.com.br/tablets</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264854</th>\n",
       "      <td>Comentários sobre Forças Terrestres - ForTe</td>\n",
       "      <td>http://www.forte.jor.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264855</th>\n",
       "      <td>IDS Tecnologia</td>\n",
       "      <td>http://www.ids.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264856</th>\n",
       "      <td>Comentários sobre IDS Tecnologia</td>\n",
       "      <td>http://www.ids.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264857</th>\n",
       "      <td>Folha de S.Paulo - Poder - Poder e Política</td>\n",
       "      <td>http://redir.folha.com.br/redir/online//poder/...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264858</th>\n",
       "      <td>cultura – ONU Brasil</td>\n",
       "      <td>https://nacoesunidas.org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264859</th>\n",
       "      <td>Comentários para ONU Brasil</td>\n",
       "      <td>https://nacoesunidas.org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264860</th>\n",
       "      <td>ONU Brasil</td>\n",
       "      <td>https://nacoesunidas.org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264861</th>\n",
       "      <td>VilmarBRO » Page not found</td>\n",
       "      <td>http://vilmarbro.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264862</th>\n",
       "      <td>VilmarBRO</td>\n",
       "      <td>http://vilmarbro.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264863</th>\n",
       "      <td>e-commerce.org</td>\n",
       "      <td>http://www.e-commerce.org.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264864</th>\n",
       "      <td>Comentários sobre e-commerce.org</td>\n",
       "      <td>http://www.e-commerce.org.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264865</th>\n",
       "      <td>Comentários sobre: O que é RSS?</td>\n",
       "      <td>http://www.e-commerce.org.br/rss-conteudo/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264866</th>\n",
       "      <td>Revista de Sociologia e Política</td>\n",
       "      <td>http://www.scielo.br/rss.php?pid=0104-44782015...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264867</th>\n",
       "      <td>ISTOÉ Independente - RSS - Assuntos</td>\n",
       "      <td>http://www.istoe.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264868</th>\n",
       "      <td>Mais E-duc</td>\n",
       "      <td>http://maiseduc.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264869</th>\n",
       "      <td>Comments for Mais E-duc</td>\n",
       "      <td>http://maiseduc.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264870</th>\n",
       "      <td>qualinter</td>\n",
       "      <td>http://www.qualinter.com.br/#!-ltimas-not-cias...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264871</th>\n",
       "      <td>ABC da Saúde</td>\n",
       "      <td>https://www.abcdasaude.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264872</th>\n",
       "      <td>SneakersBR</td>\n",
       "      <td>http://sneakersbr.co</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264873</th>\n",
       "      <td>Comentários sobre SneakersBR</td>\n",
       "      <td>http://sneakersbr.co</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264874</th>\n",
       "      <td>SCW</td>\n",
       "      <td>http://www.surfcrew.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264875</th>\n",
       "      <td>Comentários sobre SCW</td>\n",
       "      <td>http://www.surfcrew.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264876</th>\n",
       "      <td>Comentários sobre: O que é RSS?</td>\n",
       "      <td>http://www.e-commerce.org.br/rss-conteudo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264877</th>\n",
       "      <td>Comentários sobre E-commerce.org</td>\n",
       "      <td>http://www.e-commerce.org.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264878</th>\n",
       "      <td>E-commerce.org</td>\n",
       "      <td>http://www.e-commerce.org.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264879</th>\n",
       "      <td>Comentários sobre: News</td>\n",
       "      <td>http://comunicacaoepolitica.com.br/blog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264880</th>\n",
       "      <td>Comentários sobre Comunicação e Política</td>\n",
       "      <td>http://comunicacaoepolitica.com.br/blog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264881</th>\n",
       "      <td>Revista de Sociologia e Política</td>\n",
       "      <td>http://www.scielo.br/rss.php?pid=0104-44782015...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264882</th>\n",
       "      <td>Ciência Hoje</td>\n",
       "      <td>http://cienciahoje.uol.com.br</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>264883</th>\n",
       "      <td>Política: últimas notícias do cenário político...</td>\n",
       "      <td>http://ultimosegundo.ig.com.br/politica/</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>264884 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    title  \\\n",
       "0                                     Bonde. O seu portal   \n",
       "1                                   UOL Dieta e Boa Forma   \n",
       "2                                                Migalhas   \n",
       "3                                          Car And Driver   \n",
       "4                                    Jornal dos Concursos   \n",
       "5       Experiências Discovery em discoverybrasil.uol....   \n",
       "6                                            UOL Política   \n",
       "7                                            UOL Tabloide   \n",
       "8                                             UOL Ciência   \n",
       "9                                              UOL Viagem   \n",
       "10                                             UOL Carros   \n",
       "11                                            TenisBrasil   \n",
       "12                                      Go Outside.com.br   \n",
       "13                           Folha de S.Paulo - F5 - Tudo   \n",
       "14                                      UOL Internacional   \n",
       "15                                      Boteco do Tulípio   \n",
       "16                                         UOL Olimpiadas   \n",
       "17                                               UOL News   \n",
       "18                             Click Jogos - Jogos Online   \n",
       "19                                            Olhares.com   \n",
       "20                                                  AP470   \n",
       "21      Notícias do Tribunal de Justiça do Estado de S...   \n",
       "22                                                  CBMMT   \n",
       "23                              TCE MT | Últimas Notícias   \n",
       "24                                        Conversa Afiada   \n",
       "25                     Mobile Xpert - Você agora é Xpert!   \n",
       "26                                     MobileXpert | Apps   \n",
       "27                                  MobileXpert | Reviews   \n",
       "28                              MobileXpert | Smartphones   \n",
       "29                                  MobileXpert | Tablets   \n",
       "...                                                   ...   \n",
       "264854        Comentários sobre Forças Terrestres - ForTe   \n",
       "264855                                     IDS Tecnologia   \n",
       "264856                   Comentários sobre IDS Tecnologia   \n",
       "264857        Folha de S.Paulo - Poder - Poder e Política   \n",
       "264858                               cultura – ONU Brasil   \n",
       "264859                        Comentários para ONU Brasil   \n",
       "264860                                         ONU Brasil   \n",
       "264861                         VilmarBRO » Page not found   \n",
       "264862                                          VilmarBRO   \n",
       "264863                                     e-commerce.org   \n",
       "264864                   Comentários sobre e-commerce.org   \n",
       "264865                    Comentários sobre: O que é RSS?   \n",
       "264866                   Revista de Sociologia e Política   \n",
       "264867                ISTOÉ Independente - RSS - Assuntos   \n",
       "264868                                         Mais E-duc   \n",
       "264869                            Comments for Mais E-duc   \n",
       "264870                                          qualinter   \n",
       "264871                                       ABC da Saúde   \n",
       "264872                                         SneakersBR   \n",
       "264873                       Comentários sobre SneakersBR   \n",
       "264874                                                SCW   \n",
       "264875                              Comentários sobre SCW   \n",
       "264876                    Comentários sobre: O que é RSS?   \n",
       "264877                   Comentários sobre E-commerce.org   \n",
       "264878                                     E-commerce.org   \n",
       "264879                            Comentários sobre: News   \n",
       "264880           Comentários sobre Comunicação e Política   \n",
       "264881                   Revista de Sociologia e Política   \n",
       "264882                                       Ciência Hoje   \n",
       "264883  Política: últimas notícias do cenário político...   \n",
       "\n",
       "                                                     link  \n",
       "0                                http://www.bonde.com.br/  \n",
       "1                      http://boaforma.uol.com.br/ultnot/  \n",
       "2                              http://www.migalhas.com.br  \n",
       "3                  http://caranddriverbrasil.uol.com.br//  \n",
       "4                           http://www.jcconcursos.com.br  \n",
       "5           http://discoverybrasil.uol.com.br/experiencia  \n",
       "6                    http://noticias.uol.com.br/politica/  \n",
       "7                    http://noticias.uol.com.br/tabloide/  \n",
       "8                     http://noticias.uol.com.br/ciencia/  \n",
       "9                        http://viagem.uol.com.br/ultnot/  \n",
       "10                      http://carros.uol.com.br/ultimas/  \n",
       "11                          http://www.tenisbrasil.com.br  \n",
       "12                                http://gooutside.com.br  \n",
       "13      http://redir.folha.com.br/redir/online/tudo/rs...  \n",
       "14              http://noticias.uol.com.br/internacional/  \n",
       "15                       http://tulipio.uol.com.br/cartum  \n",
       "16                  http://olimpiadas.uol.com.br/ultimas/  \n",
       "17                    http://noticias.uol.com.br/uolnews/  \n",
       "18                           http://www.clickjogos.com.br  \n",
       "19                                http://olhares.sapo.pt/  \n",
       "20                          http://noticias.pgr.mpf.mp.br  \n",
       "21      http://www.tjsp.jus.br/Institucional/CanaisCom...  \n",
       "22                         http://www.bombeiros.mt.gov.br  \n",
       "23                               http://www.tce.mt.gov.br  \n",
       "24                       http://www.conversaafiada.com.br  \n",
       "25                              http://mobilexpert.com.br  \n",
       "26                         http://mobilexpert.com.br/apps  \n",
       "27                      http://mobilexpert.com.br/reviews  \n",
       "28                  http://mobilexpert.com.br/smartphones  \n",
       "29                      http://mobilexpert.com.br/tablets  \n",
       "...                                                   ...  \n",
       "264854                            http://www.forte.jor.br  \n",
       "264855                              http://www.ids.com.br  \n",
       "264856                              http://www.ids.com.br  \n",
       "264857  http://redir.folha.com.br/redir/online//poder/...  \n",
       "264858                           https://nacoesunidas.org  \n",
       "264859                           https://nacoesunidas.org  \n",
       "264860                           https://nacoesunidas.org  \n",
       "264861                            http://vilmarbro.com.br  \n",
       "264862                            http://vilmarbro.com.br  \n",
       "264863                       http://www.e-commerce.org.br  \n",
       "264864                       http://www.e-commerce.org.br  \n",
       "264865         http://www.e-commerce.org.br/rss-conteudo/  \n",
       "264866  http://www.scielo.br/rss.php?pid=0104-44782015...  \n",
       "264867                            http://www.istoe.com.br  \n",
       "264868                             http://maiseduc.com.br  \n",
       "264869                             http://maiseduc.com.br  \n",
       "264870  http://www.qualinter.com.br/#!-ltimas-not-cias...  \n",
       "264871                      https://www.abcdasaude.com.br  \n",
       "264872                               http://sneakersbr.co  \n",
       "264873                               http://sneakersbr.co  \n",
       "264874                         http://www.surfcrew.com.br  \n",
       "264875                         http://www.surfcrew.com.br  \n",
       "264876          http://www.e-commerce.org.br/rss-conteudo  \n",
       "264877                       http://www.e-commerce.org.br  \n",
       "264878                       http://www.e-commerce.org.br  \n",
       "264879            http://comunicacaoepolitica.com.br/blog  \n",
       "264880            http://comunicacaoepolitica.com.br/blog  \n",
       "264881  http://www.scielo.br/rss.php?pid=0104-44782015...  \n",
       "264882                      http://cienciahoje.uol.com.br  \n",
       "264883           http://ultimosegundo.ig.com.br/politica/  \n",
       "\n",
       "[264884 rows x 2 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links = pd.read_csv('feeds.csv')\n",
    "links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "264884"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(links)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from functools import wraps\n",
    "\n",
    "def timefn(fn):\n",
    "    @wraps(fn)\n",
    "    def measure_time(*args, **kwargs):\n",
    "        t1 = time.time()\n",
    "        result = fn(*args, **kwargs)\n",
    "        t2 = time.time()\n",
    "        print (\n",
    "            \"@timefn:\" + fn.__name__ + \" took \" + str(t2 - t1) + \" seconds\")\n",
    "        return result\n",
    "    return measure_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercício 2:** Implemente uma solução para visita sequencial às páginas correspodentes aos links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 0 ns, sys: 0 ns, total: 0 ns\n",
      "Wall time: 8.82 µs\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "792540"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time\n",
    "r = requests.get(links.link[0])\n",
    "t = r.elapsed\n",
    "t.microseconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercício 3:** Estenda o código do seu capturador para armazenar os seguintes metadados da captura em um banco SQLite: Link, status da requisição, tamanho do html em bytes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sqlite3\n",
    "conn = sqlite3.connect('meubanco.sqlite', check_same_thread = False)\n",
    "# O argumento Check same thread é necessário para permitir cursores criados nas threads poderem escrever no banco"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@timefn:insere_metadado took 0.0003044605255126953 seconds\n",
      "@timefn:insere_metadado took 3.719329833984375e-05 seconds\n",
      "@timefn:insere_metadado took 5.650520324707031e-05 seconds\n",
      "@timefn:insere_metadado took 5.0067901611328125e-05 seconds\n",
      "@timefn:insere_metadado took 3.719329833984375e-05 seconds\n"
     ]
    }
   ],
   "source": [
    "def tabela_existe(nome, cursor):\n",
    "    cursor.execute(\"SELECT name FROM sqlite_master WHERE type='table';\")\n",
    "    nomes = cursor.fetchall()\n",
    "    if len(nomes) == 0:\n",
    "        return False\n",
    "    #print(nomes)\n",
    "    return nome in nomes[0]\n",
    "\n",
    "cursor=conn.cursor()\n",
    "\n",
    "cursor.execute('drop table if exists metadado')\n",
    "cursor.execute('create table if not exists metadado (link text, status integer, tempo real)')\n",
    "\n",
    "@timefn\n",
    "def insere_metadado(r):\n",
    "    cursor=conn.cursor()\n",
    "    cursor.execute('insert into metadado values(?, ?, ?)', (r.url, r.status_code, r.elapsed.microseconds/1e6))\n",
    "\n",
    "for link in links[:5].link:\n",
    "    r = requests.get(link, verify=False)\n",
    "    insere_metadado(r)\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(129,)]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conn = sqlite3.connect('meubanco.sqlite', check_same_thread = False)\n",
    "cursor=conn.cursor()\n",
    "cursor.execute('select count(*) from metadado limit 100')\n",
    "resposta = cursor.fetchall()\n",
    "resposta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlite3.Cursor at 0x7fe18e743650>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cursor.execute('drop table if exists metadado')\n",
    "cursor.execute('create table if not exists metadado (link text, status integer, tempo real)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercício 4:** Implemente o salvamento das páginas capturadas em um banco SQLite, usando o Link como chave primária"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "conn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercício 5:** Modifique a sua implementação de captura, para que esta seja feita em paralelo. *a)* Usando Threads, *b)* Usando multiplos processos, *c)* Usando a biblioteca gevent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from threading import Thread\n",
    "import multiprocessing as MP\n",
    "import queue\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "respostas = []\n",
    "def captura(link):\n",
    "    '''faz o get'''\n",
    "    r = requests.get(link, verify=False)\n",
    "    insere_metadado(r)\n",
    "    #respostas.append(r)\n",
    "\n",
    "for l in links[:5].link:\n",
    "    print(l)\n",
    "    t= Thread(target=captura, args=(l,))\n",
    "    t.start()\n",
    "while len(respostas)<5:\n",
    "    pass\n",
    "#[insere_metadado(r) for r in respostas]\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None, None, None, None, None]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[insere_metadado(r) for r in respostas]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Worker(Thread):\n",
    "    def __init__(self, fila, conn, nome):\n",
    "        Thread.__init__(self,name=str(nome))\n",
    "        self.fila = fila\n",
    "        self.conn = sqlite3.connect('meubanco.sqlite', check_same_thread = False)\n",
    "        self.cursor = self.conn.cursor()\n",
    "    \n",
    "    def run(self):\n",
    "        while True:\n",
    "            try:\n",
    "                # gets the link from the queue\n",
    "                link = self.fila.get()\n",
    "\n",
    "                # download the link\n",
    "                print(\"* Thread \" + self.name + \" - processing URL\")\n",
    "                self.download_page(link)\n",
    "\n",
    "                # send a signal to the queue that the job is done\n",
    "                self.fila.task_done()\n",
    "            except Exception as e:\n",
    "                print(e)\n",
    "                #break\n",
    "            finally:\n",
    "                self.conn.commit()\n",
    "  \n",
    "    def download_page(self, link):\n",
    "        r = requests.get(link)\n",
    "        if (r.status_code == requests.codes.ok):\n",
    "            print(\"* Thread: {} Downloaded {} in {} seconds\".format(self.name, link, r.elapsed.microseconds/1e6))\n",
    "        else:\n",
    "            print(\"* Thread: {} failed {}: {}\".format(self.name, link, r.status_code))\n",
    "        \n",
    "        self.cursor.execute('insert into metadado values(?, ?, ?)', (r.url, r.status_code, r.elapsed.microseconds/1e6))\n",
    "\n",
    "class DownloadManager():\n",
    "  \n",
    "    def __init__(self, links, threads=5):\n",
    "        self.threads = threads\n",
    "        self.links = links\n",
    "        self.conn = None #sqlite3.connect('meubanco.sqlite', check_same_thread = False)\n",
    "        \n",
    "    def pega_links(self):\n",
    "        \"\"\"\n",
    "        Cria as threads, enche a fila de links e alimenta as threads com as urls\n",
    "        \"\"\"\n",
    "        fila = queue.Queue()\n",
    " \n",
    "        for i in range(self.threads):\n",
    "            t = Worker(fila, self.conn, i)\n",
    "            t.setDaemon(True)\n",
    "            t.start()\n",
    "        \n",
    "        for link in self.links:\n",
    "            #print(link)\n",
    "            fila.put(link)\n",
    "\n",
    "        # Espera a fila esvaziar\n",
    "        fila.join()\n",
    "        t.conn.commit()\n",
    "        return          \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "DM = DownloadManager(links.link, 100)\n",
    "DM.pega_links()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercício 6:** Faça um profiling das várias metodologias de captura."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercício 7:** Análise da captura. Crie um gráfico de barras representando quantos links retornaram cada status code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}