Scrapy - Item Pipeline

Scrapy is a web scraping library that is used to scrape, parse and collect web data. For all these functions we are having a pipelines.py file which is used to handle scraped data through various components (known as class) which are executed sequentially.

In this article, we will be learning through the methods defined for this pipeline's file and will show different examples of it.

Setting Up Project

Let's, first, create a scrapy project. For that make sure that Python and PIP are installed in the system. Then run the given commands below one by one to create a scrapy project similar to the one which we will be using in this article.

Step 1: Let's first create a virtual environment in a folder named GFGScrapy and activate that virtual environment there.

# To create a folder named GFGScrapy
mkdir GFGScrapy
cd GFGScrapy

# making virtual env there.
virtualenv
cd scripts

# activating it
activate
cd..

Hence, after running all these commands, we will get the output as shown:

Step 2: Now it's time to create a scrapy project. For that Make sure that scrapy is installed in the system or not. If not installed, install it using the given command below.

pip install scrapy

Now to create a scrapy project use the, given command below and also create a spider.

# projEct name is scrapytutorial
scrapy startproject scrapytutorial
cd scrapytutorial
scrapy genspider spider_to_crawl https://quotes.toscrape.com/

Then the output of the project directory looks like the one given in the image. (Please refer this if you want to know more about a scrapy project and get familiar with it).

Let's have a look at our spider_to_crawl.py file present inside our spiders folder. This is the file where we are writing the URL where our spider has to crawl and also a method named parse() which is used to describe what should be done with the data scraped by the spider.

This file is automatically generated by "scrapy genspider" command used above. The file is named after the spider's name. Below given is the default file generated.

Item pipeline is a pipeline method that is written inside pipelines.py file and is used to perform the below-given operations on the scraped data sequentially. The various operations we can perform on the scraped items are listed below:

• Parse the scraped files or data.
• Store the scraped data in databases.
• Validating and checking the data obtained.
• Converting files from one format to another. eg to JSON.

We will be performing some of these operations in the examples below.

Operations are performed sequentially since we are using settings.py file to describe the order in which the operations should be done. i.e. we can mention which operation to be performed first and which to be performed next. This is usually done when we are performing several operations on the items.

Let's first see the inner structure of a default pipeline file. Below is the default class mentioned in that file.

For performing different operations on items we have to declare a separated component( classes in the file) which consists of various methods, used for performing operations. The pipelines file in default has a class named after the project name. We can also create our own classes to write what operations they have to perform. If any pipelines file consists of more than one class then we should mention their execution order explicitly. The structure of components are defined below:

Each component (class) must have one default function named process_item(). This is the default method which is always called inside the class or component of the pipelines file.

Syntax: process_item( self, item, spider )

Parameters:

• self : This is reference to the self object calling the method.
• item : These are the items list scraped by the spider
• spider : mentions the spider used to scrape.

The return type of this method is the modified or unmodified item object or an error will be raised if any fault is found in item.

This method is also used to call other method in this class which can be used to modify or store data.

Additional methods: These methods are used along with the above-mentionedself-object method to gain extra control over the items.

Apart from all these methods, we can also create our own method to perform more operations like if we want to store some data then we can have the component that initializes the database and create tables in it, Another component may be there which will add the data to the database.  

Before we move ahead and refer to examples, an important point to note is that we should have to register all the components (classes) of the pipelines.py file in the settings.py of the folder structure. This is done to maintain an ordering of the components to be executed and hence produce accurate results.

Creating Items to be passed over files.

One more thing to note is that we will require a description of what our item will contain in items.py file. Hence our items.py file contains the below-given code:

# Define here the models for your scraped items
import scrapy

class ScrapytutorialItem(scrapy.Item):
  
    # define the fields for your item here like:
    # name = scrapy.Field()
    Quote = scrapy.Field()     #only one field that it of Quote.

We will require this file to be imported in our spider_to_crawl.py file. Hence in this way we can create items to be passed to pipeline. We will mainly be using the Wisdom quotes web page where we can get several quotes based on their authors and respective tags and then we will modify and use items pipelines on the scraped data throughout the example.

Example 1: Converting scraped data to JSON format

To convert the data in JSON format we will be using JSON library of python along with its dumps() method.  

The idea is that we will get the scraped data in pipelines.py file and then we will open a file named result.json (if not already present then it will be created automatically) and write all the JSON data in it.

• open_spider() will be called to open the file (result.json) when spider starts crawling.
• close_spider() will be called to close the file when spider is closed and scraping is over.
• process_item() will always be called (since it is default) and will be mainly responsible for converting the data to JSON format and print the data to the file. We will be using the concept of python web frameworks, i.e. how they convert backend retrieved data to JSON and other formats.

Hence the code in our pipelines.py looks like this:

from itemadapter import ItemAdapter
import json 

class ScrapytutorialPipeline:
    def process_item(self, item, spider):        
      
          # calling dumps to create json data.
        line = json.dumps(dict(item)) + "\n"   
        self.file.write(line)                
        return item

    def open_spider(self, spider):
        self.file = open('result.json', 'w')

    def close_spider(self, spider):
        self.file.close()

Our spider_to_crawl.py:

import scrapy
from ..items import ScrapytutorialItem    

class SpiderToCrawlSpider(scrapy.Spider):
    name = 'spider_to_crawl'

    start_urls = ['https://quotes.toscrape.com//']

    def parse(self, response):
      
          # creating items dictionary 
        items = ScrapytutorialItem()    
        Quotes_all = response.xpath('//div/div/div/span[1]')

        # These paths are based on the selectors
        
        for quote in Quotes_all:    #extracting data
            items['Quote'] = quote.css('p::text').extract()
            yield items
            

Output:

Explanation:

After using the command "scrapy crawl spider_to_crawl", The below-given steps are going to take place.

• The spider is crawled due to which result.json file is created. Now the spider scrapes the web page and collects the data in Quotes_all Variable. Then we will send each data from this variable one by one to our pipelines.py file.
• We receive item variables from spider in pipelines.py file which is, then converted to JSON using the dumps() method, and then the output is written in the opened file.

This is the JSON file which got created:

Example 2: Pipeline to upload data to database in SQLite3

Now we are going to present an items pipeline that will scrape the content of the web and store it on the database table defined by us. For simplicity, we will be using the SQLite3 database.

So we will use the idea of how to implement SQLite3 in python to create a pipeline that will receive data from spider scraping and will insert that data to the table in the database created.

 spider_to_crawl.py:

import scrapy
from ..items import ScrapytutorialItem    

class SpiderToCrawlSpider(scrapy.Spider):
    name = 'spider_to_crawl'

    start_urls = ['https://quotes.toscrape.com//']

    def parse(self, response):
      
          # creating items dictionary 
        items = ScrapytutorialItem()    
        Quotes_all = response.xpath('//div/div/div/span[1]') 
        
        # These paths are based on the selectors
        
        # extracting data
        for quote in Quotes_all:    
            items['Quote'] = quote.css('::text').extract()
            yield items
            

We are mentioning the pipeline methods below which are to be written in the pipelines.py File so that the database will be created:

pipelines.py file

from itemadapter import ItemAdapter
import sqlite3

class ScrapytutorialPipeline(object):

# init method to initialize the database
# and create connection and tables
    def __init__(self):
        self.create_conn()
        self.create_table()
    
    # create connection method to create
    # database or use database to store scraped data
    def create_conn(self):
        self.conn = sqlite3.connect("mydata.db")
        self.curr = self.conn.cursor()

    # Create table method
    # using SQL commands to create table
    def create_table(self):
        self.curr.execute("""DROP TABLE IF EXISTS firsttable""")
        self.curr.execute("""create table firsttable(
                        Quote text
                        )""")
        
# store items to databases.
    def process_item(self, item, spider):
        self.putitemsintable(item)
        return item
    
    def putitemsintable(self,item):
        self.curr.execute("""insert into firsttable values (?)""",(
            item['Quote'][0],  # extracting item.
        ))
        self.conn.commit()

Output:

Explanation:

After using the command "scrapy crawl spider_to_crawl", the below-given steps are going to take place

• In spider.py we had mentioned the code that our spider should go to that site and extract all data mentioned in the URL format and then will create items list of it and pass that list to the pipelines.py file for further processing.
• We are also creating an items object to contain data to be passed and item it at items.py file in the directory.
• Then when the spider crawls it collects data in items object and transfers it to the pipelines and what happens next is already clear from the above code with hints in comments. pipelines.py file creates a database and stores all the incoming items.
• Here the init() method is called which is called a default method always in any python file. It then calls all other methods which are used to create a table and initialize the database.
• Then process_item() method is called which is used to call a method named putitemintable() where data is added to the database. Then, after executing this method the reference is returned to the spider to pass another item to be operated.