Rule-Based Data Extraction in HTML using Python textminer Module

Last Updated : 21 Jul, 2021

While working with HTML, there are various requirements to extract data from plain HTML tags in an ordered manner in form of python data containers such as lists, dict, integers, etc. This article deals with a library that helps to achieve this using a rule-based approach.

Features of Python - textminer:

Extracts data in form of a list, dictionary, and texts from HTML.
Used rule-based system in YAML format.
Supports extraction from URL in form of scraping.

Installation:

Use the below command to install Python textminer:

pip install textminer

Functions Description:

The following functions come in handy while extracting data from HTML:

Syntax:

extract(html, rule)

Parameters:

html: The HTML to extract data from.
rule: Rule in YAML format to apply on HTML to extract data.

Syntax:

extract_from_url(url, rule)

Parameters:

rule: Rule in YAML format to apply on HTML to extract data.
url: The HTML URL from which extraction of HTML has to be performed.

Example 1: Extracting data from HTML

This basic rule in YAML format is formulated to extract data between a suffix and a prefix.

Python3

import textminer

# input html
inp_html = '<html><body><div>GFG is best for Geeks</div></body></html>'

# yaml rule string
rule = '''
value:
  prefix: <div>
  suffix: </div>
'''

# using extract() to get required data
res = textminer.extract(inp_html, rule)

print("The data extracted between divs : ")
print(res)

Output :

Example 2: Extracting a list from HTML

The python-based list can be extracted from Html which is commonly referred to using list tags, by using <li> and </li> as prefix and suffix of rule. Additionally, the "list" keyword needs to be added to achieve this.

Python3

import textminer

# input html
inp_html = """<html>
<body>
<ul>
    <li>Gfg</li>
    <li>is</li>
    <li>best</li>
</ul>
</body>
</html>"""

# yaml rule string
# extracting list using <li>
# using "list" keyword
rule = '''
list:
  prefix: <li>
  suffix: </li>
'''

# using extract() to get required data
res = textminer.extract(inp_html, rule)

print("The data extracted between list tags : ")
print(res)

Output :

Example 3: Extracting dictionary from HTML using defined data types.

Similar to the above example, a dictionary can be extracted using "dic" keyword, with mentioning "key" required to map key to, and value is extracted using defining prefix and suffix tags with a specific id. The data type can be mentioned using the "type" keyword.

Python3

import textminer

# input html
inp_html = """<html>
<body>
<div id="Gfg">Best</div>
<div id="4">Geeks</div>
</body>
</html>"""

# yaml rule string
# extracting dict. using dict
# using int to extract key in integer format
rule = '''
dict:
- key: gfg
  prefix: <div id="Gfg">
  suffix: </div>
- key: 4
  prefix: <div id="4">
  suffix: </div>
  type: int
'''

# using extract() to get required data
res = textminer.extract(inp_html, rule)

print("The data extracted between dictionary tags : ")
print(res)

Output :

Example 4: Extract HTML from URL

Apart from giving HTML as a string, HTML can also be provided using a url using extract_from_url().

Python3

import textminer

# required url
target_url = "https://www.geeksforgeeks.org/"

# extracting title from url
rule = '''
value:
  prefix: <title>
  suffix: </title>
'''

# using extract() to get required data
res = textminer.extract_from_url(target_url, rule)

print("The data extracted between title tags from url : ")
print(res)