2009 Second International Workshop on Computer Science and Engineering

The Implementation of a Web Crawler URL Filter Algorithm Based on Caching

Wang Hui-chang Ruan Shu-hua Tang Qi-jie

School of Computer Science School of Computer Science School of Computer Science
Sichuan University Sichuan University Sichuan University
Chengdu Sichuan 610065, China Chengdu Sichuan 610065, China Chengdu Sichuan 610065, China
e-mail: [email protected] e-mail: [email protected] e-mail: [email protected]

Abstract — For large-scale Web information collection, the URL pages and follow the links to find new pages and links [1,2].
filter module plays important roles in a Web crawler which is a A Web crawler usually starts from a URL address alone or
central component of a search engine. The performance of an URL URL list to visit each pages specified by each URL, extracts
filter module influents the efficiency of the entire collection system the links in each page by the content analysis and eliminates
directly. This paper introduces one URL filter algorithm based on the repetitive URLs after download each page, then adds the
caching and its implementation. The performances of stability and new links to the URL list.
paralleling of the algorithm are verified by the experiments for
Websites which handle a large number of web pages. Experiment Seed URL Internet
results show the algorithm proposed in this paper can achieve
satisfactory performances through reasonable adjustments of its
some parameters and it is suitable for the process of the URL filter
of a Website which has a number of page navigator links and index URL List Page Downloader Crawling
pages especially. Parameter
Assistor
Keywords - Web Crawler; URL Filter; Caching Link Extractor

I. INTRODUCTION
Accompanied by the informationization of society and URL Filter
rapid development of the Internet, more and more
information is on the Internet. These make search engine,
one of the main applications of modern information
technology, become one necessary tool by which people can Figure 1. A simplified web crawler
get information through the Internet more easily and quickly.
A web crawler, which is a central component of a search It is a simplified Web crawler in Figure 1. According to
engine, impacts not only on the recall ratio and precision of a Figure 1, a Web crawler starts an URL called the Seed URL
search engine, but also on the capacity of the data storage to visit the Internet. The Page Downloader gets an URL from
and efficiency of a search engine. URL List to download the page from the Internet, transfers
World Wide Web can be viewed as a directed graph the page to the Link Extractor. The Page Downloader checks
without rules and borders, and a wide range of circles exists the parameters in accordance with the requirements of
in it. Due to a large number of index pages and navigation crawling to decide whether or not to download pages. As the
contents existed in a Website, as well as a large number of crawler visits these URLs, the Link Extractor indentifies all
relevant content links existed in a web page, the URLs get the hyperlinks in the pages in accordance with the
from web pages are much more than the actual number of requirements of crawling and transfers them to the URL
URLs existed in the Internet. Filter, stores the results into URL list. The Crawling
The URL filter module, which is an important Parameter Assistor provides the parameter setting for the
component of a crawler, is used to filter the URLs analyzed needs of all parts of the crawler.
from the web pages downloaded by the crawler of a search
engine so as to improve the efficiency of the crawler. We III. THE COMMON METHODS OF THE URL FILTER
design an URL filter algorithm and implement it based on In order to avoid downloading the same page repeatedly,
caching. Experiment results show the algorithm proposed in we need to eliminate the repetitive URLs. The URL filter can
this paper can achieve satisfactory performances. easily become a bottleneck in the performance of the crawler
II. A WEB CRAWLER system because the URL list to be inquired is growing bigger
and bigger.
A Web crawler (also known as a Web spider or a Web The main idea of the URL filter is to determine whether
robot) is a program or an automated script which can get an URL exists in the known URL list. The URL query
specified pages from the Internet, extract the links from these algorithm is the key. The basic process is for each string of

978-0-7695-3881-5/09 $26.00 © 2009 IEEE 1058

453
DOI 10.1109/WCSE.2009.851
10.1109/WCSE.2009.228
Authorized licensed use limited to: UNIVERSITI TEKNOLOGI MARA. Downloaded on June 28,2024 at 10:57:26 UTC from IEEE Xplore. Restrictions apply.
the URL analyzed by a web page to be compared with the a set can’t be mistakenly believed that it has not consisted in
each string in the known URL list. If it is a repetitive URL, this set, there is no false negative. A Bloom Filter can save a
then be abandon directly, or a new URL, then be added to lot of storage space by allowing a small amount false
the URL list. positive rate.
The common methods of the URL filter are the
followings. IV. THE URL FILTER ALGORITHM BASED ON CACHING
A. The URL filter based on a hash table The integrated ideas to deal with the URL filter are that
the crawler program downloads a page to the local and
Hash table is a linear list stored by hashing method. parses out URLs from the page, then passes the list of URL
According to the method of dealing with conflict, the given to the URL Filter for eliminating the duplications. URL
hash function h(key) converts a set of keywords to a limited Filter as a key component of the crawler provides non
set of consecutive addresses (or range) in which the “map” of duplicating URL address for crawler to crawl on the Internet,
keywords is the storage location. The process is called and as a storage place for parsed URLs communicates with
hashing and the storage addresses is called hashing addresses database to complete the operations of access URL data at
or hash table. Hash table is suitable to deal with finding the same time.
string because it converts the string to an integer and Due to the limited capacity of computer memory, the
associates the string with an address to control the capacity of URL caching queue is limited accordingly. There
complexity of search algorithm within O(1), which is is a large number of duplicating URL addresses in some site,
entirely superior to the other search algorithms [3]. and their presenting frequencies are relatively high. The
The URL was stored as the key of a hash table if we use URL filter algorithm based on caching will greatly improve
hash table to store the URL list. For each URL requesting to the rate of hits by putting these high presenting frequency
add into list, we can search hash table fast to be aware of the URLs into the URL caching queue. According to a cache
URL whether or not a repetitive one. If not a repetitive one, replacement algorithm, we can improve the efficiency of the
we can deposit it into the hash table. Hash table also has the crawler system by maintaining the URL caching queue with
shortcomings that cannot be sorted and traversed and is not a high hit rate and reducing the frequency of database access.
suitable for mass data storage for being subject to the The specific working principle is: when the crawler parses
limitations of memory capacity. out an URL, we query in the caching queue first, if find the
B. The URL filter based on a Bloom Filter URL to increase its repetition rate to the page but not to deal
with, if not find it to query the database on a relatively slow
A Bloom Filter is a collection querying algorithm with speed, then if find it to put it into the different caching
high efficiency in space and time by using a hash function queues according to if it had been visited. Otherwise, load
and representing data with a bit vector, which is suitable for the new URL list which includes all the URLs under the
mass data querying operation [4]. For a Web crawler, there is same directory into the caching queue with waiting for deal
an inconsistency between a large number of URLs and and reduce the number of database access.
loading URLs into memory for fast querying. But for each It is because of this kind of access mechanism that the hit
URL, a Bloom Filter produces random k-values by using k- rate of the caching queue is very high, which means the most
hash functions and set 1 on the corresponding location of a of URLs parsed by crawler program are in the cache queue,
bit vector. So if we use a Bloom Filter to store URL list, the only about 10%-20% of them needed to check in the
size of the URL list is only the size of the bit vector and we database. This greatly saves time to query a new URL and
can optimize the space greatly and make querying fast under need not query database in the most situations. In general,
the restrictions of time and the rate of miscarriage of justice. the sequence of crawler program to determine whether the
Towards every URL which requests to add into the list, k URL is a repetition is the caching queue first, then database.
kinds of hashing calculations shall be done on it. And k The URL filter algorithm based on caching includes
numbers generated as per calculation method shall be used to some parameters, such as caching capacity, exchange rate of
set the correspondence position of the bit vector to be 1. cache and external storage, depth of crawling etc. For a
Before each positioning, determination shall be firstly given variety of network environment, the corresponding
whether or not the k correspondence positions of the bit parameters can be adjusted to achieve the corresponding
vector through hashing calculations have been set to be 1.If performance requirements.
so, determination can be given the URL has already In this paper, the crawler URL filter system based on the
consisted in the list and is a repetitive one. Otherwise cache strategy was implemented by which combined with
position 1 opposite to the bit vector shall be added to the Java 2 platform and MySql5.0 DBMS.
URL list as a new URL. The URL filter algorithm which is based on caching
A Bloom Filter has the advantages of high speed and shows in Figure 2.
saving storage space, so mass data can be represented easily The description of the algorithm is as the followings:
and accessed fast, but in the use of a Bloom Filter to 1) To initialize the lists of URL in the URL Filter.
determine whether an element belongs to a set, there will be There are two URL caching lists in the URL Filter,
a certain false positive rate. That is, an element that is one is visitingList in which URLs will be visited
inexistent in a set may be mistakenly believed that it already and the other is visitedList in which URLs had been
consisted in the set. But an element that is already existent in visited respectively.

1059
454

Authorized licensed use limited to: UNIVERSITI TEKNOLOGI MARA. Downloaded on June 28,2024 at 10:57:26 UTC from IEEE Xplore. Restrictions apply.
 If there is an URL list of Websites to be crawled in V. ALGORITHM IMPLEMENTATION AND PERFORMANCE
database, select the number of initNum of URLs EVALUATION
which are ordered descending by the number of
The performance of the URL filter algorithm based on
repetition to initialize visitingList, otherwise
caching, which is used in large-scale webpage to eliminate
initialize the visitingList with the index page of the
duplication, was analyzed and validated through experiments
Website.
by the prototype system.
In the experiments, having set the parameters (caching
Parsed URL capacity, exchange rate of cache and external storage, and
depth of crawling) according to different crawling situations,
we obtained basic data from a wide variety of Websites, and
Yes
In visitedList ? then got the interrelated data for URL filter based on the
synthetic analysis of basic data. The effect comparison of
URL filter algorithm, which is implemented in this paper,
No
shows in Figure 3, in which the average links of a page (the
Yes average number of effective links per page in a Website) is
In visitingList ? as X-coordinate, the rate of hitting the cache (the ratio of a
pending URL can be found in cache) is as Y- coordinate.
duplicated number + 1
100
No
90
The rate 80
No New URL, add to
In database ? of hitting
visitingList 70
the cache
Yes 60
(%)
50
No Add to visitingList,

2 .7

3. 1
3 .2
3. 3
4 .7

12
2
2.5

2.9

9.7
10 .6
11. 3
Visited ? duplicated number + 1
First Non-First The average links of a page
Yes
Figure 3. The effect comparison of URL filter
Add to visitedList,
duplicated number + 1 We know, from the experimental data, the higher the
Finish average links of a page, the higher the rate of hitting the
cache, the more efficient the URL filter, especially when
Figure 2. URL filter algorithm based on caching crawling the Website again, the rate of hitting the cache has
been increased significantly and the performance of the URL
2) To add the URLs parsed from one page and filter has been improved accordingly. So that it is very
eliminated duplication into the related URL lists in efficient to crawl the Website which has a large number of
the URL Filter. page navigator links and index pages by using the web
The specific filter steps are: for a parsed URL, crawler URL filter algorithm based on caching. In addition, a
searching in the visitedList of URL Filter first, if large number of experimental data shows that the algorithm
found, it is a repetition of some URL and increase has consistent performances of stability and parallel
its duplicated number, otherwise searching in processing while crawling all types of Websites. The
visitingList. If found, it is a repetition of some URL experimental data shows in table 1.
and increase its duplicated number, otherwise The experimental data in table 1 indicated that with the
searching in database. If not found, it is a new URL increase in the number of links, the rate of hitting the cache
and add it to visitingList, otherwise determining it a also increase, in particular, the higher the average links of a
repetition of some URL, increasing its duplicated page, the higher the efficiency of eliminating duplication.
number and adding it to the related URL lists The prototype system, which centers on the database storage,
according to it had been visited or not. has more than one processing node and each node can crawl
3) If the size of the URL caching lists in the URL parallel. The performance of eliminating duplication of each
Filter is larger than the setting value of the crawling node will not reduce with the increasing of node, and the
parameters, transfer and store the URL collection time processing performance of the whole system only will
with the setting capacity into database, otherwise be influenced by database concurrent processing and
load the some block of URLs from database into the network environment, so the stability and parallel
URL lists according to the replacement strategy. performances of the algorithm are good.

1060
455

Authorized licensed use limited to: UNIVERSITI TEKNOLOGI MARA. Downloaded on June 28,2024 at 10:57:26 UTC from IEEE Xplore. Restrictions apply.
 Table I. URL filter data for different types of Websites experiments for Websites which handle a large number of
web pages. Experiment results show the algorithm proposed
Website URL NL NLAF RHCF RHCNF
in this paper can achieve satisfactory performances through
reasonable adjustments of its some parameters and is suitable
for the process of the URL filter of a Website which has a
http://cs.scu.edu.cn 1408 399 59.5 100
number of page navigator links and index pages especially.
http://www.jtstar.com 13986 1321 89.7 96.6
REFERENCES
http://www.scu.edu.cn 23550 7837 64.5 77.9
[1] Yuan Wan, and Hengqing Tong, “URL Assignment Algorithm of
http://news.sina.com.cn 38763 11861 67.1 78.7 Crawler in Distributed System Based on Hash,” IEEE International
Conference on Networking, Sensing and Control, 2008.
www.xinhuanet.com 61918 10368 68.0 88.6
[2] WU Lihui, WANG Bin, and YU Zhihua, “Design and Realization of
where: a General Web Crawler,” Computer Engineering, February 2005,
NL — Number of links pp.123-124.
NLAF — Number of links after filtering
RHCF — the Rate of Hitting the cache (First %) [3] Christopher Martinez, Wei-Ming Lin, and Parimal Patel, “Optimal
RHCNF — the Rate of Hitting the cache (Non-First %) XOR Hashing for A Linearly Distributed Address Lookup in
Computer Networks,” Symposium On Architecture For Networking
This paper introduces one URL filter algorithm based on And Communications Systems, 2005, pp. 203-210.
caching and its implementation. The stability and parallel [4] Xiao-Guang Liu, and Jun Lee, “K-Divided Bloom Filter Algorithm
performances of the algorithm are verified by the and Its Analysis,” Future generation communication and networking
(fgcn 2007), 2007, 1(6-8), pp. 220-224.

1061
456

Authorized licensed use limited to: UNIVERSITI TEKNOLOGI MARA. Downloaded on June 28,2024 at 10:57:26 UTC from IEEE Xplore. Restrictions apply.