WEB SCRAPER FOR ONLINE PRICE COMPARISON 1

PROJECT REPORT

ON

“WEB SCRAPER FOR ONLINE PRICE COMPARISON”

FINAL YEAR PROJECT 2021

BACHELOR OF SCIENCE

IN

COMPUTER SCIENCE

COMPUTER SCIENCE PROJECT (COM 423)

RONGO UNIVERSITY

DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE.

Muthusi Mathew Musee

CSC/030/2017

Supervised by: Dr. Charles Oguk

WEB SCRAPER FOR ONLINE PRICE COMPARISON 2

TABLE OF CONTENTS

CHAPTER 1: INTRODUCTION………………………………………………………….2

Background of Study .....................................................................................2

Problem Statement .........................................................................................2

Aims & objectives ………...............................................................................3

CHAPTER 2: LITERATURE REVIEW……………………………………………….….5

Overview……………………………………………………………………...5

Previous Work……………………………………………………………….6

CHAPTER 3: METHODS…………………………………………………………………14

Systems Analysis…………………………………………………………….14

System Design.………………………………………………………………15

CHAPTER 4: RESULTS…………………………………………………………………...18

Implementation and Testing………………………………….…………….18

Technology and Hardware Used………………………….…….………….19

CHAPTER 5: DISCUSSION……………………………………………………………….21

Conclusion and Bibliography……………………………………………………………….23

WEB SCRAPER FOR ONLINE PRICE COMPARISON 3

Web Scraper for Online Price Comparison

CHAPTER 1: INTRODUCTION

Background of Study

Knowledge is power and information is liberating. The World Wide Web consists of an

interlinked network of information, which is presented through websites to the users. World

Wide Web has significantly changed the way we share, collect, and publish data. The amount of

presented information grows constantly. The internet can provide you with an incredibly rich

data source for every subject and topic you’ve ever dreamed of. Also we live in a perfect time

where you can easily harvest data from the web to create useful / predictive insights.

As data grows in amount, variety, and importance, business leaders must focus their

attention on the data that matters the most. Not all data is equally important to businesses or

consumers. The enterprises that thrive during this data transformation will be those that can

identify and take advantage of the critical subset of data that will drive meaningful positive

impact for user experience, solving complex problems, and creating new economies of scale.

Business leaders should focus on identifying and servicing that unique, critical slice of data to

realize the vast potential it holds.

Problem statement

Considering the customers’ busy lifestyle especially those who are living in the city area,

most of the consumers prefer to buy their needs through the internet because it save their time.

Besides, consumers always go for the cheaper price in purchasing products therefore by using

price comparison website, customers don’t have to travel from shop to shop only to survey the
WEB SCRAPER FOR ONLINE PRICE COMPARISON 4

price offered by different shops for the same product. They can just check it from the price

comparison website itself and decide where they should buy the products they need.

Consumers have the right to know whether the price they are seeing in the shops are good

deals as it is claimed or not. Thus it is an advantage for the consumers who are always aware

about the current price of a certain product so that they are not cheated by the big words

advertised by the shops. The website also beneficial to the sellers/retailers as they will be able to

advertise their products and promote their shops at the same time.

With the usage of Web as a new marketing and sales channel the quantity of content has

multiplied. Online merchants offer large packs of data to describe their products. With this

unorganized growth, it is no longer possible to manually track and record all available sources.

That moment, is when Web Scraping evolved. Automated techniques allow the collection of a

massive amount of data from the Web compared to manual data extraction.

Aims & objectives

Customers love nice deals. They flock to discounts and cherish special offers. Price

comparison sites are deal-finders and a way for users to easily locate and compare offers around

the internet and to zero in on the best price for a product they want to purchase. Even though

customers love nice deals, retailers hate for their prices to be exposed and compared. So, as

much as comparison sites may be disliked, they’re useful intermediaries between demand and

supply.

It’s not an enviable position to be the only seller in a market. If you’re the only provider

of a product or service, you will have to be both the evangelist and the market-maker. Price

comparison engines are audience magnets, and they therefore assume the market-maker role for
WEB SCRAPER FOR ONLINE PRICE COMPARISON 5

business. They can be used to expand the distribution of your e-commerce products and

influence market forces. All price comparison sites have four main functions, pick a category of

product to venture in, acquire the product data or information, compare the product from several

retailers and then keep track of that product all the time from the involved retailers.

The objective of this project is to develop an application that collects the price of specific

products from a various vendor websites and represent this information in a single site to the

user. Further, this application comprises of two major components. The first component is a

utility that is responsible on connecting to the World Wide Web, collecting the information using

the web scraping techniques and holds that data for presentation. Where the other component is a

website that is responsible for presenting on a web page the collected information.

Project justification

Nowadays people usually conduct first research on the internet regarding a particular

product or service that they want which is very convenient as compared to what consumers have

to do previously when there is no internet. Back then, consumers have to go through books,

magazines, newspaper, encyclopedia and other medium to find needed information meanwhile

yellow pages, directories, advertisements and others are useful to locate business and their

products. Everything has to be done manually and requires a lot of time, energy and resources to

find information without the help of internet. Comparison sites are known as the second type of

search technologies as it helps users find products and services according to users’ have chosen.

It helps consumers to easily compare between the alternatives available in the market and reduce

consumer search costs at the same time.

WEB SCRAPER FOR ONLINE PRICE COMPARISON 6

As much as a lot has been done to advance technology, Kenya and African in general still

remains behind in most of the small advancements and technological implementation that matter

to common people. So there are two main reasons why I felt the need to undertake this project.

First is inclusivity and the other is the access to the most relevant data in this region of the globe,

having in mind that the highest percentage of Kenya and African population are consumers other

than producers even the governments import more than they export that means most people buy

more than they sell. And I believe having the information to make a decision on the purchases

will be of great help.

On inclusivity, almost all the current and existing price comparison websites are foreign,

they present products not available or not at the reach of Kenyan consumers. First the currency

that those sites use is either Dollars $, Pound £ or Euro €. Few people really know the price of a

dollar or a Pound off head in Kenyan Shilling Ksh.

Another issue on inclusivity is the vendors or retailers that the known price comparison

sites present. They mostly consider the giant and international online retailers which to some

extent not most Kenyans have ever purchased a product from or even know the process of

buying products from them. For instance, am pretty confident that less than 10 percent of

/Kenyans have bought an item from Amazon or eBay and event this small percentage are

retailers who purchase at high volumes at wholesale prices. Shopping sites like Jumia, Kilimall

and Jiji are in most times not considered, though few Kenyan sites use them but are not

recognized.
WEB SCRAPER FOR ONLINE PRICE COMPARISON 7
WEB SCRAPER FOR ONLINE PRICE COMPARISON 8

CHAPTER 2: LITERATURE REVIEW

Overview

In order to make this project more credible, previous projects and site that are related to

price comparison has been gathered and analyzed. Hence, this chapter covers the literature

review which is the analytical, critical and objective review of existing sites. There are hundreds

of sites that are being used today and for this literature review I will review some of them. Some

of these include, Google and Yahoo shopping, Big shopping, Shopify and Shopzilla just to

mentions a few and local ones like Yoota Kenya and Shopit.co.ke. Below is a brief description of

these sites.

Previous work

Here I will look at sites that are international/foreign and also local. First let’s look at the

international ones.

1. Google Shopping

To retailers trying to identify rivals selling similar products as well as compare product's price,

Google Shopping is really a great website. Whenever it concerns to price comparison websites,

Google is without a question just at forefront. Retailers can also list their items on Google

Shopping to increase traffic to their locations. Google Shopping has been used by several drop

shippers to list their items. Customers can sort items according to pricing and sellers.

One can count on Google to ever deliver the finest results. Google gathers information from all

across the internet and delivers it to you in a manner that you can appreciate. Google Shopping's

comparison site function is an integral aspect of the search query. Everything you have would be
WEB SCRAPER FOR ONLINE PRICE COMPARISON 9

to go to the Shopping area and look for the item. When simply choose a product, you'll get a few

photographs of that as well, a product description, ratings, and price data from different of

merchants. The cheapest cost is always displayed as the first choice, however if users don't like a

particular vendor, simply may broaden the selections to see the whole table.

2. Yahoo Shopping

Yahoo Shopping’s price comparison website is similar to Google Shopping. Store owners can

add their products to the platform. You’ll find products sold by brands like Walmart and Amazon

on the platform.

The Yahoo comparison shopping engine is fairly easy to use. You simply have to enter the

product you’re looking for into the search bar and you’ll be presented with pages of results

matching your search query.

3. Shopify

Shopify is an ecommerce platform that hosts your online store. It helps organize the layout of

your store's appearance through themes. It also offers payment processing, which allows one to

accept and receive payment for the products you sell.

Shopify helps sellers post their products and the prices, this helps sellers see competitors' prices

for similar products by looking at stores that sell similar products can one determine the average

price of the products you're selling

4. Idealo

Idealo is a price comparison UK website that shows the best product prices from popular sites

like Amazon and eBay. Drop shippers are likely to find items they sell on this price comparison
WEB SCRAPER FOR ONLINE PRICE COMPARISON 10

website. Shipping prices are listed with the product price showing drop shippers the total cost of

the product for a customer.

This price comparison website is fairly simple to use. All you have to do is search for your

desired product in the search bar, and you’ll get pages’ worth of products, with their price listed

below them. If you click on any individual item, you’ll get product descriptions, the number of

offers they have, reviews and even international prices.

5. BizRate

BizRate allows customers to find the best prices, set price alerts and search through countless

deals on their price comparison search engine. Whether customers are looking for a specific

product or to scroll through collections, BizRate’s platform offers a great variety of product

deals. When it comes to the best price comparison shopping engines, BizRate is definitely high

up on the list. This is because of its user friendly website and wide range of results. A few

features that make BizRate stand out include the option to download links to PDF user manuals

for hundreds of devices and gadgets. It also has a price alert feature, which is quite simple to use.

All you have to do is enter your email address and a price threshold, and BizRate will notify you

whenever the price of your selected product has fallen within your alert range.

6. Pricena

Pricena is a great price comparison app with a website for those in the Middle East, Nigeria, or

South Africa. If one has an online store sells to these countries, monitoring product prices on this

app can help you stay competitive. You can also use their website to search product pricing.

Their app shows reviews, price fluctuations, product details, and more. This price comparison
WEB SCRAPER FOR ONLINE PRICE COMPARISON 11

app is super helpful if you’re looking to compare Amazon prices. You can use it for all sorts of

product categories like fashion, sports, laptops, or even for an iPhone price comparison.

7. Big Shopping

Bing Shopping is similar to Google in that it was launched as a complementary service to its

search offering. Bing is the Microsoft search engine and shares a data marketing approach with

its sister company LinkedIn.

8. Pricerunner

Price Runner’s UK price comparison website compares websites from top retailers such as

Amazon, ASOS, House of Fraser, and other top UK brands. Store owners can compare prices

with reputable brands to determine how to best price the products on their website. This price

comparison website has a fresh and easy-to-use interface, with a rather modern outlook. You

even have access to price histories, price alerts, price information, and reviews. And if you prefer

to buy the product locally, it’ll even direct you on where to go.

9. CamelCamelCamel

CamelCamelCamel is a price comparison platform for Amazon. Customers can look through

great deals that motivate them into an impulse purchase. The website lists the current price, the

average price from various sellers and the list price to show customers the deal they’re getting

from buying the product on Amazon. It also features product price drops which can be valuable

for store owners looking to get a competitive edge. The great thing about this comparison

shopping engine is that if you want to compare Amazon prices, you’ll also be presented with

“3rd Party New” and “3rd Party Used” product prices, so that you can find the best deals online.
WEB SCRAPER FOR ONLINE PRICE COMPARISON 12

That’s why, if you’re looking for a fast Amazon price check, CamelCamelCamel is the right

comparison shopping engine for you.

10. Shopzilla

Shopzilla is part of Connexity and is one of the original price comparison sites. Millions of

visitors check out Shopzilla every month to compare the prices of all types of products. As a site

that provides a huge shopping engine to visitors you can add your drop shipping products and

start selling more online.

11. PriceSpy

PriceSpy’s price comparison website allows customers to find the best prices for product

categories like beauty, photography, fitness, and more. Customers can narrow their search based

on various parameters to find the best deals on products. They can also use the search bar to find

the best prices and best deals online for specific items.

There are also many Kenyan based price comparison sites that are existing and are used

presently in the market.

1. Yakini Kenya

Yakini Compares prices from all major shopping malls in Kenya such as Jumia Kenya, Kilimall

Kenya, Saruk Kenya and Avechi Kenya. They have aggregated all the products from all

shopping malls in Kenya in one place so that one can enjoy their shopping with ease. It presents

a range of products like smartphones, clothing and electronics from the best Kenyan online shops

at a very cheap price. Also gives an alert of offers from all over the country when one signs up

for the alerts.

WEB SCRAPER FOR ONLINE PRICE COMPARISON 13

2. Yaoota Shopping engine

Yaoota! helps Compare Prices in Kenya and shop online from hundreds of thousands of different

products. They are buyers save money and stay informed about the price and availability of stuff.

Yaoota! is a machine built for catching up-to-date prices, deals in Kenya and product

information from serious, trustworthy online stores that sell original and high quality products.

They display products based on best price in Kenya and do not favor an online store over

another, well at least that’s what they claim. They cover most of the products that are available in

online shops from electronics, footwear to clothing and kitchen ware and food products.

3. NaijaTechGuide

NaijaTechGuide is among the leading Gadget focused tech blog with a sizable audience in

Kenya. The site offers extensive information about tech products and services. With the addition

of this price comparison search engine, one can find the best prices for gadgets and software at

leading online stores in Kenya.

NaijaTechGuide Price Comparison Search Engine offers prices from only the most reputable

online stores in Kenya. So, one does not just get the best prices, but they also get the best quality

products.

4. Sokompare

This is focused on car sales and insurance and banking, Sokompare aims to allow Kenyans to

easily buy, sell and compare cars, insurance quotes and banking services, without the hassles of

shopping around, ensuring maximum efficiency for the ever-busy Kenyans.

WEB SCRAPER FOR ONLINE PRICE COMPARISON 14

Coined from Soko, Swahili for Market and Kompare slang for compare, the site aims to bring

trusted, reliable prices online by working with partners from various sectors in the market.

5. Trivago

Trivago’s hotel search allows users to compare hotel prices in just a few clicks from more than

400 booking sites for 1.8 million+ hotels in over 190 countries. With 1.4 billion visits annually to

this site, travelers regularly use the hotel comparison to compare deals in the same city.

6. Hotels Scanner

Hotels-scanner.com is a convenient search tool for comparing hundreds of hotels all around the

world and giving you the best and cheapest hotels. They scan prices in all global booking

systems and provide the information in a clear and user-friendly manner. Upon comparing prices

and finding your fine variant, you can then book it with any method you prefer.

7. Hotels combined

Hotels Combines helps you to compare all the hotels in top travel sites in one simple search and

find just what you’re after. You also get to compare the best hotel deals in a second. You can

sign up with Hotels Combined to get the best hotel deals straight to your inbox.

8. Kayak

One thing I like with Kayak.com is the side by side comparison in an absolutely user-friendly

interface which is just flawless. You can then link with the many booking sites available to book

your fine variant. You can also compare flight and even cruise prices here.

9. Fare compare
WEB SCRAPER FOR ONLINE PRICE COMPARISON 15

Fare compare helps you search thousands of hotels and compare prices against all the top travel

sites so you can find the best deal. Whether you are on the lookout for a high-end, luxury hotel

reservation, or you want to stay on the cheap with budget accommodations, or maybe something

in between. Whatever you’re looking for, Fare Compare’s search technology makes it simple to

compare hundreds of hotels from all the big search sites as well as prices from the hotels directly.

So you know you’re getting the best deal possible. You can also compare flight and car fares.
WEB SCRAPER FOR ONLINE PRICE COMPARISON 16

CHAPTER 3: METHODS

Systems Analysis

This section will include information about the project basses and the best features of a

price comparison site should look like. This will explain why these specific features are needed

in the site. This section will also present the relevant project phases ad milestones. Will also

present what my system has over the existing sites presented in the previous Chapter.

Compared to the price comparison site from its first being established and act as online

classifieds, nowadays comparison sites able to do a lot of things such as draw the products’ cost,

able to trawl sites and give the sites much greater control over the users with the help of current

technology. It gathers and aggregate price, products and other relevant information from third-

party sellers and presents it to the consumers with its own way.

First Stage: Project Planning

This is the stage where I identified the problem related to the project and determined the

significance of the project. I also outlined the objective and also scope of the project and the

feasibility of the project work assurance for development within the time frame given. I studied

the solution to the problem statement and the types of system to develop and tools to us for

developing the system.

Second Stage: Research on any Existing Similar Systems

This is generally what has been covered in chapter 2 of this report. I performed this stage

to check if any similar system exists. The main objective of doing research on similar existing
WEB SCRAPER FOR ONLINE PRICE COMPARISON 17

system was to know how it works, what concept is being applied in the systems, what is being

computed by the systems and how the systems helps solving the problem of window shopping.

Third Stage: Drafting the Main Components of the System

After the research on similar existing systems, the next step needed was to identify what

will be the main components making up the system to be developed in order for the visitors of

this website to use it services. Based on time given, this project will be simple and easy to access

and use. It will be a single site website with only smartphones as products being compared, from

just two most recognized e-commerce sites in Kenya, Jumia and Kilimall. The site will present

the latest and most selling smartphones in those the two retailers and their prices and a mention

of the price source.

Fourth Stage: System Development

The last phase was to develop the system and how it will work. This will have a clear

picture and understanding on how the system will operate and avoid a system that does not

solving the problem it intended to solve or have a hard to use the system. This stage is the base

for the next subtopic “System Design”.

System Design

This section will include the design techniques used in this project.

This project is aimed at easing the job of gathering product prices from different online

retailers. This is why the best technology as of the time of the development of this project is used

in its implementation. To get data from the web pages the Web scraper technology is the best to

automate the process to fetch data from multiple web pages by their unified resource locators.
WEB SCRAPER FOR ONLINE PRICE COMPARISON 18

Web scraping, also known as web extraction or harvesting, is a technique to extract data

from the. Commonly, web data is scrapped utilizing Hypertext Transfer Protocol or through a

web browser. This is accomplished either manually by a user or automatically by a bot or web

crawler. Due to the fact that an enormous amount of heterogeneous data is constantly generated

on the WWW, web scraping is widely acknowledged as an efficient and powerful technique for

collecting big data. Technically Web Scraping involves the process of querying a source,

retrieving the results page and parsing the page to obtain the results.

Enormous amounts of source information, available on the World Wide Web, are still in

the format of a Hypertext Markup Language page which makes the process of web scraping

generally universal. Automated extraction is difficult, because the intended reader was a human.

Data collection from online sources became one of the market research methods. It offers

much faster response, compared to a classical surveying. Consumers are active in the online

world and share their experience, frustration or motivation. Companies that wish to learn more

from consumers can add online sources of information. Web scraping is one of the method to

collect such data. Targeted data collection from e-shop and advertising servers helps to update

Indexes. Which are based on frequently changed prices. Indexes built through automated Web

scraping can offer more frequent update intervals.

Incompatible enterprise technologies are common by larger projects. Still a unified

presentation of data from several systems is necessary. In some specific cases the solution is

based on Web Scraping. Also Social media has become one of the major data sources for

quantitative communication research over the past decade. By employing simple programing

tools, researchers can extract relevant messages from social media platforms for various research

purposes.
WEB SCRAPER FOR ONLINE PRICE COMPARISON 19

This project employed the waterfall methodology as the most appropriate development

method to use for this particular project. With the four main phases mentioned above; Project

Planning, Research on any Existing Similar Systems, Drafting the Main Components of the

System, and System Development.

WEB SCRAPER FOR ONLINE PRICE COMPARISON 20

CHAPTER 4: RESULTS

Implementation and Testing

To tell whether the system will function as expected real time data extraction had to be

used, Sample phones and their respective prices from Jumia and Kilimall are used. The prices of

similar phones compared. The system ease of use and navigation was rated by asking few friend

to try and use the system to compare the smartphone prices and the results were encouraging as

most of them found it simple and easy to navigate.

Technology and Hardware Used

The tools used within this project to demonstrate the web scraping advantages based on

Prices in top Kenyan e-shops. Python language used for developing the web scraping agent and

Microsoft Visual Studio Code is a development environment. Further, the Beautiful Soup used to

connect to the internet and download the targeted web page. The technology used to implement

this part of the project is python Django development server. Python Django is used to develop

server scripts, JavaScript and HTML to develop client side what will render on the browser.

HTML -Hypertext Markup Language - This is currently the core of the web world; it is a

language used to makeup web page. It is the glue that holds everything together. Although have

HTML for the implementation of this system and it is highly compatible with extensible HTML

(XHTML) which is designed to be a replacement of HTML made to handle data and is also

portable between different browsers and platforms with little or no alterations in code.

Django (Python framework) – Powerful python framework for back-end web scripting

high security and speed in code execution ease to understand the codes and less codding hence

good for delivering in time especially for a project with limited time like this. Also has a large
WEB SCRAPER FOR ONLINE PRICE COMPARISON 21

and great community in case one needs help and a well-organized documentation. Testing is

done with Django development server.

WEB SCRAPER FOR ONLINE PRICE COMPARISON 22

CHAPTER 5: DISCUSSION

This Project allows user to view prices from two leading Kenyan online Retailers and see

how much each seller is charging for thee similar product.

Constrains

Electricity blackouts since it’s the main source of power for my development laptop.

Slow internet connectivity, many resources am getting from the internet and mostly will

use cellular data for connection.

Time period - time for development of the whole project is somehow short than I need

this may course some things being rushed, which may lead to unexpected errors at the very last

time.

Lack of deep knowledge on the technologies used in the project having in mind that

everything is entirely self-taught.

Web scraping process uses the parser wrapper to navigate through the HTML document

to extract the targeted data. The web scraping search path to locate the required data is unique for

each web page, and this determined by the page HTML structure, which conclude that Web

scraping techniques lacks generalization. This is a challenge since for each web page I will need

to script it separate code, which will be tedious and repetitive.

Legalities

The purpose of websites is to provide information to the human through visiting and

viewing the website pages at a computer. In contrast, web scraping subvert that by finding a way

to simulate human browsing by modifying headers, forging POST requests and other techniques.
WEB SCRAPER FOR ONLINE PRICE COMPARISON 23

Web scraping requires ingenuities to solve problems and determine the way to obtain the

needed data. In addition, there often a few guidelines that has been tested and proved to be

reliable to follow while different websites required tailored code path to retrieve data from the

websites. Further, this sort of techniques in web scraping might consider hacking act. But

Following the terms of use and the documents copyright on the website, it is a good rule of

thumb to web scraping to avoid getting in trouble. Unless explicitly prohibited by the terms of

service, there is no fundamental difference between accessing a website through a browser, and

accessing it through an automated script.

Conclusion

In analyzing, designing, implementing, and maintaining standards, I considered these

characteristics as the foundation. These standards are universal for price comparison sites. Web

scraping is a valuable method to obtain data of the World Wide Web. There are many kind of

applications for web content mining and web monitoring that use web scraping to retrieve their

needed data from the web. The web scraping process imitate human exploration to the World

Wide Web data. Web scraping is advantageous and entirely legal and can provide powerful

insight that can be useful to business owners even used by government agencies. Even scraping

social sites can be source of personal information for individual profiling by law enforcement

agencies.
WEB SCRAPER FOR ONLINE PRICE COMPARISON 24

References and Bibliography

Mehak, S., Zafar, R., Aslam, S., & Bhatti, S. M. (2019, January). Exploiting filtering approach

with web scrapping for smart online shopping: Penny wise: A wise tool for online

shopping. In 2019 2nd International Conference on Computing, Mathematics and

Engineering Technologies (iCoMET) (pp. 1-5). IEEE.

Ullah, H., Ullah, Z., Maqsood, S., & Hafeez, A. (2018). Web Scraper Revealing Trends of

Target Products and New Insights in Online Shopping Websites. INTERNATIONAL

JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS, 9(6), 427-

432.

Vargiu, E., & Urru, M. (2013). Exploiting web scraping in a collaborative filtering-based

approach to web advertising. Artif. Intell. Res., 2(1), 44-54.

Ambre, A., Gaikwad, P., Pawar, K., & Patil, V. (2019). Web and android application for

comparison of e-commerce products. no, 4, 266-268.

Singrodia, V., Mitra, A., & Paul, S. (2019, January). A review on web scrapping and its

applications. In 2019 International Conference on Computer Communication and

Informatics (ICCCI) (pp. 1-6). IEEE.