PWP-22616 WEB-SCRAPER

MICRO PROJECT REPORT

ON
“ WEB-SCRAPER ”

Submitted by
Shivam Sainath Korpakwad
Guided by
Mr. Dad K. V.
TO
DEPARTMENT OF COMPUTER ENGINEERING
GRAMIN TECHNICAL & MANAGEMENT
CAMPUS, VISHNUPURI, NANDED-431606

MAHARASHTRA STATE BOARD OF TECHNICAL EDUCATION

(MSBTE), MUMBAI

ACADEMIC YEAR 2024-25

I
PWP-22616 WEB-SCRAPER

MICRO PROJECT REPORT

ON
“ WEB-SCRAPER ”

Submitted by
Shivam Sainath Korpakwad
Guided by
Mr. Dad K.V
TO
In Partial Fulfillment for the Award of the Diploma In

DEPARTMENT OF COMPUTER ENGINEERING

GRAMIN TECHNICAL & MANAGEMENT CAMPUS,
VISHNUPURI, NANDED - 431606

II
PWP-22616 WEB-SCRAPER

ACADEMIC YEAR 2024-25

III
PWP-22616 WEB-SCRAPER

CERTIFICATE

This is to certified that the project entitled

“WEB SCRAPER”
Being submitted by Mr. Shivam Sainath Korpakwad to State Board Of
Technical Education Mumbai as a partial fulfillment of award of Diploma in
COMPUTER ENGINEERING is record of Bonafide work carried out by his under
supervision and guidance of Mr. Dad K. V. The assigned project is performed
satisfactorily in the academic year 2024-25.

Mr.Dad K.V Mr. Pathan F. S.

Guide Head of Department

Dr. Pawar V. S.
Principal

IV
PWP-22616 WEB-SCRAPER

ACKNOWLEDGEMENT

I take this opportunity to express my deep sense of gratitude to words Mr. Dad K.V.
Course in charge of Programming With Python who has been a constant source of
inspiration to us and without his valuable guidance this work could not possible.

I am thankful to all faculty members of my department also for their guidance,

support and encouragement for the accomplishment of our micro-project. I would like to
thankful to Mr. Pathan F. S. HOD of COMPUTER ENGINEERING (Polytechnic) for his
valuable comments and suggestion for me to improve my creativity regarding project
work.

I also express my sincere thanks to my friends for their assistance and comments
for the betterment of this micro project.

Sincerely:
Mr. Shivam Sainath Korpakwad

V
PWP-22616 WEB-SCRAPER

Department of Computer Engineering

VI
PWP-22616 WEB-SCRAPER

ANNUEXURE II

Evaluation Sheet for the Micro Project

Academic Year : 2024 – 2025 Name of faculty : Mr. Dad K. V.

Course : Programing with Python
Course code : 22616 Semester : 6th Semester
Title of Project : WEB-SCRAPER

Course outcome:

Major Learning Outcomes achieved by doing the Project:

1. Practical Outcomes

These outcomes focus on the hands-on skills and competencies the learner will gain by
completing the Warli Art Shop project (without using a database):

 Static Web Design and Layout: Learners will build a visually appealing web interface for an art
shop using HTML, CSS, and Bootstrap.
 Responsive Design Implementation: Learners will create a layout that adjusts gracefully
across different screen sizes using CSS media queries and responsive practices.

2. Unit Outcomes in Cognitive Domain:

This domain focuses on how learners understand, process, and apply knowledge:

 Understanding of Web Technologies: Learners will grasp the purpose and interaction of
HTML, CSS, PHP, and JavaScript in static web development.

 Application of PHP for Page Reusability: Learners will apply basic PHP to include common UI
components across multiple pages without duplicating code.

 Problem-Solving with Layout and Styling: Learners will analyze layout issues and apply CSS
strategies to improve visual structure.

3. Outcomes in Affective Domain:

This domain relates to attitudes, values, and personal growth during the learning process:

 Aesthetic Appreciation: Learners will cultivate an appreciation for traditional Warli art and
express it through respectful and creative design.
 Sense of Ownership and Confidence: Learners will build confidence by completing a
functional, stylized mini e-commerce site independently.

VI
I
PWP-22616 WEB-SCRAPER

 Attention to Detail: Learners will demonstrate precision in styling, layout, and image
presentation.
 Comments / Suggestions about team work/ leadership/ internship/ inter-personal
communication (if any)

Roll Student Name Marks out of Marks out of Total

No. 6 for 4 for out of
performance performance in 10
in oral/presentation
group activity
24 Shivam Sainath Korpakwad

Mr. Dad K. V.
(Name & Signature of facul

VI
II
PWP-22616 WEB-SCRAPER

INDEX

Sr. no. Point Page no.

s
1 Abstract 2

2 Introduction 3

3 Brief description 4

4 What is Web-Scraper 5

5 Code 6

6 Output 7

7 Working of Web-Scraper 8

8 Web-Scraper 9

9 Advantages 10

10 Disadvantages 11

11 Future Scope 12

12 Conclusion 13

13 References 14

GT&MC, Vishnupuri, Nanded. 1

PWP-22616 WEB-SCRAPER

ABSTRACT
The Web Scraper project is a Python-based tool designed to automate the
extraction of webpage titles from a given URL. By utilizing the requests library for sending
HTTP requests and BeautifulSoup for parsing the HTML content, this scraper fetches the
title of a webpage by targeting the <title> tag within the page's HTML structure. The tool is
built to handle common errors, such as invalid URLs, network issues, or missing title tags,
providing clear feedback to the user in case of failure. This project is useful for scenarios
like SEO analysis, web crawling, and metadata collection. It serves as an introductory tool
for web scraping, showcasing the basic concepts of making requests, parsing HTML, and
handling errors effectively. The Web Scraper project is an efficient solution for anyone
seeking a simple way to gather webpage titles, and it can be further extended to handle
more complex scraping tasks.

GT&MC, Vishnupuri, Nanded. 2

PWP-22616 WEB-SCRAPER

INTRODUCTION

Introduction to Web Scraping

Web scraping is a technique used to extract data from websites. It involves fetching a web
page and extracting the desired information from its content. This process can be automated
using various programming languages and libraries, making it a powerful tool for data
collection, analysis, and research.

Why Use Web Scraping?

Data Collection: Web scraping allows users to gather large amounts of data from multiple
sources quickly and efficiently. This is particularly useful for market research, competitive
analysis, and academic studies.

Automation: Manual data collection can be time-consuming and prone to errors. Web
scraping automates this process, allowing users to focus on analysis rather than data
gathering.

Real-Time Data: Many websites provide dynamic content that changes frequently. Web
scraping can be used to collect real-time data, which is essential for applications like price
monitoring, news aggregation, and social

GT&MC, Vishnupuri, Nanded. 3

PWP-22616 WEB-SCRAPER

BRIEF DESCRIPTION

Web scraping is the automated process of extracting data from websites. It involves
sending requests to web pages, retrieving their HTML content, and parsing that content to
extract specific information. This technique is widely used for data collection, market
research, and analysis, allowing users to gather large amounts of data quickly and
efficiently.

Key benefits of web scraping include:

Automation: Reduces the time and effort required for manual data collection.
Real-Time Data: Enables access to frequently updated information.
Structured Data: Converts unstructured web content into structured formats for easier
analysis.
Common tools for web scraping include programming languages like Python, along
with libraries such as BeautifulSoup and Scrapy. While web scraping is powerful, it's
important to consider legal and ethical guidelines, including respecting a website's terms
of service.

GT&MC, Vishnupuri, Nanded. 4

PWP-22616 WEB-SCRAPER

WHAT IS WEB-SRAPER

A web scraper is a software tool or program designed to automatically extract data

from websites. It simulates human browsing behavior to retrieve web pages and then
parses the HTML or XML content to extract specific information. Web scrapers can be
used for various purposes, including data collection, market research, price monitoring,
and content aggregation.
A web scraper in Python typically follows a systematic process to extract data from
websites. Below is a step-by-step breakdown of how a web scraper works, along with
code examples to illustrate each step.

1. Import Required Libraries

To start, you need to import the necessary libraries. The most commonly used libraries for
web scraping in Python are requests for making HTTP requests and BeautifulSoup for
parsing HTML.

2. Send an HTTP Request

The first step in web scraping is to send an HTTP request to the target website to retrieve
its HTML content. This is done using the requests library.

3. Parse the HTML Content

Once you have the HTML content, the next step is to parse it using BeautifulSoup. This
allows you to navigate the HTML structure and find the elements you want to extract.

GT&MC, Vishnupuri, Nanded. 5

PWP-22616 WEB-SCRAPER

CODE

import requests
from bs4 import BeautifulSoup

def get_page_title(url):
try:
response = requests.get(url)
response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)

soup = BeautifulSoup(response.content, "html.parser")

title_tag = soup.find("title")

if title_tag:
return title_tag.text.strip()
else:
return "Title not found."

except requests.exceptions.RequestException as e:
return f"Error: {e}"
except Exception as e:
return f"An unexpected error occurred: {e}"

def main():
url = input("Enter the URL of the webpage: ")
title = get_page_title(url)

if title:
print(f"Title: {title}")

if _name_ == "_main_": # Corrected this line

main()

GT&MC, Vishnupuri, Nanded. 6

PWP-22616 WEB-SCRAPER

OUTPUT

GT&MC, Vishnupuri, Nanded. 7

PWP-22616 WEB-SCRAPER

WORKING OF WEB-SRAPER

A web scraper in Python typically follows a systematic process to extract data from
websites. Below is a step-by-step breakdown of how a web scraper works, along with
code examples to illustrate each step.

1. Import Required Libraries:

To start, you need to import the necessary libraries. The most commonly used libraries for
web scraping in Python are requests for making HTTP requests and BeautifulSoup for
parsing HTML.
2. Send an HTTP Request:
The first step in web scraping is to send an HTTP request to the target website to retrieve
its HTML content. This is done using the requests library.

3. Parse the HTML Content Once you have the HTML content, the next step is to
parse it using BeautifulSoup. This allows you to navigate the HTML structure and find the
elements you want to extract.

A web scraper is a tool or program designed to automatically extract data from

websites. It simulates human browsing behavior to retrieve web pages and then parses
the content to extract specific information. Web scrapers are widely used for various
applications, including data collection, market research, content aggregation, and more.

GT&MC, Vishnupuri, Nanded. 8

PWP-22616 WEB-SCRAPER

WEB-SRAPER

A web scraper is a software tool or program designed to automatically extract data

from websites. It simulates human browsing behavior to retrieve web pages and then
parses the HTML or XML content to extract specific information. Web scrapers can be
used for various purposes, including data collection, market research, price monitoring,
and content aggregation.

GT&MC, Vishnupuri, Nanded. 9

PWP-22616 WEB-SCRAPER

ADVANTAGES
1. Automated Data Collection:
Web scrapers automate the process of gathering data from multiple web pages,
significantly reducing the time and effort required for manual data collection.

2. Efficiency:
They can extract large volumes of data quickly and efficiently, making them ideal
for tasks that involve monitoring changes over time, such as price tracking or news
aggregation.

3. Access to Unstructured Data:

Web scrapers can convert unstructured data (like HTML content) into structured
formats (like CSV or JSON), making it easier to analyze and use in applications.

4. Real-Time Data:
They can be set up to run at regular intervals, allowing users to collect real-time
data from websites that frequently update their content.

GT&MC, Vishnupuri, Nanded. 10

PWP-22616 WEB-SCRAPER

DISADVANTAGES

1.Legal and Ethical Issues:

Web scraping can lead to legal challenges if it violates a website's terms of service or
copyright laws. It's essential to respect a site's robots.txt file and adhere to ethical
guidelines.

2.Website Changes:
Websites frequently change their structure, which can break scrapers. This requires
ongoing maintenance and updates to the scraping code to ensure it continues to function
correctly.

3.IP Blocking:
Many websites implement measures to prevent scraping, such as rate limiting or IP
blocking. Excessive requests from a single IP address can lead to temporary or
permanent bans.

4.Data Quality:
The quality of the extracted data can vary based on the website's structure and the
scraper's design. Poorly designed scrapers may extract inaccurate or incomplete data.

GT&MC, Vishnupuri, Nanded. 11

PWP-22616 WEB-SCRAPER

FUTURE SCOPE

Future scope of a web scraper project can be expanded in various directions,

depending on the specific goals, technologies, and applications. Here are several potential
areas for enhancement and development.

The future scope of a web scraper project is vast and can be tailored to meet
specific user needs and industry demands. By incorporating advanced features, improving
user experience, and ensuring ethical practices, the project can evolve into a powerful tool
for data extraction and analysis, serving a wide range of applications across various
domains.

GT&MC, Vishnupuri, Nanded. 12

PWP-22616 WEB-SCRAPER

CONCLUSION
A web scraper is a tool or program designed to automatically extract data from
websites. It simulates human browsing behavior to retrieve web pages and then parses
the content to extract specific information. Web scrapers are widely used for various
applications, including data collection, market research, content aggregation, and more

GT&MC, Vishnupuri, Nanded. 13

PWP-22616 WEB-SCRAPER

REFERENCES

 https://pypi.org/project/pydroid/

 https://openai.com/index/chatgpt/

 https://www.python.org/downloads/

GT&MC, Vishnupuri, Nanded. 14