LAB MANUAL

COLOR DETECTION AND

SORTING USING COMPUTER
VISION KIT

(Do -It- Yourself)

Version: 1.1.0
Copyright 2024, PRAGYATMIKA, The Power of Intelligence

1
Web : www.pragyatmika.co.in Email : [email protected]
 Colour Sorting Robot using OpenCV

1. PROBLEM STATEMENT

In the field of automated systems and robotics, colour detection and sorting is a
fundamental task that can be applied to various industrial and commercial applications. The
goal of this project is to design and develop a Colour Sorting Robot using OpenCV, a
computer vision library, which can accurately detect and count the number of different
coloured objects in real-time video streams. The system should be capable of:

1. Capturing live video feed from a camera or webcam.

2. Identifying multiple predefined colours including red, green, blue, orange, black,
pink, yellow, purple, and brown.
3. Counting the occurrence of each colour in each video frame.
4. Displaying the colour counts on the video feed in real-time.
5. Providing a user-friendly interface to visualize the colour detection results.

The system aims to serve as a foundation for a colour sorting robot, where the colour
detection capabilities can be used to identify and categorize objects based on their colours.
This project will utilize OpenCV for image processing and colour detection in the HSV colour
space, allowing for robust colour identification under various lighting conditions.

The final output should display the live video feed with overlaid colour count information,
updating in real-time as the camera captures new frames. The system should be able to
handle multiple colours simultaneously and be easily extendable to include additional
colours if needed.

2. THE DESCRIPTION

This project implements a real-time colour detection system using computer vision
techniques, specifically leveraging the OpenCV library. The system is designed to identify
and quantify the presence of various colours in a live video stream, serving as a foundation
for a colour sorting robot.
2
Web : www.pragyatmika.co.in Email : [email protected]
Key features of the project include:

1. Real-time Video Processing: The system captures live video input from a webcam or
camera, processing each frame to detect colours.
2. Multi-Colour Detection: It can simultaneously detect multiple colours, including red,
green, blue, orange, black, pink, yellow, purple, and brown.
3. HSV Colour Space: The colour detection is performed in the HSV (Hue, Saturation,
Value) colour space, which is more robust to lighting variations compared to RGB.
4. Colour Counting: For each frame, the system counts the number of pixels that fall
within the defined colour ranges, providing a quantitative measure of each colour’s
presence.
5. Live Display: The processed video feed is displayed in real-time, with overlaid text
showing the count for each detected colour.
6. User Interface: A simple graphical interface shows the live video feed with colour
detection results, allowing users to visually confirm the system's performance.
7. Extensibility: The colour range definitions are stored in a dictionary, making it easy
to add or modify colour definitions.

This project demonstrates practical applications of computer vision in robotics and

automation. It could be extended to sort objects based on colour in manufacturing
processes, quality control in production lines, or even in educational settings to teach
principles of computer vision and colour theory.

The system is designed to run continuously, processing and displaying results in real-time
until the user chooses to exit the application. This real-time capability makes it suitable for
integration with physical robotic systems for immediate colour-based decision making and
sorting.

3
Web : www.pragyatmika.co.in Email : [email protected]
 3. SOFTWARE AND HARDWARE REQUIREMENTS
A. Software Tools Required:
1.Arduino IDE:
The Arduino IDE (Integrated Development Environment) is an
open-source software platform designed primarily for
programming Arduino boards, but it also supports a wide
range of other microcontroller platforms, including ESP32. It
provides a user-friendly interface for writing, compiling, and
uploading code to various microcontrollers. The IDE is cross-
platform, available for Windows, macOS, and Linux, making it accessible to a broad user base.
It comes with a text editor for writing code, a message area for feedback, a text console for
output, and a toolbar with common function buttons. The Arduino IDE uses a simplified
version of C++, making it easier for beginners to start programming microcontrollers. It
includes a rich set of built-in libraries for common tasks and supports the addition of third-
party libraries, expanding its capabilities. For more advanced users, the IDE also allows
integration with external tools and supports programming in standard C++ when needed.
With its extensive documentation, large community support, and compatibility with
numerous hardware platforms, the Arduino IDE has become a popular choice for both
hobbyists and professionals working on embedded systems projects.The Arduino IDE
(Integrated Development Environment) is an open-source software platform designed
primarily for programming Arduino boards, but it also supports a wide range of other
microcontroller platforms, including ESP32. It provides a user-friendly interface for writing,
compiling, and uploading code to various microcontrollers. The IDE is cross-platform,
available for Windows, macOS, and Linux, making it accessible to a broad user base. It comes
with a text editor for writing code, a message area for feedback, a text console for output,
and a toolbar with common function buttons. The Arduino IDE uses a simplified version of
C++, making it easier for beginners to start programming microcontrollers. It includes a rich
set of built-in libraries for common tasks and supports the addition of third-party libraries,

4
Web : www.pragyatmika.co.in Email : [email protected]
expanding its capabilities. For more advanced users, the IDE also allows integration with
external tools and supports programming in standard C++ when needed. With its extensive
documentation, large community support, and compatibility with numerous hardware
platforms, the Arduino IDE has become a popular choice for both hobbyists and professionals
working on embedded systems projects.

2. Python IDE:

Python is a high-level, interpreted programming language known for its

simplicity, readability, and versatility. Created by Guido van Rossum and first
released in 1991, Python has grown to become one of the most popular
programming languages in the world. It emphasizes code readability with its
notable use of significant whitespace, making it easier for beginners to learn and
for experienced programmers to maintain.

Python supports multiple programming paradigms, including procedural, object-oriented,

and functional programming. Its comprehensive standard library, often described as
"batteries included," provides tools for a wide range of tasks, from web development to
scientific computing. Python's extensibility through a vast ecosystem of third-party
packages, managed through tools like pip and conda, further enhances its capabilities.

5
Web : www.pragyatmika.co.in Email : [email protected]
Key features of Python include:

1. Easy to learn and read syntax

2. Dynamic typing and automatic memory management
3. Support for multiple programming paradigms
4. Large and active community, providing extensive resources and libraries
5. Cross-platform compatibility (Windows, macOS, Linux, and more)
6. Excellent for data analysis, artificial intelligence, machine learning, web
development, and automation

Python is widely used in various fields, including web and software development, data
analysis, artificial intelligence, scientific computing, and automation.

3.Visual Studio Code IDE:

Visual Studio Code (VS Code) is a free, open-source code editor

developed by Microsoft. It has gained immense popularity among
developers due to its lightweight nature, speed, and extensive
feature set. VS Code is designed to be a versatile tool that supports
a wide range of programming languages and development tasks.

Key features of Visual Studio Code include:

1. Cross-platform compatibility: Available for Windows, macOS, and Linux.

2. IntelliSense: Provides smart completions based on variable types, function
definitions, and imported modules.
3. Debugging support: Includes built-in debugging tools for various languages.
4. Git integration: Offers source control features directly within the editor.
5. Extensibility: Supports a vast marketplace of extensions that add new languages,
themes, debuggers, and tools.

6
Web : www.pragyatmika.co.in Email : [email protected]
 6. Customizability: Allows users to adjust settings, keybindings, and appearance to their
preferences.
7. Integrated terminal: Provides a built-in command-line interface.
8. Live Share: Enables real-time collaborative editing and debugging.

VS Code's architecture is based on Electron, allowing it to combine the simplicity of a code

editor with the advanced features of an IDE. Its lightweight core can be enhanced with
extensions to create a personalized development environment tailored to specific needs.

The editor also features regular updates, bringing new features and improvements based on
user feedback and evolving development practices. This responsiveness to the developer
community has contributed significantly to its widespread adoption across various
programming domains.

B. Hardware Required:

B.1. Esp32 Cam:

The ESP32 camera module is a compact, low-cost, and versatile development board that
combines the powerful ESP32 microcontroller with a digital camera. It's designed for
Internet of Things (IoT) applications, especially those requiring image capture and
processing capabilities.

Key features of the ESP32 camera module include:

1. Processor: Dual-core ESP32 chip with Wi-Fi and Bluetooth capabilities.

2. Camera: Typically uses the OV2640 sensor, capable of capturing 2MP images.

7
Web : www.pragyatmika.co.in Email : [email protected]
 3. Programmability: Can be programmed using the Arduino IDE, ESP-IDF, or
MicroPython.
4. Connectivity: Built-in Wi-Fi allows for wireless image transmission and remote
control.
5. GPIO pins: Provides access to various GPIO pins for additional sensors or actuators.
6. Low power consumption: Suitable for battery-powered applications.
7. Small form factor: Compact size makes it ideal for space-constrained projects.
8. Cost-effective: Offers a good balance of features and affordability.

B.2.Base board for esp32 cam

The base board for the ESP32-CAM is an auxiliary board designed to enhance the
functionality and ease of use of the ESP32-CAM module. This board serves several important
purposes:

1. Power Management: It typically includes a voltage regulator to provide stable 5V or

3.3V power to the ESP32-CAM, often from a USB input or battery source.
2. USB-to-UART Bridge: Most base boards incorporate a USB-to-UART chip (like
CH340G or CP2102) to facilitate easy programming and communication with a
computer via USB.
3. Reset and Flash Buttons: These physical buttons allow for easy resetting of the
ESP32-CAM and putting it into flashing mode for uploading new firmware.

8
Web : www.pragyatmika.co.in Email : [email protected]
 4. Pin Breakout: The base board often exposes the GPIO pins of the ESP32-CAM in a
more accessible format, sometimes using standard pin headers.
5. Antenna Connector: Certain models provide an external antenna connector to
improve Wi-Fi range.
6. SD Card Slot: While the ESP32-CAM has an onboard microSD slot, some base boards
offer an additional, more accessible SD card slot.
7. LED Indicators: Often include LEDs for power, data transmission, and other status
indications.
8. Camera Lens Options: Some base boards are designed to accommodate different
camera lenses or lens holders.

The base board significantly improves the usability of the ESP32-CAM by addressing its main
limitations, particularly in terms of power supply and programming interface. It transforms
the compact camera module into a more robust development platform, making it easier to
prototype and develop projects without the need for additional circuitry or adapters.

B.3.USB cable

A USB cable is an essential tool for programming and powering the ESP32 CAM module.
While the ESP32 CAM itself does not have a built-in USB-to-serial interface, a USB cable can
be used in conjunction with a USB-to-serial converter to program the module and upload
code from a computer.

4. PROGRAMMING LANGUAGES USED

This project leverages two main programming languages: Arduino C and Python. Each
language plays a distinct and crucial role in the development and functionality of the Smart
Color sorting Robot using the ESP32 CAM module.

4.1. C++: For Generating Dynamic IP Address on ESP32 CAM using Arduino IDE

C++ is the primary language used for programming the ESP32 CAM module. The ESP32 CAM
is initially configured using the Arduino IDE, a popular development environment for
9
Web : www.pragyatmika.co.in Email : [email protected]
embedded programming. The setup involves writing and uploading Arduino code (sketch) to
the ESP32 CAM, enabling it to connect to a Wi-Fi network and stream video data.

● Network Configuration: The Arduino C++ code is used to configure the ESP32 CAM to
connect to a local Wi-Fi network. The code typically includes the SSID (network
name) and password of the Wi-Fi network. When the ESP32 CAM is powered on, it
uses this information to connect to the network and requests a dynamic IP address
from the router using DHCP (Dynamic Host Configuration Protocol). This allows the
ESP32 CAM to obtain an IP address automatically, making it accessible over the
network.
● Camera Initialization and Stream Setup: The C++ code initializes the camera module
and sets up an HTTP server to stream the video feed. Once the ESP32 CAM has a
dynamic IP address, the video stream can be accessed via this IP address using any
web browser or a software client like OpenCV.
● Dynamic IP Address Assignment: By using the DHCP functionality, the ESP32 CAM
automatically receives an IP address from the router each time it connects. This IP
address can be used to access the camera's live video stream. The dynamic IP makes
the system more flexible and easier to integrate into different network
environments without manual IP configuration.

4.2. Python: For Colour sorting Using ESP32 CAM Video Stream

Python is pivotal in the development of the Colour Sorting Robot using OpenCV by enabling
efficient color detection and sorting through real-time video processing. Its combination of
simplicity and powerful libraries like OpenCV makes it an ideal choice for implementing
computer vision tasks. Here's how Python is used in this project:

● Accessing the Video Stream:

Python uses the OpenCV library to capture the video stream from a webcam or any
compatible video capture device. By initializing the video capture object
(cv2.VideoCapture(0)), Python can access live video frames for further processing.
10
Web : www.pragyatmika.co.in Email : [email protected]
 This allows the robot to analyze the visual input in real-time, making it suitable for
dynamic environments.

● Color Detection:

Once Python captures each frame from the video stream, it leverages OpenCV's
image processing functions to detect specific colors. The process involves:

➢ Converting the Frame to HSV Color Space: The frame is converted from BGR
(Blue, Green, Red) to HSV (Hue, Saturation, Value) color space using
cv2.cvtColor(frame, cv2.COLOR_BGR2HSV). HSV is more effective for color
detection as it separates chromatic content (color) from intensity.
➢ Applying Color Masks: Python applies predefined HSV range masks to isolate
different colors. Using cv2.inRange(), it creates a binary mask for each color
that highlights the regions in the frame matching the color's HSV range.
➢ Counting Color Pixels: The script counts the number of non-zero pixels in
each color mask using cv2.countNonZero(), providing a count of how many
pixels in the frame match each color. This count helps in identifying the
presence and quantity of different colors in real-time.
● Output and Visual Feedback:

The processed video stream is displayed with annotated text showing the color
counts. Python uses cv2.putText() to overlay the color names and their respective
counts onto the video frame. This visual feedback allows users to verify the accuracy
of the color detection and observe the results in real-time.

● Integration with the Color Sorting Mechanism:

Python's real-time processing capabilities enable seamless integration with any

sorting mechanism, such as robotic arms or conveyor belts. The color detection
results can be fed to a sorting system, which can then sort objects based on their
detected color, showcasing the potential for industrial applications.
11
Web : www.pragyatmika.co.in Email : [email protected]
Together, Arduino C and Python create a seamless and efficient workflow for implementing
a Color Sorting Robot that utilizes the ESP32 CAM's capabilities for real-time video
streaming and color detection.

5. LIBRARIES USED

The Color Sorting Robot project utilizes a variety of libraries in both the Arduino C and
Python programming environments. These libraries play critical roles in enabling
functionalities such as connecting to Wi-Fi, streaming video, capturing, and processing
images, and performing color detection. Below is a detailed overview of the libraries used in
the project:

5.1. Arduino Libraries:

● WiFi.h

Purpose: The WiFi library is essential for enabling the ESP32 CAM to connect to a Wi-
Fi network. It provides the necessary functions to manage Wi-Fi connections,
including setting up the Wi-Fi credentials (SSID and password), connecting to the
network, and obtaining an IP address.

● esp_camera.h

Purpose: The esp_camera.h library provides functions and definitions specific to the
ESP32 CAM module. It handles the initialization of the camera hardware,
configuration settings, and capturing images. This library is crucial for setting up the
camera parameters and starting the video stream.

5.2. Python Libraries:

● Requests Command Set:

Command: pip install requests

12
Web : www.pragyatmika.co.in Email : [email protected]
 Description: The requests library is a simple and elegant HTTP library for Python. It
provides a simple and intuitive way to interact with web services and APIs, enabling
developers to send HTTP requests to servers and receive responses. The library is
known for its ease of use, readability, and reliability.

● opencv-python:

Command: pip install opencv-python

Description: OpenCV (Open Source Computer Vision Library) is a powerful library for
computer vision tasks. It provides tools to capture, process, and analyze images and
video streams. In this project, OpenCV is used to access the video stream from the
ESP32 CAM, perform image processing (e.g., grayscale conversion, blurring, edge

● NumPy Library
Command: pip install numpy

Purpose: NumPy is a fundamental package for scientific computing in Python. It

provides support for large multi-dimensional arrays and matrices, along with a
collection of mathematical functions to operate on these arrays. In this project,
NumPy is used indirectly through OpenCV for handling array operations required for
image processing tasks.

6. SOFTWARE LOGIC

This project involves two main components: generating a dynamic IP address using the
Arduino IDE for the ESP32-CAM module and using Python to detect shapes through the
video stream provided by the ESP32-CAM.

6.1. Generating a Dynamic IP Address using Arduino IDE

13
Web : www.pragyatmika.co.in Email : [email protected]
To allow the ESP32-CAM module to stream video over a network, it first needs to connect to
a Wi-Fi network and be assigned a dynamic IP address. This IP address will be used by the
Python script to access the video feed.

Steps to Generate Dynamic IP Address:

1. Setting Up the Arduino IDE for ESP32-CAM:

○ Install the ESP32 board package in the Arduino IDE. You can do this by adding
the URL https://dl.espressif.com/dl/package_esp32_index.json to the
Arduino IDE’s Board Manager under Preferences and installing the ESP32
board.
○ Select the correct board and port in the Arduino IDE: choose the "ESP32
Wrover Module" or a similar ESP32-CAM option from the board list.
2. Writing the Code to Connect to Wi-Fi:
○ Use the WiFi.h library to manage Wi-Fi connections.
○ Define your network credentials (SSID and password) in the sketch.
○ Use WiFi.begin(ssid, password); to initiate the connection to the Wi-Fi
network.
○ Use WiFi.localIP(); to fetch and print the dynamic IP address assigned to the
ESP32-CAM by the router.
3. Deploying the Code:
○ Connect the ESP32-CAM to your computer using a USB-to-Serial adapter and
upload the code using the Arduino IDE.
○ Once uploaded, open the Serial Monitor to view the dynamic IP address
assigned by the router. This IP will be used to access the video stream.
4. Streaming Video:
○ With the IP address assigned, the ESP32-CAM can start streaming video using
the ESP32 camera library.
○ Use the URL http://<dynamic_ip>:81/stream in your Python code to access
the live video feed.

14
Web : www.pragyatmika.co.in Email : [email protected]
6.2.Color Detection in Python using OpenCV

The core functionality of the Color Sorting Robot involves detecting specific colors from a
video stream. Python, with the OpenCV library (`cv2`), is used for processing video frames
and identifying colors in real time.

Steps for Color Detection in Python:

1. Initialize the Video Capture:

● Use OpenCV’s `VideoCapture` function to access the video stream from a webcam or
video capture device.
● This step ensures that Python can read video frames continuously for color
detection.

2. Capture Frames and Preprocess:

● Read each frame from the video stream using the `read()` method.
● Convert each frame from BGR (Blue, Green, Red) color space to HSV (Hue,
Saturation, Value) using `cv2.cvtColor()`. HSV is more suitable for color detection as
it separates color information from intensity.

3. Apply Color Masks:

● Define HSV color ranges for the colors of interest (e.g., red, green, blue, etc.).
● Use `cv2.inRange()` to create binary masks for each color. Each mask highlights the
pixels in the frame that fall within the specified HSV range.

4. Count the Color Pixels:

● For each color mask, count the number of non-zero pixels using
`cv2.countNonZero()`. This count indicates how many pixels in the current frame
match the specified color.
● Store the count in a dictionary to track the number of detected pixels for each color.

15
Web : www.pragyatmika.co.in Email : [email protected]
5. Display the Video with Detected Colors:

● Overlay text on the video frames to display the count of each detected color using
`cv2.putText()`. This provides real-time feedback on the color detection process.
● Use `cv2.imshow()` to display the video stream with annotated text showing the
detected color counts.

6. User Interaction and Loop Termination:

● Allow users to terminate the loop and close the video window by pressing the 'q'
key. This is implemented using the `cv2.waitKey()` function, which waits for a key
press event.

7. Release Resources:

● After the color detection is complete, release the video capture object using
`cap.release()` to free up resources.
● Close all OpenCV windows using `cv2.destroyAllWindows()`.

This approach allows the Color Sorting Robot to efficiently detect and count specific colors
in real time using Python and OpenCV. By following these steps, the robot can be integrated
into various applications requiring automated color sorting, demonstrating the power of
Python for practical computer vision tasks.

7. PROGRAM CODE

7.1 Arduino code:

16
Web : www.pragyatmika.co.in Email : [email protected]
#include "esp_camera.h"

#include <WiFi.h>

//

// WARNING!!! PSRAM IC required for UXGA resolution and high JPEG quality

// Ensure ESP32 Wrover Module or other board with PSRAM is selected

// Partial images will be transmitted if image exceeds buffer size

//

// You must select partition scheme from the board menu that has at least 3MB APP
space.

// Face Recognition is DISABLED for ESP32 and ESP32-S2, because it takes up from
15

// seconds to process single frame. Face Detection is ENABLED if PSRAM is enabled

as well

// ===================

// Select camera model

// ===================

//#define CAMERA_MODEL_WROVER_KIT // Has PSRAM

//#define CAMERA_MODEL_ESP_EYE // Has PSRAM

//#define CAMERA_MODEL_ESP32S3_EYE // Has PSRAM

//#define CAMERA_MODEL_M5STACK_PSRAM // Has PSRAM

17
Web : www.pragyatmika.co.in Email : [email protected]
//#define CAMERA_MODEL_M5STACK_V2_PSRAM // M5Camera version B Has PSRAM

//#define CAMERA_MODEL_M5STACK_WIDE // Has PSRAM

//#define CAMERA_MODEL_M5STACK_ESP32CAM // No PSRAM

//#define CAMERA_MODEL_M5STACK_UNITCAM // No PSRAM

//#define CAMERA_MODEL_M5STACK_CAMS3_UNIT // Has PSRAM

#define CAMERA_MODEL_AI_THINKER // Has PSRAM

//#define CAMERA_MODEL_TTGO_T_JOURNAL // No PSRAM

//#define CAMERA_MODEL_XIAO_ESP32S3 // Has PSRAM

// ** Espressif Internal Boards **

//#define CAMERA_MODEL_ESP32_CAM_BOARD

//#define CAMERA_MODEL_ESP32S2_CAM_BOARD

//#define CAMERA_MODEL_ESP32S3_CAM_LCD

//#define CAMERA_MODEL_DFRobot_FireBeetle2_ESP32S3 // Has PSRAM

//#define CAMERA_MODEL_DFRobot_Romeo_ESP32S3 // Has PSRAM

#include "camera_pins.h"

// ===========================

// Enter your WiFi credentials

// ===========================

const char *ssid = "PRAGYATMIKACOE";

const char *password = "Mcet@prag220124";

18
Web : www.pragyatmika.co.in Email : [email protected]
void startCameraServer();

void setupLedFlash(int pin);

void setup() {

Serial.begin(115200);

Serial.setDebugOutput(true);

Serial.println();

camera_config_t config;

config.ledc_channel = LEDC_CHANNEL_0;

config.ledc_timer = LEDC_TIMER_0;

config.pin_d0 = Y2_GPIO_NUM;

config.pin_d1 = Y3_GPIO_NUM;

config.pin_d2 = Y4_GPIO_NUM;

config.pin_d3 = Y5_GPIO_NUM;

config.pin_d4 = Y6_GPIO_NUM;

config.pin_d5 = Y7_GPIO_NUM;

config.pin_d6 = Y8_GPIO_NUM;

config.pin_d7 = Y9_GPIO_NUM;

config.pin_xclk = XCLK_GPIO_NUM;
19
Web : www.pragyatmika.co.in Email : [email protected]
config.pin_pclk = PCLK_GPIO_NUM;

config.pin_vsync = VSYNC_GPIO_NUM;

config.pin_href = HREF_GPIO_NUM;

config.pin_sccb_sda = SIOD_GPIO_NUM;

config.pin_sccb_scl = SIOC_GPIO_NUM;

config.pin_pwdn = PWDN_GPIO_NUM;

config.pin_reset = RESET_GPIO_NUM;

config.xclk_freq_hz = 20000000;

config.frame_size = FRAMESIZE_UXGA;

config.pixel_format = PIXFORMAT_JPEG; // for streaming

//config.pixel_format = PIXFORMAT_RGB565; // for face detection/recognition

config.grab_mode = CAMERA_GRAB_WHEN_EMPTY;

config.fb_location = CAMERA_FB_IN_PSRAM;

config.jpeg_quality = 12;

config.fb_count = 1;

// if PSRAM IC present, init with UXGA resolution and higher JPEG quality

// for larger pre-allocated frame buffer.

if (config.pixel_format == PIXFORMAT_JPEG) {

if (psramFound()) {

config.jpeg_quality = 10;
20
Web : www.pragyatmika.co.in Email : [email protected]
 config.fb_count = 2;

config.grab_mode = CAMERA_GRAB_LATEST;

} else {

// Limit the frame size when PSRAM is not available

config.frame_size = FRAMESIZE_SVGA;

config.fb_location = CAMERA_FB_IN_DRAM;

} else {

// Best option for face detection/recognition

config.frame_size = FRAMESIZE_240X240;

#if CONFIG_IDF_TARGET_ESP32S3

config.fb_count = 2;

#endif

#if defined(CAMERA_MODEL_ESP_EYE)

pinMode(13, INPUT_PULLUP);

pinMode(14, INPUT_PULLUP);

#endif

// camera init
21
Web : www.pragyatmika.co.in Email : [email protected]
esp_err_t err = esp_camera_init(&config);

if (err != ESP_OK) {

Serial.printf("Camera init failed with error 0x%x", err);

return;

sensor_t *s = esp_camera_sensor_get();

// initial sensors are flipped vertically and colors are a bit saturated

if (s->id.PID == OV3660_PID) {

s->set_vflip(s, 1); // flip it back

s->set_brightness(s, 1); // up the brightness just a bit

s->set_saturation(s, -2); // lower the saturation

// drop down frame size for higher initial frame rate

if (config.pixel_format == PIXFORMAT_JPEG) {

s->set_framesize(s, FRAMESIZE_QVGA);

#if defined(CAMERA_MODEL_M5STACK_WIDE) ||
defined(CAMERA_MODEL_M5STACK_ESP32CAM)

s->set_vflip(s, 1);

22
Web : www.pragyatmika.co.in Email : [email protected]
 s->set_hmirror(s, 1);

#endif

#if defined(CAMERA_MODEL_ESP32S3_EYE)

s->set_vflip(s, 1);

#endif

// Setup LED FLash if LED pin is defined in camera_pins.h

#if defined(LED_GPIO_NUM)

setupLedFlash(LED_GPIO_NUM);

#endif

WiFi.begin(ssid, password);

WiFi.setSleep(false);

while (WiFi.status() != WL_CONNECTED) {

delay(500);

Serial.print(".");

Serial.println("");

Serial.println("WiFi connected");
23
Web : www.pragyatmika.co.in Email : [email protected]
 startCameraServer();

Serial.print("Camera Ready! Use 'http://");

Serial.print(WiFi.localIP());

Serial.println("' to connect");

void loop() {

// Do nothing. Everything is done in another task by the web server

delay(10000);

7.2 Python code:

import cv2

import numpy as np

import time

Import requests

# ESP32 URL

URL = "http://192.168.0.106"

AWB = True
24
Web : www.pragyatmika.co.in Email : [email protected]
# Define the color ranges in HSV

color_ranges = {

'red': ([0, 120, 70], [10, 255, 255]),

'red2': ([170, 120, 70], [180, 255, 255]), # Second range for red

'green': ([36, 50, 70], [89, 255, 255]),

'blue': ([94, 80, 2], [126, 255, 255]),

'orange': ([5, 50, 50], [15, 255, 255]),

'black': ([0, 0, 0], [180, 255, 30]),

'pink': ([160, 50, 50], [170, 255, 255]),

'yellow': ([22, 93, 0], [45, 255, 255]),

'purple': ([129, 50, 70], [158, 255, 255]),

'brown': ([10, 100, 20], [20, 255, 200]),

def detect_colors(frame):

# Convert image to HSV color space

hsv_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

# Dictionary to store the color counts

25
Web : www.pragyatmika.co.in Email : [email protected]
 color_count = {

'red': 0,

'green': 0,

'blue': 0,

'orange': 0,

'black': 0,

'pink': 0,

'yellow': 0,

'purple': 0,

'brown': 0,

# Iterate through each color and count the occurrences

for color, (lower, upper) in color_ranges.items():

lower_bound = np.array(lower, dtype="uint8")

upper_bound = np.array(upper, dtype="uint8")

# Create a mask for the color

mask = cv2.inRange(hsv_frame, lower_bound, upper_bound)

# Count the non-zero pixels in the mask

26
Web : www.pragyatmika.co.in Email : [email protected]
 count = cv2.countNonZero(mask)

# Add count to the respective color in the dictionary

if color == 'red':

color_count['red'] += count # Combine both red ranges

else:

color_count[color] = count

return color_count

def main():

# Open the webcam (or any video capture device)

cap = cv2.VideoCapture((URL + ":81/stream")

if not cap.isOpened():

print("Error: Could not open video capture device.")

return

# Initialize the timer

last_update_time = time.time()

27
Web : www.pragyatmika.co.in Email : [email protected]
 # Initialize the cumulative color counts

cumulative_color_count = {

'red': 0,

'green': 0,

'blue': 0,

'orange': 0,

'black': 0,

'pink': 0,

'yellow': 0,

'purple': 0,

'brown': 0,

while True:

# Capture frame-by-frame

ret, frame = cap.read()

if not ret:

print("Error: Could not read frame.")

break

28
Web : www.pragyatmika.co.in Email : [email protected]
 # Detect colors in the current frame

color_count = detect_colors(frame)

# Check if 5 seconds have passed since the last update

current_time = time.time()

if current_time - last_update_time >= 5:

# Update the cumulative color count

for color in cumulative_color_count:

if color_count[color] > 0: # Only increment if color is detected

cumulative_color_count[color] += 1

# Reset the last update time

last_update_time = current_time

# Display the cumulative color count on the frame

y_position = 30

for color, count in cumulative_color_count.items():

cv2.putText(frame, f"{color}: {count}", (10, y_position),

cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 0), 2)

y_position += 30

# Display the resulting frame

29
Web : www.pragyatmika.co.in Email : [email protected]
 cv2.imshow('Color Detection', frame)

# Break the loop on 'q' key press

if cv2.waitKey(1) & 0xFF == ord('q'):

break

# When everything done, release the capture

cap.release()

cv2.destroyAllWindows()

if __name__ == "__main__":

main()

8. TEST STEPS

These test steps ensure that the color sorting robot functions correctly, detecting and
counting various colors in real-time from the video stream. By following this testing
procedure, you can validate that the system meets its requirements and can be reliably used
for color sorting applications.

Getting started with esp32 cam using arduino IDE

Step1: To use the ESP32-CAM, or any ESP32, with the Arduino IDE, you must first install the
ESP32 board (also known as the ESP32 Arduino Core) via the Arduino Board Manager.
30
Web : www.pragyatmika.co.in Email : [email protected]
Step 2: Go to the files menu and then select examples. From the example’s menu, select the
ESP32 and then Camera. Then finally open the CameraWebServer Sketch.

Step 3 : You need to modify this code a little bit. Select the Camera Module by
uncommenting the model that you are using. If you are using AI-Thinker Board uncomment
the AI-Thinker Part and comment rest of the part.

31
Web : www.pragyatmika.co.in Email : [email protected]
Step 4 : change your WiFi SSID and Password.

Step 5: Now connect the ESP32-CAM to the Base Board and connect it to your computer
using a USB cable.

Step 6: navigate to Tools > Board > esp32 and select AI-Thinker ESP32-CAM also select the
COM port to which your esp32 cam is connected.

32
Web : www.pragyatmika.co.in Email : [email protected]
Step 7: Then, navigate to Tools > Port and choose the COM port to which the ESP32-CAM is
connected.

Step 8: upload the code. The code will take more than a minute to compile and upload.

Step 9: Once you have uploaded the sketch, open the serial monitor at baud rate 115200

and press the Reset button on the ESP32-CAM. You should see the IP address in the Serial

Monitor.

33
Web : www.pragyatmika.co.in Email : [email protected]
Step 10 : Launch a browser and enter the IP address shown on the serial monitor. Ensure
that the web browser is on the same network that the ESP32-CAM is connected to.

The ESP32-CAM should display a web page. To begin video streaming, press the Start
Stream button.

34
Web : www.pragyatmika.co.in Email : [email protected]
At the bottom you can see there is an option to start streaming, you can click con Start

Streaming and see yourself in the video. Maximize the resolution using the Camera Option.

You can move the camera in any direction to see yourself in the video. The Video data is live

and you can observe it on the local WebServer always.

Step 11: Copy the IP address since we will be using this IP address to integrate our esp32

cam to our python code instead of webcam.

35
Web : www.pragyatmika.co.in Email : [email protected]
Now our esp32 Cam is ready to interface with python code

Steps for Setting Up and Testing the Color Sorting Robot:

Step 1: Open Terminal and Install Required Libraries:

Use the following commands to install the necessary Python libraries:

pip install opencv-python

pip install requests

pip install numpy

Step 2: Open Visual Studio Code and Install Python Extension:

Before starting to work with Python in Visual Studio Code, install the Python extension. This

can be done through the Extensions view (Ctrl+Shift+X), then search for and install the

Python extension provided by Microsoft.

Step 3: Create a New Python File:

Navigate to File -> New File and choose Python File. Name your file (e.g.,

color_sorting_robot.py).

Step 4: Copy and paste the color detection code into your newly created Python file.

36
Web : www.pragyatmika.co.in Email : [email protected]
Make sure to adjust the URL variable's value with the dynamic IP address obtained from the

ESP32-CAM in the Arduino IDE.

Step 5: Run the Python Script:In the Terminal, navigate to the directory where your Python

script is saved using the cd command. For example:

cd path/to/your/script

Step 6: Ensure your ESP32-CAM is connected to the same network as your system.

Execute the Python script using the following command:

python color_sorting_robot.py

Step 7: Verify Video Feed and Check for Errors. Confirm that a window titled “Color

Detection” appears, displaying the live video feed from the ESP32-CAM. Monitor the

terminal for any error messages related to video capture or network issues.

Step 8: Place multiple-coloured objects in front of the camera to check whether the

detection is accurate. Observe the color counts displayed on the output screen to verify if

the detection and increment are functioning correctly. While Detecting the color, make sure

that the background of object should be plain. There should be proper light in the test area

and no bright reflecting focus should be there on object.

37
Web : www.pragyatmika.co.in Email : [email protected]
These steps should guide you through setting up, running, and testing the color detection

functionality effectively. Make sure to handle different lighting conditions and recalibrate if

necessary for optimal accuracy.

OUTPUT SCREENS

Conclusion:

The color detection and sorting robot can detect the presence of colors available in a
multicolour object and count and display it on screen. We can mount this setup on a
wheeled robotic platform and it can traverse the area for color detection and sorting.

38
Web : www.pragyatmika.co.in Email : [email protected]