8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd

| Random Nerd Tutorials

 Menu 

ESP32-CAM Robotics with OpenCV: Autonomous

and Teleop Operation with XBOX Controller

In this article, we want to share a work developed by one of our readers that shows
how to use the ESP32/ESP32-CAM streaming servers in robotic systems using
OpenCV or OpenCV.js.

This project/tutorial was created by Andrew R. Sass (one of our readers/followers).

This post is just an introduction to his work (Andrew Sass). The complete work is
compiled in a PDF paper with full descriptions and explanations of the code. You can
also have access to all the codes used throughout the paper. You can access and
download all his work on the GitHub page:

PDF: Introduction to ESP32 Robotics Systems

Access all code on Github:
Browser Client System
https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 1/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

Python Client System

Andrew Sass has already written a tutorial about OpenCV.js color detection and
tracking on an ESP32-CAM video streaming server: ESP32-CAM Web Server with
OpenCV.js: Color Detection and Tracking.

Introduction
The ESP32-CAM can act as a video streaming server for different clients. In his work,
he covers the ESP32-CAM acting as a video streaming server for a browser client
and for a Python client window. Then, he shows how to process the video using
OpenCV.js or OpenCV to get output for robotics autonomous and manual operation.
Additionally, he also created a user interface/control using an XBOX controller.

ESP32-CAM Video Streaming Server – Browser Client

When the ESP32-CAM acts as a video streaming server for a browser client, we can
use OpenCV.js on the browser to process the image, and send the results back to the
server.

ESP32-CAM Video Streaming Server – Python Client

On the other hand, when the ESP32-CAM acts as a video streaming server for a
Python client, we have access to a more extensive library of OpenCV software than
is currently available in OpenCV.js, because OpenCV is resident in the users’ laptop
as opposed to the ESP32.

However, unlike the Browser Streaming System, data communication back to the
server is not inherently available in the Python Streaming System. Since
communication back to the server is essential for any complete robotic system, a
novel method for this is presented in this work. It is, admittedly more complex, yet
hopefully balanced by the advantages of the above-mentioned Python OpenCV
library for those users not bound to a Browser application.

General Description of the Systems

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 2/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

Following is a brief introductory pictorial description of the Browser Streaming System

operation.

Browser Streaming System

The Browser Server publishes its IP address and is accessed by using a web
browser. After pressing the Color Track button on the Chrome screen, the Client
video screen is seen below.

The red screen patch above shows sample sensor data transmitted by the ESP32
Server to the Client.

The XBOX Controller transmits Joystick, Button, Trigger, and Hat data to the ESP32.
The ESP32 is the front end of the Robot and sends/receives data to the Robot’s main
processor. This mode, in which a human operator controls the robot, is called the
Teleop Mode.

When the list box selector below the left side of the video is set to NO, the view in the
following picture is seen.

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 3/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

This is the Autonomous Mode in which the robot is under program control. In this
mode, OpenCV (computer vision) is used to guide the robot. The top left view is the
author wearing a red hat. The bottom right view shows a white dot approximately in
the lower center screen. This dot is moveable and probes the position of the screen
where the program can measure the RGB content of the picture. In this example, the
white dot probe was initially moved to the red hat where the measurement was made
and then moved back to its initial position.

The RGB sliders were set to the red hat RGB measurement. The top right view
shows the binary image corresponding to the color range set by the sliders. The hat
is the largest portion of the image corresponding to the sliders’ ranges. When the
Tracking button is pressed, a blue contour rectangle surrounds the biggest single
portion of white, identifying it as the target. The coordinates of the centroid of the blue
rectangle are shown on the screen and transmitted to the ESP32 and then to the
main processor of the robot for Autonomous guidance (see picture below).

A diagram describing the Robot Browser Streaming System Diagram is pictured

below.

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 4/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

For a complete description of the system, please download the complete PDF paper.

Python Streaming System

The ESP32-CAM Streaming Server (described in the paper) publishes its IP address
immediately after download.

The python client program uses this IP address and displays the python window
shown on the left in the picture below.

The XBOX controller program, (also described in the paper) also uses this IP address
in port 82/ws to display the Graphical User Interface(GUI) shown at the right.

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 5/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

The white stripe displaying data in the left window of the above figure and the white
data at the bottom of the GUI are data exchanged by the two programs. In other
words, the XBOX Controller can control the program execution of the Client and the
Client data can be used by the XBOX Controller and can be transmitted by the
Controller to the ESP32 Server.

In the above figure, the XBOX Controller commands the Client program to identify the
RED object. The Client program then transmits the centroid coordinated of the RED
object to the XBOX Controller. If the human operator sets the Controller to
Autonomous Mode, the centroid coordinates are transmitted to the Server and then
to the main processor of the robot for guidance purposes.

The Python Streaming System Diagram is pictured below.

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 6/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

For a complete description of the system, please download the complete PDF paper.

Wrapping Up
This was just a quick description of the systems developed and described by Andrew
Sass in his paper. You can access all his work on the following links:

PDF: Introduction to ESP32 Robotics Systems

Access all code on Github:
Browser Client System
Python Client System

All questions should be posted on his Github page (ESP32 Robot SystemsTutorial
repository) so you can have the chance to get your question answered by Andrew.

He had already written another tutorial about color detection and tracking using
OpenCV.js that you can check on the following link:

ESP32-CAM Web Server with OpenCV.js: Color Detection and Tracking

We hope you found this project useful.

Thanks for reading.

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 7/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

SMART HOME with

Raspberry Pi, ESP32,
ESP8266 [eBook]

Learn how to build a home automation system and we’ll cover the following main subjects:
Node-RED, Node-RED Dashboard, Raspberry Pi, ESP32, ESP8266, MQTT, and InfluxDB
database DOWNLOAD »

Recommended Resources

Build a Home Automation System from Scratch » With Raspberry Pi, ESP8266,
Arduino, and Node-RED.

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 8/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

Home Automation using ESP8266 eBook and video course » Build IoT and
home automation projects.

Arduino Step-by-Step Projects » Build 25 Arduino projects with our course, even
with no prior experience!

What to Read Next…

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 9/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

ESP8266 NodeMCU: ESP-NOW Web Server Sensor Dashboard (ESP-NOW

+ Wi-Fi)

ESP8266 NodeMCU WebSocket Server: Control Outputs (Arduino IDE)

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 10/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

ESP-NOW Two-Way Communication Between ESP32 Boards

Enjoyed this project? Stay updated by subscribing our

newsletter!

Your Email Address

 SUBSCRIBE

3 thoughts on “ESP32-CAM Robotics with OpenCV:

Autonomous and Teleop Operation with XBOX
Controller”

Oleg
July 16, 2022 at 3:16 am

Video of the work?

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 11/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

Reply

Mike
July 19, 2022 at 5:52 am

It looks very sophisticated and complex, but also very interesting. As always,
you know how to find the right terms to make this kind of articles very
understandable. I will follow with interest.
Thank you.

Reply

AH
July 19, 2022 at 5:26 pm

Just amazing project that open a lot of possibilities and uses on CV projects.
Thanks for sharing,
AH

Reply

Leave a Comment

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 12/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

Name *

Email *

Website

Notify me of follow-up comments by email.

Notify me of new posts by email.

Post Comment

Affiliate Disclosure: Random Nerd

Tutorials is a participant in affiliate
advertising programs designed to provide a
means for us to earn fees by linking to
Amazon, eBay, AliExpress, and other sites.
We might be compensated for referring
traffic and business to these companies.

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 13/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

Learn ESP32 with Arduino IDE (2nd

Edition) Course » Complete guide to
program the ESP32 with Arduino IDE!

SMART HOME with Raspberry Pi,

ESP32, and ESP8266 » learn how to build
a complete home automation system.

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 14/15
8/3/24, 12:39 AM ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller | Random Nerd Tutorials

🔥 Learn Raspberry Pi Pico/Pico W with

MicroPython​» The complete getting
started guide to get the most out of the the
Raspberry Pi Pico/Pico W (RP2040)
microcontroller board using MicroPython
programming language.

About Support Terms and Conditions Privacy Policy Refunds Complaints’ Book

MakerAdvisor.com Join the Lab

Copyright © 2013-2024 · RandomNerdTutorials.com · All Rights Reserved

https://randomnerdtutorials.com/esp32-cam-robotics-opencv-autonomous/ 15/15