Paper ID #35947

Object Detection on Raspberry Pi

Prof. Xishuang Dong, Prairie View A&M University
Xishuang Dong is Assistant Professor of Electrical and Computer Engineering Department, Roy G. Perry
College of Engineering, Prairie View A&M University. His research interests include deep learning,
object detection, natural language processing, computer systems biology, and Internet of Things.
Xavier Alexander Dukes
Mr. Joshua Littleton, Prairie View A&M University
Tri’Heem Neville
Christopher Rollerson
Arthur L Quinney

c American Society for Engineering Education, 2022

 Session XXXX

Object Detection on Raspberry Pi

Joshua Littleton, Xavier Dukes, Arthur Quinney, Tri’Heem Neville, Christopher Rollerson,
Xishuang Dong
Electrical and Computer Engineering Department
Prairie View A&M University

Abstract

Internet of Things (IoT) refers to the billions of edge devices around the world connected to the
internet for various applications such as collecting and analyzing data. Data analytics on these edge
devices can protect user privacy and reduce communication costs. Specifically, computer vision is a
core technique to build such applications on these devices, where object detection is an imperative
task that is to recognize categories of objects and label their locations. This paper presented a senior
design project that implemented object detection on Raspberry Pi via machine learning models. It is
to employ Raspberry Pi Kits and a web-camera to detect predefined objects by running mobile deep
learning models. Moreover, it was extended to recognize weapons in images with transfer learning
techniques, which is to fine-tune the models on new annotated datasets collected from Internet.
Specifically, it applied a Google USB accelerator to improve the detection speed. Preliminary
validation results demonstrate that the models can effectively detect general objects such as person,
keyboard, and laptop, as well as predefined weapons on Raspberry Pi.

Introduction

The Internet of Things (IoT) refers to the billions of edge devices around the world connected to the
internet for collecting and analyzing data, which is evolving rapidly and widening existing
communication systems. It was predicted that more than 50 billion IoT devices will be involved in
IoT1 after 2020 and these devices will be linked with each another to exchange and generate data to
make everyday life easier in different aspects such as healthcare2, military3, smart home4, and
agriculture5 by developing various applications. Specifically, computer vision is a core technique to
build these applications, where object detection is an imperative task in this field that is to recognize
categories of objects and label their locations.
This paper presents a senior design project that implemented object detection on Raspberry Pi by
running deep learning models, where the edge devices include Raspberry Pi 3 and 4, Model B+ (Plus)
Complete Starter Kit1 and a web camera. It consists of three steps: 1) hardware configuration: it is to
configure the Raspberry Pi and mount the web-camera on the Raspberry Pi; 2) software installation:
it is to install necessary software such as TensorFlow and OpenCV; 3) deploying mobile deep learning
models on Raspberry Pi to run object detection. Experimental results demonstrate the effectiveness of
this implementation. Moreover, we leverage the USB accelerator to improve the detection speed.

1
https://www.canakit.com/raspberry-pi-3-model-b-plus-starter-kit.html
Proceedings of the 2022 ASEE Gulf-Southwest Annual Conference
Prairie View A&M University, Prairie View, TX
Copyright ã 2022, American Society for Engineering Education
Finally, we extend this implementation to weapon detection through fine-tuning pretrained deep
learning models on data samples collected and annotated manually.

Project
This project implemented object detection on Raspberry Pi. As a core task in the field of computer
vision, object detection is to detect instances of visual objects with a certain class (such as airplane,
car, or animal) in digital images. It is implemented by developing computational models and
techniques to recognize objects and their locations, which can be applied to many downstream
computer vision tasks. Recent years have witnessed the breakthrough of object detection with the
emergence of deep neural networks (DNNs)6, especially, convolutional neural network (CNN)7.
Compared to traditional techniques, the CNN can extract more specific and advanced features with
deeper architectures layer by layer. Moreover, the expressivity and robust training algorithms allow
CNN to learn informative object representations without the need to design features manually8.
However, these models need
to run with many Input Images Object Class and
computational resources, Web camera Mobile Deep
Location

which cannot be afforded by Edge Device Learning Model

edge devices for IoT. To Figure 1. Object detection on an edge device with a mobile deep learning model
overcome this challenge,
mobile deep learning models have been implemented on these devices. With this advanced technique,
this project is implemented as Figure 1. Web camera took pictures as input images to the mobile deep
learning models that run on an edge device. Once completing detection, it will output object classes
and corresponding locations on the
image.
Moreover, this project is extended
to detect weapons. It is observed that
a large amount of population
reconciles violence related to various
weapons such as guns and knives all
over the world. Therefore, it will be a
good proactive solution to detect
these weapons in public area to
protect citizens. In this work, we Figure 2. Flow of weapon detection
implemented weapon detection on
images, which is designed as Figure 2. Camera will help obtain the input data that would be video
streams. Then, it will process the video and isolate images with image acquisition. Afterwards, we
will apply mobile deep learning model to detect weapons with image recognition, where the output is
to confirm if there is a weapon. If there is a weapon, it will send a notification. Otherwise, it will
process the next image.

Implementation and Preliminary Validation

Implementation:
We employed specific a deep learning model and an edge device to implement this project. For deep
learning model, this paper employed SSD-MobileNet13 to implement object detection on edge devices.
Proceedings of the 2022 ASEE Gulf-Southwest Annual Conference
Prairie View A&M University, Prairie View, TX
Copyright ã 2022, American Society for Engineering Education
SSD-MobileNet is a single-shot detector with high performance for multiple categories of objects,
where the architecture contains MobileNet9. To improve detection accuracy, it generates predictions
of different scales from feature maps with different scales, and explicitly separates the predictions by
aspect ratios. For edge device, Raspberry Pi was selected as the edge device to run SSD-MobileNet
for object detection. It is designed as credit-card-sized single board computer for stimulating the
teaching of basic computer science in schools. It supports Debian and Arch Linux ARM distributions
and Python as the main programming language. In addition, NumPy, SciPy, Matplotlib, IPython, and
PyLab can be used for computational mathematics as well as the analysis of experimental data or
control systems10. It is widely used in many areas, such as for weather monitoring11 because of its low
cost, modularity, and open design. In addition, it is used by computer and electronic hobbyists, due to
its adoption of HDMI and USB devices.
For weapon detection,
implementation design is
shown in Figure 3. In
addition, we build a data
set for weapon detection,
where Figure 4 shows four
examples of knives held by
a person, where the white
box is the ground truth.
Although we collected 900
samples, the number of
samples is not big enough
to train the SSD-MobileNet
from scratch. Therefore,
we employed deep transfer Figure 3. Implementation design of weapon detection
12
learning to resolve this
challenge. It tries to transfer the knowledge from the source domain to the target domain by relaxing
the assumption that the training data and the test data must be independent and identically
distributed (i.i.i.d). It is to fine-tune SSD-MobileNet on the data collected for weapon detection.

Figure 4. Examples of data samples for weapon detection

Proceedings of the 2022 ASEE Gulf-Southwest Annual Conference
Prairie View A&M University, Prairie View, TX
Copyright ã 2022, American Society for Engineering Education
Preliminary Validation:

Figure 5. Examples of results of object detection on Raspberry Pi

For object detection on Raspberry Pi, we ran SSD-MobileNet on Raspberry Pi 3 to implement object
detection. In addition, we employed accelerator to speed up the detection. Compared to the detection
without this accelerator, the detection speed can be improved from 1.13 frame per second from to
10.51 frame per second on averages. Figure 5 presents three examples of detection results. The left
example indicates that the implementation can detect multiple objects including a person and a phone
even if the person was wearing face mask. In addition, the locations of these two objects are accurate
in terms of bounding boxes. The middle example shows that the implementation can detect the object
with high confidence (96%). The right example presents a more complex case that the locations of
objects have overlaps. For instance, although the phone’s location has overlaps with that of the person,
the implementation can still detect these objects correctly.
For weapon detection on Raspberry Pi, we ran fine-tuned SSD-
MobileNet on Raspberry Pi 4, where Figure 6 presents one example
of detection results. Currently, we utilized the camera on laptop for
this validation. In terms of the preliminary detection results, current
version of implement is able to detect certain weapons such as
handgun. However, the detection speed is not high enough for real-
world applications. In addition, it is still limited with predefined Figure 6. An example of detecting
weapons. weapons

Conclusion

This paper presents preliminary work of the implementation of object detection on Raspberry Pi for a
senior design project. We run a mobile deep learning model, SSD-MobileNet, on Raspberry Pi to
detect various objects. Preliminary valudation results demonstrate the effectiveness of this
implementation. Moreover, the on-going work is to improve weapon detection. Future work will focus
on completing comprehensive and systematical validation on weapon detection in different testing
scenarios.

References
1. Dave, E. (2011). How the next evolution of the internet is changing everything. The Internet of Things.
2. Islam, S. R., Kwak, D., Kabir, M. H., Hossain, M., and Kwak, K. S. (2015). The internet of things for health care: a
comprehensive survey. IEEE access, 3, 678-708.
3. Cohen, A. E., Jiang, G. G., Heide, D. A., Pellegrini, V., and Suri, N. (2018, October). Radio frequency IoT sensors in
Proceedings of the 2022 ASEE Gulf-Southwest Annual Conference
Prairie View A&M University, Prairie View, TX
Copyright ã 2022, American Society for Engineering Education
 military operations in a smart city. In MILCOM 2018-2018 IEEE Military Communications Conference
(MILCOM) (pp. 763-767). IEEE.
4. Salman, L., Salman, S., Jahangirian, S., Abraham, M., German, F., Blair, C., and Krenz, P. (2016, December). Energy
efficient IoT-based smart home. In 2016 IEEE 3rd World Forum on Internet of Things (WF-IoT) (pp. 526-529). IEEE.
5. Kjellby, R. A., Cenkeramaddi, L. R., Frøytlog, A., Lozano, B. B., Soumya, J., and Bhange, M. (2019, April). Long-
range & self-powered IoT devices for agriculture & aquaponics based on multi-hop topology. In 2019 IEEE 5th World
Forum on Internet of Things (WF-IoT) (pp. 545-549). IEEE.
6. Kang, K., Li, H., Yan, J., Zeng, X., Yang, B., Xiao, T., and Ouyang, W. (2017). T-cnn: Tubelets with convolutional
neural networks for object detection from videos. IEEE Transactions on Circuits and Systems for Video
Technology, 28(10), 2896-2907.
7. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014). Rich feature hierarchies for accurate object detection and
semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 580-
587).
8. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep learning. MIT press.
9. Ali, 2013 M.,Vlaskamp J.H.A,Eddiny N.N.,Falconer B. and Oram c., Technical Development and Socioeconomic
Implications of the Raspberry Pi as a Learning Tool in Developing Countries,5th Computer Science and Electronic
Engineering Conference (CEEC), pp. 103- 108, 2013.
10. Arabi, Saeed, Arya Haghighat, and Anuj Sharma. A deep learning based solution for construction equipment
detection: from development to deployment. arXiv preprint arXiv:1904.09021 (2019).
11. Upton, L. (2013). The Raspberry Pi in scientific research.
12. Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., and Liu, C. (2018, October). A survey on deep transfer learning.
In International conference on artificial neural networks (pp. 270-279). Springer, Cham.
13. COCO SSD MobileNet: https://www.tensorflow.org/lite/examples/object_detection/overview

Joshua Littleton, Xavier Dukes, Arthur Quinney, Tri’Heem Neville, and Christopher Rollerson are
senior-level undergraduate students of Electrical and Computer Engineering Department, Roy G.
Perry College of Engineering, Prairie View A&M University.

XISHUANG DONG
Dr. Xishuang Dong currently serves as Assistant Professor of Electrical and Computer Engineering
Department, Roy G. Perry College of Engineering, Prairie View A&M University. His research
interests include deep learning, object detection, natural language processing, computer systems
biology, and Internet of Things.

Proceedings of the 2022 ASEE Gulf-Southwest Annual Conference

Prairie View A&M University, Prairie View, TX
Copyright ã 2022, American Society for Engineering Education