2019 6th International Conference on Electric Vehicular Technology (ICEVT)

November 18-21, 2019, Bali, Indonesia

Real-Time Image Processing Method Using

Raspberry Pi for a Car Model
Mochammad Ariyanto Ismoyo Haryanto Joga Dharma Setiawan
Department of Mechanical Engineering Department of Mechanical Engineering Department of Mechanical Engineering
Diponegoro University Diponegoro University Diponegoro University
Semarang, Indonesia Semarang, Indonesia Semarang, Indonesia
National Center for Sustainable National Center for Sustainable National Center for Sustainable
Transportation Technology, Indonesia Transportation Technology, Indonesia Transportation Technology, Indonesia
[email protected] [email protected] [email protected]

M. Munadi M. Sri Radityo

Department of Mechanical Engineering Department of Mechanical
Diponegoro University Engineering
Semarang, Indonesia Diponegoro University
National Center for Sustainable Transportation Semarang, Indonesia
Technology, Indonesia [email protected]
[email protected]
Abstract—This paper presents the development of a car camera [1]–[8], but some of utilized sensors are RFID [9].
model that can detect edge, line, and corner of the road image The camera is used to acquire image or video. The image
and also the model can detect the red color of a traffic light then is processed using image processing and pattern
image. The car model is equipped with a camera which is used recognition method for detection and pattern recognition.
for computer vision purpose. The image comes from a camera The image processing method is used for lane detection and
is read by using Raspberry Pi single-board computer. The traffic light detection. Ref [6] used color extraction to detect
algorithms for image processing methods are selected to detect boundary lines of the lane under various lighting conditions.
edge, line, corner, and traffic light color of the road model. Y. Wang [7] proposed a Catmull–Rom spline-based lane
The algorithms are developed in Simulink diagram block and
model which describes the perspective effect of parallel lines
embedded into Raspberry Pi using Simulink Support Package
for Raspberry Pi Hardware. The embedded algorithms for
[10].
detecting line, edge, corner and red color of traffic light will be Recently, the research of the lane and traffic detection
tested. The test will be conducted in real-time mode. Based on have grown increasingly. Many of them used Raspberry Pi
the test results, the embedded image processing algorithms can single-board computer for low-cost embedded systems [8],
successfully detect line, edge, and corner of the road images, [11]–[13]. For engineers who do not have knowledge in
and detect the red color of traffic light image. programming especially in python or C language, they can
use Matlab/Simulink for image processing method [14]. A
Keywords—image processing, Raspberry Pi, real-time, car
model
lot of research uses Raspberry Pi for signal processing,
image processing, and in robotic applications.
I. INTRODUCTION In this paper, the car model is equipped with a low-cost
Raspberry Pi single-board computer and a camera. Image
WHO's Global status report on road safety 2015 presents processing algorithm for lane detection and traffic light
information on road safety from 180 countries. Indicates that detection will be developed under MATLAB/Simulink
worldwide the total number of road traffic deaths has environment. The image processing method is aimed to
plateaued at 1.25 million per year. Between 20 and 50 detect line, edge, corner of the road image, and it can detect
million more people suffer non-fatal injuries, with many the red color of traffic light image. In future research, after
incurring a disability as a result of their injury. Road traffic developing the image processing method, it will be
injuries cause considerable economic losses to individuals, implemented in a car model, and it will be controlled using
their families, and to nations as a whole. Road traffic crashes artificial intelligence.
cost most countries 3% of their gross domestic product.
The vehicle safety system should be increased to reduce A. Car Model
the number of road accidents. An autonomous vehicle is a The modified car model is powered by Dynamo N20 DC
technology that has a great role in improving the safety of 6-12V 300 RPM Gear Motor High Torque. This car model
driver. The first step that we used to create autonomous uses one micro servo motor to control the steering angle of
vehicle is image processing and computer vision in order to the car model. This car model employs Arduino Nano to
car model recognize and tracking the white lane markers of control its speed and steering angle by giving the command
the roadway and can detect traffic light intersection, if red to the DC motor and micro servo motor. A selected
color is on, the car must stop. functional car model can be seen in Fig. 1. The car model is
A lot of research in lane detection and recognition use equipped with camera module and Raspberry Pi for real-time
Phyton under OpenCV software. The widely used sensor is a image processing.

978-1-7281-2917-4/19/$31.00 ©2019 IEEE 46

 Fig. 2. The Raspberry Pi Camera Rev 1.3 board (left) and Raspberry Pi 2
Model B V1.1 (right).

II. IMAGE PROCESSING METHOD

The proposed method uses Raspberry Pi board as the
main computation for real-time image processing. Using
real-time image processing methods, many functions are
performed such as line detection, edge detection, corner
detection and color of traffic light detection for detection of
the road condition. In this study, the used image of road
model and the car model can be depicted in Fig. 3. The
previous study of image processing method can be seen in
[1], [8], [10], [15].
Fig. 1. Modified car model equipped with Raspberry Pi and camera
module (R/C 1:10).

B. Raspberry Pi 2 Model B and Raspi Camera

The Raspberry Pi 2 Model B is the second-generation
Raspberry Pi. It replaced the original Raspberry Pi 1 Model
B+ in February 2015. The Raspberry Pi 2 Model B (Raspi)
originally intended for educational purposes has become
famous immediately after its introduction in 2012. The
Raspberry Pi 2 delivers 6 times the processing capacity of
previous models. This second-generation Raspberry Pi has
an upgraded Broadcom BCM2836 processor, which is a
powerful ARM Cortex-A7 based quad-core processor that
runs at 900MHz. The board also features an increase in Fig. 3. Original Image of the car model and the road line model.
memory capacity to 1Gbyte.
For computer vision purposes, the Raspberry Pi Camera A. Line Detection and Recognition
rev 1.3 Board (Raspi Camera) plugs directly into the CSI
connector on the Raspberry Pi. The Raspberry Pi Camera Lanes in the roadway are normally marked by white
Board plugs directly into the CSI connector on the Raspberry segments, which can be identified and captured by real-time
Pi. It’s able to deliver a crystal clear 5 MP resolution image, vision system's optical sensor (RPi Cam) in RGB format.
the camera is capable of 2592 x 1944 pixel static images, and The usefulness of road line readings is to keep the car's
also supports 1080p/30fps, 720p/60fps, and 640x480p 60/90 position on track when touching the road line, then the car
video recording. will automatically get back on the track.
Raspi Camera is custom designed and manufactured by In this study, the used method in line detection is Hough
the Raspberry Pi Foundation in the UK. The Raspberry Pi Transform. Hough Transform is a very often used method
Camera Board features a 5 MP Omni vision 5647 sensor in a for reading patterns such as line, circle, ellipse, and
fixed-focus module. The module attaches to Raspberry Pi, by parabola. The advantage of using Hough Transform is to
way of a 15 Pin Ribbon Cable, to the dedicated 15-pin MIPI detect an edge with gap at the feature boundary and is
Camera Serial Interface (CSI), which was designed
relatively unaffected by noise.
especially for interfacing to cameras. The CSI bus is capable
of extremely high data rates, and it exclusively carries pixel The general equation for finding the line is as follows:
data to the BCM2835 processor.
The board itself is tiny, at around 25x20x9 mm, and x cos(θ ) + y sin(θ ) = r (1)
weighs just over 3g, making it perfect for autonomous car
model applications where size and weight are important.
The proposed of single-board computer and the camera can
be seen in Fig. 2.

Fig. 4. The coordinate space of x and y for finding the line.

47
B. Edge Detection corner detection algorithms and defines a corner to be a
Edge detection is an operation performed to detect point with low self-similarity [10].
borders that constrain two homogeneous imagery areas that There is more sophisticated corner detectors named after
have different brightness levels. The goal of edge detection their authors. Harris and Stephens algorithm is selected in
is to convert the 2D image into a curve shape. Edge is some detecting the corner of the line of road image. The Harris
part of the image where the intensity of the brightness and Stephens formula can be found in ref. [16], [17]. Harris
changes drastically. and Stephens Corner Detection is a corner detection
There are several methods in edge detection, among algorithm that is often used because it can produce a
them are Robert method, Prewitt method, Sobel method, and consistent value in the image that experiencing the rotation,
others. The method used in this experiment is the Sobel scaling, different light intensity conditions or have a lot of
method. The Sobel method has the ability to reduce noise noise in the image [8].
before performing edge detection calculations so that the
edge is generated more than before. This method uses two D. Traffic Light Detection
kernels of 3x3 pixels for the gradient calculation. To reduce the accident that happened at the traffic light
The gradient quantities calculated using a Sobel operator intersection, we develop Raspi camera based on algorithm
are as follows: which is real-time to detect red light of traffic light. We use
simple color extraction method. This is useful for
controlling the vehicle to stop if the red traffic light when it
G = Sx2 + S y2 (2) is turned on. The color detection of traffic light has been
studied previously in ref [18]. The original traffic light
image for red light detection can be seen in Fig. 5. In this
where method, we use the color threshold for converting the RGB
G is for = Gradient of Sobel Operator image to binary image. The threshold is selected based on
the red color. The proposed method will be tested in Fig. 5
Sx=Horizontal Sobel Gradient and the resulting image is presented in section III.
Sy=Vertical Sobel Gradient
G is also the gradient at the midpoint of the kernel and
the partial derivatives that are calculated using the following
equation as expressed in equation (3) and (4).

S x = (a2 + ca3 + a4 ) − (a0 + ca7 + a6 ) (3)

S y = ( a0 + ca1 + a2 ) − ( a6 + ca5 + a4 ) (4)

Where c is a constant value of 2. Sy and Sx are

implemented into the following kernels:
Fig. 5. The original image of Traffic Light Detection.

 −1 0 1 
S x =  −2 0 2 (5) III. IMPLEMENTATION IN MATLAB/SIMULINK
 −1 0 1  MATLAB is an interactive program for numerical
computation and data visualization. Simulink is a MATLAB
graphical extension to model and simulate a system. In
Simulink, the system is described as a block diagram,
 −1 2 1 
including transfer function, summing junction, in which
S y =  −2 0 2  (6)
there are also virtual input and output devices. While in
 −1 −2 −1 Simulink, data/information from various blocks sent to other
blocks connected with line. The user without programming
C. Corner Detection In this study, the image processing algorithms discussed
Corner detection is an approach used within computer in section III will be developed under MATLAB/Simulink
vision systems to extract certain kinds of features and infer environment. The image processing block uses Simulink
the contents of an image. Corner detection is frequently used Support Package for Raspberry Pi Hardware. it can be
in motion detection, image registration, video tracking, downloaded freely on the MathWorks web. For reading the
panorama stitching, 3D modeling and object recognition. image/video from Raspi camera, video capture block is
Moreover, corner detection algorithm is one of the earliest used. While the image processing algorithm uses Computer

48
Vision System Toolbox. detection used Raspberry Pi 2 Model B with Raspberry
Camera. The proposed methods from Fig. 6 to Fig. 9 are
The diagram block for line detection can be seen in Fig. embedded to Raspberry Pi single-board computer using
6. The image/video is captured using Video Capture block. Simulink Support Package for Raspberry Pi Hardware. The
The image is converted to intensity image using color space communication of host computer and Raspberry Pi is
conversion. It processed to edge detection using Sobel conducted by using ethernet connection. The selected pixel
operator and then it processed using Hough transform and is 640x480. The chosen frame per second (fps) in this
Hough line. research is 30 fps. This is the highest frame per second that
can be used in image/video processing using Raspberry Pi.
The line and traffic light detection are tested in indoor room
with fixed lighting.

A. Line Detection
Line detection of RGB image is conducted after the RGB
image is converted to an intensity image. The process of
converting the RGB image to intensity is conducted using
Color space conversion block. Intensity image can be
Fig. 6. Block diagram of line detection in the Simulink block diagram. represented as a single matrix. Each element of the matrix in
intensity image corresponding to one image pixel. The
For the corner detection, image/video is captured using resulted from line detection is shown in Fig. 10. Based on
Video Capture block. The captured image/video is the figure, it can find the right lane on the road image and
converted into an intensity image. The image then processed detect line on the right side of the road image. The detected
using corner detection method block using Harris & Stephen line is indicated with blue line. Based on the resulted image,
as shown in Fig. 7. For edge detection, image/video is the proposed method cannot detect dashed line and line
captured using Video Capture block. The captured having corner. It can detect perfectly on the continuous line.
image/video is converted into intensity image. The image
then processed by edge detection block using Sobel operator B. Edge Detection
as shown in Fig. 8. In red light detection, the algorithm uses The initial image for edge detection is RGB image. The
color selection using threshold. The threshold is selected image must be converted into an intensity image. The
based on the red color as shown in Fig. 9. resulted image from edge detection is shown in Fig. 11. The
edge detection using Sobel operator can detect the straight
double lines on the left side and right side, as well as dashed
line in the center of the road. The edge detection detects all
of the edges in Fig. 10 including the car model image.

Fig. 7. Block diagram of corner detection in Simulink block diagram.

Fig. 8. Block diagram of edge detection in Simulink block diagram.

Fig. 10. Resulted line detection.

Fig. 9. Block diagram of traffic light detection in Simulink block diagram

for red.

IV. EXPERIMENTAL RESULTS

Real-time image processing for line detection, edge
detection, corner detection and red light of traffic light

49
 V. CONCLUSION AND FUTURE RESEARCH
Based on the results in section IV, the line detection
method can be implemented for an autonomous car model
that use camera as vision sensor. The line detection will be
used in detection of road images when the road is straight
line. The Harris and Stephens corner detection can be
implemented for autonomous car model for detecting the
corner of the road. For edge detection, it is hard to
implement the method for an autonomous car because it will
detect all of the edges captured by camera. Red color
detection of traffic light still has the challenge to implement
it on the outdoor. Because the proposed method is not robust
Fig. 11. . Edge Detection by Sobel Operator. to the various lighting condition.
In future research, the corner detection will be enhanced
C. Corner Detection for estimating the angle of the line from o degree to 90
The performance of corner detection using the road image degrees. The traffic light detection will be improved using
can be depicted in Fig. 12. The Harris and Stephens corner pattern recognition method with neural network. The
detection can detect the left corner of the road image. The method will be tested under various lighting conditions.
detected left corner on the road image can be indicated with
a green mark as shown in Fig. 12. Based on the image, the
method can detect sharp corner with the angle of 90 degrees. ACKNOWLEDGMENT
The method can also detect the tip of the line road of the The authors gratefully acknowledge partial funding
image. support from the United States Agency International
Development (USAID) under the Sustainable Higher
Education Research (SHERA) program and research fund
from the Faculty of Engineering, Diponegoro University.

REFERENCES

[1] G. Mu, Z. Xinyu, L. Deyi, Z. Tianlei, and A. Lifeng, (2015), Traffic

light detection and recognition for autonomous vehicles, J. China
Univ. Posts Telecommun., vol. 22, no. 1, pp. 50–56.
[2] M. Omachi and S. Omachi, (2009), Traffic light detection with color
and edge information, in 2009 2nd IEEE International Conference on
Computer Science and Information Technology, 2009, pp. 284–287.
[3] M. B. Jensen, M. P. Philipsen, A. Møgelmose, T. B. Moeslund, and M.
M. Trivedi, (2016), Vision for Looking at Traffic Lights: Issues,
Fig. 12. . Result of corner detection image using Harris & Stephen. Survey, and Perspectives, IEEE Trans. Intell. Transp. Syst., vol. 17, no.
7, pp. 1800–1815.
[4] K.-H. Lu, C.-M. Wang, and S.-Y. Chen, (2008), Traffic light
D. Traffic Light Detection recognition,” J. Chin. Inst. Eng., vol. 31, no. 6, pp. 1069–1075.
The developed algorithm can detect the red light on the [5] N. Fairfield and C. Urmson, (2011), Traffic light mapping and
traffic light as shown in Fig. 13. The detected red light is detection,” in 2011 IEEE International Conference on Robotics and
indicated by red square line with a circle green mark on its Automation, 2011, pp. 5421–5426.
center. When the yellow and green are light on, the image [6] H. Y. Cheng, B. S. Jeng, P. T. Tseng, and K. C. Fan, Lane Detection
With Moving Vehicles in the Traffic Scenes, IEEE Trans. Intell.
processing method does not detect the colors. Transp. Syst., vol. 7, no. 4, pp. 571–582, Dec. 2006.
[7] Yue W, Dinggang S, and Eam K. T, Lane detection using spline
model, Pattern Recognit. Lett., vol. 21, no. 8, pp. 677–689, Jul. 2000.
[8] K. Horak and L. Zalud, (2016), Image Processing on Raspberry Pi for
Mobile Robotics, Int. J. Signal Process. Syst., pp. 494–498.
[9] A. Ali and H. A. Hussein, (2017), Traffic lights system based on RFID
for autonomous driving vehicle, in 2017 Annual Conference on New
Trends in Information Communications Technology Applications
(NTICT), pp. 122–127.
[10]Muhammad S, Muhammad F, and Prashant P, (2013), Robust lane
detection and object tracking. Sweden: Master’s Thesis, School of
Engineering, Blekinge Institute of Technology.
[11] M. Malathi and R. Geetha, “A Real Time Image Processing Based Fire
Safety Intensive Automatic Assistance System using Raspberry Pi,”
Int. J. Mod. Trends Sci. Technol., pp. 34–38, 2016.
[12] Vikas Reddy Sudini, Real-time Vision-Based Lane Detection with 1D
Haar Wavelet Transform on Raspberry Pi. Master Thesis, Utah State
University, 2017.
Fig. 13. . Red Color Traffic Light Detection.

50
[13] L. Kalms, J. Rettkowski, M. Hamme, and D. Göhringer, “Robust lane [17] Y. Xiao, Y. Wang, X. Dan, A. Huang, Y. Hu, and L. Yang, “Corner
recognition for autonomous driving,” in 2017 Conference on Design detection and sorting method based on improved Harris algorithm in
and Architectures for Signal and Image Processing (DASIP), 2017, pp. camera calibration,” presented at the Society of Photo-Optical
1–6. Instrumentation Engineers (SPIE) Conference Series, 2016, vol. 0023.
[14] Peter Corke, Robotics, Vision and Control - Fundamental Algorithms [18] T. H. P. Tran, C. C. Pham, T. P. Nguyen, T. T. Duong, and J. W. Jeon,
in MATLAB. Springer, 2011. “Real-time traffic light detection using color density,” in 2016 IEEE
[15] M. Malathi, and R. Geetha, “A Real Time Image Processing Based International Conference on Consumer Electronics-Asia (ICCE-Asia),
Fire Safety Intensive Automatic Assistance System using Raspberry 2016, pp. 1–4.
Pi,” Int. J. Mod. Trends Sci. Technol., vol. 2, pp. 34–38, 2016.
[16] C. Harris and M. Stephens, “A combined corner and edge detector,” in
In Proc. of Fourth Alvey Vision Conference, 1988, pp. 147–151.

51