Embedded Image Processing On Raspberry Pi Connecte
Embedded Image Processing On Raspberry Pi Connecte
Jozef Hrbček*
ORCID ID: 0000-0002-6984-9470
Emília Bubeníková
ORCID ID: 0000-0003-3053-0822
University of Žilina, Slovak Republic
INTRODUCTION
Nowadays, process control, the automatic measurement system and the data
processing systems are using programmable logic controllers (PLCs). The
architecture and computation power of PLC is the same as PC, but their realization is
suitable for use in industrial applications. These are suitable for their determination –
resistance to weathering, disturbance and continuous operation (IEC 61131, 2019).
Last but not least, the reliability features are also included. The used PLCs from B&R
Company can work in redundant connection to increase the control system reliability.
The Raspberry Pi is used to process the image and provide the requested information
about the controlled system to the PLC via implemented industrial Ethernet
POWERLINK (EPL). Our solution requires the real-time communication and
computation power for image processing for the purpose of control and monitoring
the systems. These functionalities are important for industrial application. Our main
goal is creation and using the low cost solution instead of using the vision system
offering by a few manufacturers (e.g. Siemens, Cognex, B&R, Keyence, etc.).
Computer vision is an important field of research, too. The general steps used in this
process are acquiring, processing and analyzing the image and convert it into
numerical or symbolic form. Part of the computer vision is image processing, image
analysis and machine vision is also closely related to a computer vision. There are a
large number of applications in many areas in the world where the image processing
is used; for example, in medicine (Koniar et al., 2017), access control systems,
transport (Bubeníková, 2015), (Zolotová and Lorenčík, 2018), biometrics and
especially in industrial processes (measurement, product quality control,
identification, localization), (Mičieta et al., 2015), navigation of robots or MAV (Hruboš
et al., 2016), (Pirník et al., 2016).
RESEARCH METHODOLOGY
In our paper, we describe a concept system that processes the acquired image
information from the 8MP RPI Camera Board for the Raspberry PI V2.1 and the image
data from it is processed through a Raspberry Pi single-board computer. The camera
can give the image with 3280x2464 pixels resolution and video mode with 1080p30
or 720p60. The image processing unit extracts data from the image and transmits it
through the communication interface. The system may also be equipped with
additional USB surveillance cameras. More information about the surveillance
cameras can be found in (Halgaš & Pirník, 2015). Cooperation between the vision
systems and automation solutions based on the high-performance industrial
communication protocol open up new perspectives. The PLC system with CPU
redundancy provides a layer between image processing unit and superior level (HMI).
This concept has been designed to realize the reliable operation of the device under
operating conditions.
Our application uses the low cost component with possibility of image processing
algorithms. The implementation of the algorithms has been designed to allow testing
and validating various object measurement approaches, e.g. distance, radius and
speed or detection of different objects such as detecting circles or faces. Our
application uses connected embedded system with HD camera. This paper is focused
on the system which are the processing unit and the PLC system. Modular PLCs are
a system of modules that can be slotted together to build up a system. The basic
modules contain a power supply, central processing unit (CPU), communications
modules and the input and output modules. This system can be expanded by an
amount of input and output terminals. Some PLCs use the open source
communications networks and allows us to connect the embedded devices to this
modular system – it depends on the system manufacturers. The currently most used
PLCs are (sorted alphabetically): B&R, Beckhoff, Mitsubishi, Omron, Rockwell,
Schneider-Electric and Siemens. Only the B&R producer prefer the open
communication protocol with high speed (the communication cycle less than 400µs)
and real-time deterministic approach. Moreover, the B&R PLCs support the local and
remote visualizations, web server, ftp server and http/https communication with a
cloud according to the concept INDUSTRY 4.0. These PLCs also support the third
party devices connection without need of any additional modules. Usage of the B&R
control system with image processing unit guarantees the successful solution.
Embedded systems can be connected as a node of EPL bus by adding the XDD (XML
device description) file from the openPOWERLINK source code stack to the
programming tool Automation Studio (Automation Studio, 2019).
COMMUNICATION PRINCIPLE
The deterministic, reliable and fast communication bus is the important requirement.
Ethernet POWERLINK (EPL) is the open source real-time Ethernet protocol adopting
the CANopen object dictionary concept and its communication mechanisms that
provide features like interoperability, flexibility and configurability. The lower layers of
the EPL software are specifically designed to follow the EPL cycle schedule and to
guarantee fast response times for real-time events on the bus. For comparison, the
shortest communication cycle time for EPL is 100µs comparing to Profinet’s support
of 1000µs. New solution for real-time communication is OPC UA TSN. This
networking solution uses IEEE-standard Ethernet with TSN (Time Sensitive
Networking) using OPC UA as its communication protocol. The OPC UA TSN is going
to supersede various industrial networking based on Ethernet. The prototype with 200
nodes can communicate with 50µs communication cycle. In the future 10µs cycle time
is expected. It depends on the hardware and software used (Bruckner et al., 2018).
Our solution uses industrial network Ethernet POWERLINK. This open source
network protocol was implemented to the embedded system.
Implemented EPL stack version 2.7.1 on Raspberry Pi realizes deterministic
communication with the redundant control system (Fig. 1). The communication cycle
time is now 2000µs. Image processing realizes the Raspberry Pi with the operating
system Raspbian Jessie kernel 4.14 with implemented RT-PREEMPT. RT-PREEMPT
is a popular patch for the Linux kernel to transform Linux into such a real-time
operating system. The cyclic test program provides an easy way to assess the
maximum latency of a system. The cyclic test basically measures how much it takes
to respond to an interrupt generated by the CPU timer.
Table 1
The cyclic test
Operating system Max latency Average latency
Raspbian without RT-PREEMPT 502µs 14µs
Raspbian with RT-PREEMPT 42µs 7µs
Implemented RT-PREEMPT makes the maximum latency more than 10 times shorter.
It makes possibility for faster real-time communication. The tests proved that the
maximum communication cycle time could by only 10000µs without RT-PREEMPT. It
is very important to have a low time delay for receiving and sending commands and
data in automated production processes. This can be ensured by Ethernet-based
industrial Ethernet networks. One of them is Ethernet POWERLINK, which is
supported by B&R devices. OpenPOWERLINK is a complete protocol solution for
masters and slaves. This implementation can be ported to any target system
programmed in ANSI C.
As a first step we have needed to install CMake, CMake GUI, libpcap and WiringPi
on the Raspberry Pi. Libpcap is a system-independent interface that is designed for
capturing packets at the user-level providing a framework for network monitoring at
lower levels. By modifying the source code in linux.cmake, main.c, app.c, and app.h,
we have defined the required data to be transmitted. We have defined the input and
output structures and then we have created local and global application variables.
These variables are used to transfer information between control system and
Raspberry Pi. Then we have set up the configuration files and have compiled the
openPOWERLINK source code.
IMAGE PROCESSING
Image processing is a method that performs certain operations to obtain an improved
image or to obtain useful information from an image. We use the 8MP camera v2.1
described above to capture images. It is possible to use other external USB camera.
Image processing basically consists of three parts:
− acquisition of an image,
− Image analysis and manipulation,
− an output that can be a modified image or an information based on the analysis.
The expansion of image recognition technology effect open-source tools that make
programming easier, and make computing more affordable. Open-source frameworks
and libraries today make possibilities for companies to benefit from image recognition
technology exponentially. For example, libraries as IPL (Image Processing Library)
Thresholding
Thresholding is an image processing operation that converts a grayscale image to
binary. In the simplest implementation, the output is a binary image representing
segmentation of processed image information. Segmentation is determined by a
parameter called the T intensity threshold:
0 ݂ݔ(݂ ݎ, ܶ ≥ )ݕ
݃(ݔ, = )ݕ൜ ൠ. (1)
1 ݂ݔ(݂ ݎ, ܶ < )ݕ
where:
f (x, y) is the input grayscale matrix,
g (x, y) is the threshold obtained,
T is defined threshold.
When the image is thresholded, each pixel of the input image matrix is compared to
the threshold T. If on the other hand the intensity of the pixel is less than a selected
threshold T, the value of respective binary pixel is set to 0 (Thresholding, 2019). In
the practical application, we used the function to threshold the image:
cv2.threshold (src, thresh, maxValue, cv2.THRESH_BINARY); where,
the individual arguments are functions: cv2.threshold(src, thresh,
maxValue, cv2.THRESH_BINARY); where, the individual arguments are: src –
source image, thresh – value of threshold, maxValue – maximal value,
cv2.THRESH_BINARY – type of thresholding method. It is possible using one of 5
methods of thresholding. For our application we choose the simplest and generaly
using type of thresholding. Fig. 2 shows an original and thresholded image with a T =
120 threshold.
Fig. 2 The input image (left) and image after process of thresholding (right)
Color filtering
In a color image, each pixel is represented by three color values, for a red, green, and
blue channel. Each of these values occupies 8 bits, and in the form of the processed
data, it takes values from 0 to 255, meaning that in program execution it represents
an integer date type (RGB color space, 2019). In the created subprogram, this can
be used e.g. when is detecting color objects, the algorithm used a comparison and
detection of the maximum RGB intensity for each pixel of the obtained image. The
color of each pixel in the output image is set by assigning color to the maximum input
image intensity.
Circle detection
The circle is mathematically defined by equation (2) where (x_center, y_center) are
the coordinates of the circle center and r is its radius. The equation implies that three
parameters are important in circular detection, which would lead to the use of a 3D
accumulator for Hough transformation, which would be very inefficient. Hence,
OpenCV uses the Hough gradient method that uses edge gradient information:
( ݔ− ݔ௧ )ଶ + ( ݕ− ݕ௧ )ଶ = ݎଶ (2)
Function cv2.HoughCircles(image, method, dp, minDist), which we
used in our application CV_HOUGH_GRADIENT method to detect circles in images.
In Fig. 3 (on the right) we can see results of circle detection.
The output of the function is circles - the output vector of the found circles, each vector
is encoded by 3 elements x coordinates (x_center), y coordinates (y_center) and
radius (r) (Hough Circle Transform, 2019).
Fig. 3 Face detection using Haar cascade classifier (left) and Circle detection (right)
At the same time, the detected circles are rendered green and their rectangles
centered in red. This output vector is send to the control system as the structured
variable.
We have also created the GUI for testing the developed application with the basic
image operations: capturing the original image, color filtration, face recognitions, edge
detection, thresholding and circle detection with measurement.
CONCLUSION
In our paper, we described created application which used low-cost components and
allows testing functionality of some image processing algorithms. For image
processing, it has been used the following items: Raspberry Pi with HD camera and
redundant modular B&R control system. The library used for image processing has
been openCV and the programming language has been Python and C. The image
processing consists of several stages, capture the image, process the image using
the Raspberry Pi on the PLC demand and sending back the information. The
programmed image processing system can be used in practice for measurement,
checking product quality, sorting products according to given specification, or
promoting production line safety or access control. This information processing
solution was used in CNC Production Stand (as a Controlled Process) for the process
control, monitoring the quality and safety functions that is part of the KEGA project.
This solution fulfils all requirements specified in the main concept and it is usable for
the industrial control systems.
ACKNOWLEDGEMENT
The paper has been written with the support of the project KEGA 016ŽU-4/2018
Modernization of teaching methods of management of industrial processes based on
the concept of Industry 4.0.
REFEENCES
Bruckner, D. (2018). OPC UA TSN - A new solution for industrial communication. B&R
Industrial Automation. [online] Available at: https://www.automationworld.com/
sites/default/files/opc_ua_tsn_whitepaper_1.pdf [Accessed 10 Apr. 2019].
Bubeníková, E. (2015), The ways of streamlining digital image processing algorithms used for
detection of lines in transport scenes video recording, In: PDES 2015: 13th IFAC and
IEEE conference on Programmable devices and embedded systems: Cracow, Poland,
pp. 174-179.
B&R systems documentation. Redundancy. (2019). [online] Available at: https://www.br-
automation.com/en/technologies/redundancy/ [Accessed 10 Apr. 2019].
Camelia library, C++ (2019), [online] Available at: http://camellia.sourceforge.net [Accessed
29 Apr. 2019].
EPSG (The Ethernet POWERLINK Standardization Group). (2019). An Open Source
POWERLINK protocol stack: openPOWERLINK. [online] Available at:
http://openpowerlink.sourceforge.net/web/ [Accessed 10 Apr. 2019].
Face detection using OpenCV and Python: A beginner´s guide. Superdatascience, [online]
Available at: https://www.superdatascience.com/opencv-face-detection/, [Accessed 29
Apr. 2019].
Face detection using Haar cascades. OpenCV python tutorials, [online] Available at:
http://opencv-python-
tutroals.readthedocs.io/en/latest/py_tutorials/py_objdetect/py_face_detection/py_face_
detection.html#face-detection, [Accessed 29 Apr. 2019].
Halgaš, J., Pirník, R. (2015). Monitoring of parking lot traffic using a video detection. In: Acta
Technica Corviniensis - Bulletin of engineering, 8(3), pp. 17-20.
Hough Circle Transform. OpenCV Python tutorials, [online] Available at: http://opencv-python-
tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_houghcircles/py_houghcir
cles.html#hough-circles, [Accessed 29 Apr. 2019].
Hrbček, J., Šimák, V., Janota, A. and Pirník, R. (2014). Tunnel central control system
enhanced with modern control approaches. In: Archives of Transport System
Telematics, 7(3), pp. 3-7.
IEC 61131. (2019). Standard for programmable controllers. Parts 1-4.
Janota, A., Šimák, V., Nemec, D. and Hrbček, J. (2015). Improving the precision and speed of
euler angles computation from low-cost rotation sensor data. Sensors.
Automation Studio. (2019). 3rd-party devices. Bernecker + Rainer Industrie Elektronik GmbH
Help Explorer.
Hrbček, J., Šimák, V., Janota, A. and Pirník, R. (2014). Tunnel central control system
enhanced with modern control approaches. In: Archives of Transport System
Telematics, 7(3), pp. 3-7.
Hruboš, M. (2016), Searching for collisions between mobile robot and environment, In:
International journal of advanced robotic systems, 13(5), pp. 1-11.
Koniar, D. (2017). Visual system-based object tracking using image segmentation for
biomedical applications. Electrical Engineering, 99(4), pp. 1349-1366.
Librare Open CV (2019), [online] Available at: https://opencv.org [Accessed 29 April. 2019].
Library IPL-Image processing (2019). [online] Available at:
https://computervisiononline.com/software/1105138501, [Accessed 29 Apr. 2019].
Mičieta,B., Edl, M. and Krajčovič, M. (2015), Delegate MASs for coordination and control of
one-directional AGV systems: a proof-of-concept, In: The International Journal of
Advanced Manufacturing Technology [print], 94(1-4), pp. 415-431.
OpenCv library, OpenCV, AdaptiveThreshold (2019) [online] Available at:
https://docs.opencv.org/2.4/modules/imgproc/doc/miscellaneous_transformations.html
#cv.AdaptiveThreshold [Accessed 29 April. 2019].
Pirník, R. (2016), Integration of Inertial Sensor Data into Control of the Mobile platform. In:
Advances in Intelligent Systems and Computing, pp. 271-282.
RGB color space. Rapidtables. [online] Available at:
https://www.rapidtables.com/web/color/RGB_Color.html, [Accessed 29 April. 2019].
Sacimage - image processing library (2019). Available at: https://www.tandfonline.com/
doi/abs/10.1080/01431169308904451 [Accessed 29 Apr. 2019].
Scikit-image, Image processing in Python library (2019). [online] Available at: https://scikit-
image.org/ [Accessed 29 Apr. 2019].
Abstract: This paper deals with the image processing from the camera for Raspberry
Pi connected with real-time communication network to the control system (PLC). The
low time delay for receiving and sending commands, data, etc. is very important in
the automating production processes. This can be provided by industrial real-time
network based on Ethernet. The Ethernet POWERLINK, which is supported on B&R
PLCs, is one of them. It is a simple solution for a variety of applications because the
POWERLINK is publicly available as the open source. Connecting the PLC and
Raspberry Pi with Ethernet POWERLINK opens up many applications in industrial
automation. For example, image data obtained using a camera attached to Raspberry
Pi can be used to sense image of manufacturing processes and products and
evaluate their quality in industrial automation. This article focuses on an image
processing unit and the PLC system with CPU redundancy used in the industrial
application. Vision systems are often used to improve products quality control, saving
costs and time.