ECE 484W: Assignment #4

Author:
Warren Shields
UIN: 01080181

Group Members:
Jermahre Moore

Lab Due: April 12, 2021

Honor Code:

ODU Honor Code must be adhered to. This means that homework assignments and design
projects are to be the work of an individual student group only. Evidence such as identical results
and/or wording of sections of a report, if strong enough, will be reported to the University
Hearing Officer in charge of administrating the ODU Honor Code. If the violation is deemed
sufficient, a permanent record of this infraction will be placed on the student's official university
transcript for the first offense! Subsequent offenses could result in dismissal from the university.
Abstract
Table of Contents
List of Figures...............................................................................................................................................4
List of Tables................................................................................................................................................5
1. Introduction.........................................................................................................................................6
1.1. Objective......................................................................................................................................6
1.2. Image Overlay..............................................................................................................................6
1.3. Brightness and Contrast Control..................................................................................................6
1.4. Report Structure..........................................................................................................................6
2. Design Methodology............................................................................................................................7
2.1. Image Overlay..............................................................................................................................7
2.1.1. Sender..................................................................................................................................8
2.1.2. Receiver.............................................................................................................................11
2.2. Brightness and Contrast Control................................................................................................13
2.2.1. Sender................................................................................................................................13
2.2.2. Receiver.............................................................................................................................14
3. Analysis and Results...........................................................................................................................15
4. Alternative Design.............................................................................................................................15
5. Additional Considerations in Design Process.....................................................................................15
6. Broader Impacts................................................................................................................................15
7. Individual Contribution......................................................................................................................15
References.................................................................................................................................................16
Appendix...................................................................................................................................................16
List of Figures
Figure 1: Image overlay process..................................................................................................................7
Figure 2: Image overlay communication flow..............................................................................................7
Figure 3: New transfer buttons added to GUI..............................................................................................8
Figure 4: UDPclass constructor binds to address and port..........................................................................9
Figure 5: Algorithm for portioning number of bytes written.....................................................................10
Figure 6: Writing start and operation identifier to socket.........................................................................11
Figure 7: Writing end identifier to socket..................................................................................................11
Figure 8: Image overlay receiver GUI.........................................................................................................12
Figure 9: Append all bytes from operation to private member variable...................................................12
Figure 10: Image overlay algorithm...........................................................................................................12
Figure 11: Brigtness/contrast control sender GUI.....................................................................................13
Figure 12: sendSlider function definition...................................................................................................14
Figure 13: Brightness/contrast control receiver........................................................................................15
List of Tables
Table 1: Image overlay receiver operations...............................................................................................12
Table 2: Brightness/contrast control receiver operations.........................................................................15
1. Introduction
1.1. Objective
The objective of this lab was to modify the previously designed Qt GUIs where one could
overlay an image over a base image, and one could alter the brightness and contrast of a loaded
image via sliders. Modifications will allow image transfer to the VEEK-MT2S FPGA board from
the Qt GUIs via Ethernet interface. Qt’s UDP socket library, QUdpSocket, is utilized for data
transfer between the two devices. The GUI is then to communicate values over to the board to
perform the appropriate image processing techniques.

1.2. Image Overlay

The image overlay GUI application designed in assignment 1, allows its users to load and display
a base and overlay image via buttons. The user can then press a button which performs the
overlay and displays it on the GUI. For assignment 4, the GUI is modified to send the data to the
FPGA board’s address via UDP interface. The application is split into two Qt applications for
sending the data and for receiving the data on the FPGA. The board will need to perform the
overlay as controlled by the GUI on the host machine.

1.3. Brightness and Contrast Control

The brightness and contrast control GUI designed in assignment 1, allows its users to load an
image to which can then have its brightness and contrast altered via two sliders. The application
is modified to allow the user to transfer the image to the FPGA board and control the brightness
and contrast of the on-board image via the Qt GUI’s sliders. Like the image overlay application,
the host Qt GUI communicates the slider values to the board and then the board performs the
appropriate image processing techniques.

1.4. Report Structure

Section 2 will present the design methodology for the aforementioned tasks. Section 3 will
present the results and analyses for the designs along with the approximate average times that it
takes for images to be fully received by the FPGA. Section 4 proposes an alternative design.
Section 5 discusses design considerations for the assignment, Section 6 discusses the broader
impacts for the assignment, and Section 7 discusses the individual contributions of the group
members.
2. Design Methodology
2.1 Sender
Qt Creator was used to design the sender program which ultimately serves as the GUI that allows
user inputs. The capabilities of the GUI user input allow users to load and save images to and
from files, transfer the selected loaded image to the FPGA, and modulate the contrast and
brightness slider values of the display stream on the camera of the FPGA.

2.1.1. Send Image

We defined a UDP class is defined to establish member function for the transferal of the image.
The member functions utilize the Qt Creator library, QUdpSocket, which successfully
establishes UDP socket communication between the GUI and VEEK FPGA board. The image
must be configured into datagrams so that the image can be sent to the binded UDP socket. To
accomplish this process, we defined a function called sendImage which converts the loaded
image into datagrams so that we can transmit the image to the UDP socket. The function initially
writes a 9-byte array to the socket where in the first five bytes read “Start”. This process notifies
the receiver that data from the GUI is being sent to the receiver and needs to be further decoded.
The last 4 bytes of the 9-byte array read “Load” that is stored in a string id variable format. This
communicates to the receiver that the image data is being delivered and must be decoded
accordingly. The process is displayed in Fig. 2

Figure 1: Starts communication for sending the image through the socket.
The image is downscaled to 256 by 256 before the actual image is converted to a byte
array. This method was performed to prevent data loss that would’ve interfered with sending
large packets of information via UDP. We must keep in mind that the QUdpSocket library only
enables users to write to the socket at 1,024 bytes at a time. Due to this constraint, the datagrams
must be sent in multiple portions of 1,024 bytes or less. An algorithm was constructed that writes
the datagrams to the socket in portions of 1,024 bytes or less until all data are sent. The
implementation of this design in Qt is displayed in Fig. 3.

Figure 2: Datagrams written to the socket in portions of 1024 or less.

Equation 1 displays the process of how the number of loops is formulated.

¿ of Bytes
¿ Loops= +1 (1)
1024

The sendImage function is then called in the slot function, on_transfer_btn_clicked, for
the button on the GUI labeled “Transfer Image”. This will result in the sending of the image data
to the receiver side when the “Transfer Image” button on the GUI is pressed.

Finally, “End” is delivered to the socket for it to communicate to the receiver that all data
has been sent. Once, the receiver receives “End”, then the decoding process is concluded, and the
final image can be displayed.

2.1.2. Send Slider

The sendSlider function is defined in the UPD class function which is ultimately responsible for
delivering the numerical value that is associated with the moved slider to the receiver end. Once
the values are delivered to the receiver end, it is then to be decoded. Similar to the sendImage
function, the sendSlider function initializes its communication process by sending a 9-byte
array. In the byte array, the first five bytes are “Start” and the last four read a 4 byte identifier.
For this particular function, when the brightness slider is moved, the id is “Brig”. When the
contrast slider is moved, the id is “Cont”. The number that is affixed with the slider position is
then converted to a 2-byte array and appended to the end of the nine preceding bytes of the array.
Likewise, to the sendImage function, “End” is written to the very end of the socket to
communicate to the receiver that all data has been sent. Once all data is completed, the contrast
and brightness can be applied.

The sendImage function is then called in both slot functions for the contrast and
brightness sliders. Once the function is called, “Brig” is passed as the identifier into brightness
slot function and “Cont” is passed into the identifier in the contrast slot function.

2.2 Receiver
The program for the Receiver end is written in C++. It is a modification of the existing
VEEK-MT2s “camera_in” project which captures each frame from the FPGA’s on-board
camera display. Once each frame is capture, it displays it to the board’s LCD monitor. The
program must read pending datagrams from the socket. For this to established, we used the
“sys/socket.h” library and C++ coding to implement the UDP networking capabilities in the
program. Fig. 4 displays the complete flow diagram of the original design.
 Figure 3: Flow Diagram for the Original Reciver.

2.2.1 Non-blocking UDP Socket

The program works in series as displayed in Fig. 4. Due to the series operating functionality of
the program, we implemented a non-blocking UDP socket with the sys/sockets fcntl function
because a blocking socket would cause the camera to only capture when there are pending
datagrams. Hence, the video display will seem to be frozen when the user is not modulating the
image via the GUI. The non-blocking UDP socket is useful because it will allow the program to
continue its process even when there are no pending datagrams to be read. An infinite loop is
created after implementing the socket, by using a “while (1)” statement. When the escape key is
pressed, the program is then terminated. The program can also be terminated via command line
as well. First, the program calls the D8MCapture class’ read function which retrieves the camera
frame and stores it as a mat variable or OpenCV matrix. After the calling of the D8MCapture
class, the program is followed by the sys/socket’s recvfrom function which writes the pending
datagram from the socket to an array labeled as buffer. The recvfrom function also returns the
number of characters in the datagram to int n. Below, Fig. 5 shows this implementation in code.

Figure 4: Retrieve camera frame and receive datagram.

2.2.2 Datagram Decode and Append

Recall from Section 2.1. that the sendImage and sendSlider functions in the Sender program
both established communication by sending “Start” to the socket. A private member bool in the
D8MCapture class is set to true when the “Start” is received. Also, the D8MCapture class
contains a private member string combineBuffer. The character array buffer, holding the
received characters from the datagram, is appended to combineBuffer. This process will
continue in sequence until the sockets read “End”. Once the sockets read “End”, it will clear the
combineBuffer and set start to false. Setting start to false will stop the program from entering
the if statement.
 Inside this for loop, the program also checks for the id sent from the sendImage and
sendSlider functions. The provided identifiers determine the opcode for the receiver which
determines the appropriate operation for the program. Table 1 details the program’s behaviors for
different opcodes. Note that the opcode is initialized as 0.

String Identifier Opcode Operation

Load 1 Convert combineBuffer to image
Brig 2 Set brightness variable to value stored in buffer
Cont 3 Set contrast variable to value stored in buffer
Table 1: Receiver decode.
Note that the string identifiers are only read in the first iteration of datagrams being sent for a
specific input. Hence, the opcodes are read in the first iteration. The described operations in
Table 1 only occur once “End” is received from the socket. Once “End” is received, the opcode
is set back to 0. When the opcode is 1, the private member bool connected is set to true to
indicate to the program that an overlay image exists.

To avoid a contrast equivalent to 0, which would completely blacken the display, the
contrast value is initialized as 10 and will remain 10 until the transmitted value is greater than 10.
Note that 10 is selected as the minimum value because when the contrast is applied, the value is
multiplied 0.1. The reasoning for this will be explained later.

2.2.3 Perform Overlay and Apply Brightness and Contrast

After the if statement, the program checks to see if an overlay image exists by checking
connected. If true, the program overlays the overlay image in the top corner of the camera frame
using OpenCV’s copyTo function. The program then applies the brightness and contrast values
with OpenCV’s convertTo function. As stated earlier, the contrast is multiplied by 0.1. This is
because convertTo intensifies the contrast exponentially. By dividing the slider values by 10, the
contrast intensifies more gradually as the sliders are moved. The brightness value is doubled to
get more immediate results when the brightness slider is moved.
 The program then displays the new image by using OpenCV’s imshow function. The
program then continues to loop. As expected, the FPGA only displays what is on the camera as
there has yet to be an overlay or alteration of the brightness or contrast.

Figures 8-10 depict the results when the brightness and contrast sliders are changed.

Figure 5: Brightness set to 35 and contrast set to 0.

Figure 6: Brightness set to 0 and contrast set to 20.

 Figure 7: Brightness set to 0 and contrast set to 86.
In Fig. 8 it can be observed that increasing the brightness slider brightens the FPGA’s display
and makes the image more visible. Adjusting the contrast, visibly improves visibility of what is
shown in the camera frames. In the same way as getting the time for the average image load
time, we find an average brightness and contrast update time of approximately 51.2ms.

Although the program successfully worked and satisfied each of its requirements, its
performance is not as sufficient. The average image load time of 1.76 is not instantaneous and
the brightness and contrast control time end up causing poor latency issues which results into
delayed contrast/brightness modulations when the slider values are changed quickly. Due to the
undesirable performance, quickly dragging the sliders can potentially result in discrepancies
between the slider values on the GUI and the brightness and contrast display on the FPGA. We
measured the average CPU usage of the design which resulted in 96% usage. This leads to the
program crashing when the contrast is changed by a lot in an instant amount of time. Our
Alternative Design which is included in the next section, discusses these problems and solutions
we use to remedy the effect.

3. Alternative Design
Recollecting back to the original design, the process captures frames from the camera, receives
data from the UDP socket, and decodes the datagrams in series. This process takes longer to
complete due to it being in series and must wait for one process to finish. It can be observed that
the camera capture and the UDP datagram receive and decode process, can be ran as independent
processes. Also, it is imperative to know that the non-blocking UDP sockets can lead to higher
CPU consumption due to the constant reading for data, even when there are no pending
datagrams to be read.

In our alternative design, the program is split into two different threads by using the C++
“thread” library, so that they can work in parallel. One process has the responsibility of capturing
the frames from the camera, performing the overall, and applying the brightness and contrast
values coherently. On the opposing end, the new thread is responsible for receiving the
datagrams from the socket and then decoding and processing the data.
 Figure 8: Alternative design flow diagram.
Implementation of the new thread shifts the relevant blocks of a code to the function
receiver, which ultimately is made a class function for the D8MCapture. In the receiver
function, the socket is set to blocking because the it no longer blocks the camera capturing.
Implementation of this methodical approach makes the non-blocking setting unnecessary. The
new thread is then called in the main() as ‘thread
fprocessData(&D8MCapture::receiver,cap);’. This allows the function to be ran as a
separate thread that runs simultaneously with the main thread. Below, Table compares the results
between the original and alternative designs.
 Metrics Original Alternative
Avg. Image Load Time ~1.76 sec ~25 ms
Avg. Brightness/Contrast Update Time ~51.2 ms ~0.16 ms
Avg. CPU Usage ~96% ~83%
Table 2: Original vs. alternative design.
As displayed, the collected shows that the alternative design gives approximately 70x
faster image load time, approximately 313x faster brightness and contrast update time, and
reduces CPU usage by 13%. To achieve the most optimal performance, it can be observed that
the use of multithreading and blocking sockets instead of non-blocking sockets will enhance
overall performance.