People’s Democratic Republic of Algeria

Ministry of Higher Education and Scientific Research

University M’Hamed BOUGARA - Boumerdes

Institute of Electrical and Electronic Engineering

Department of Electronics

Project Report Presented in Partial Fulfillment of the Requirements of the

Degree of

‘LICENCE’

In Electrical and Electronic Engineering

Title:

Fast GUI creator for embedded systems

Presented By:

MADAOUI Zakaria

Supervisor:
Mr ZITOUNI Abdelkader

181831028391/2021
Abstract

The main work of this project is to take advantage of the pocket computer that we carry with

us every day and transform it into a tool for creating quick Graphical User Interfaces that can

interact with micro-controllers and computers in general. The graphical user interfaces can be

created using modular components which can send customized commands via different types

of connections.

i
Contents

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 The problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.3 Solution proposed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.4 Features and advantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.5 Other use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.6 Similar solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Theoretical background and basics 4

2.1 The Android framework and Android general concepts . . . . . . . . . . . . . 4

2.1.1 Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.2 Fragment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.1.3 View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.1.4 ViewGroups and Containers . . . . . . . . . . . . . . . . . . . . . . . 6

2.1.5 Manifest file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.6 Broadcast Receivers . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.7 Intents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Theory of some communication protocols . . . . . . . . . . . . . . . . . . . . 8

2.2.1 Bluetooth and RFCOMM sockets . . . . . . . . . . . . . . . . . . . . 8

2.2.2 TCP/IP protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

ii
 2.2.3 UART and USB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3 Design patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3.1 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3.2 Singleton pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 Android app implementation 14

3.1 User interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.2 Database structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3 User defined Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.4 Custom commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.5 Widgets (modular components) . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.6 Graphical user interfaces (GUIs) . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.7 Implementing & Managing connections . . . . . . . . . . . . . . . . . . . . . 27

3.7.1 Bluetooth Connection . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.7.2 TCP/IP connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.7.3 UART connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4 The helper library implementation 35

4.1 Command and parameters representation . . . . . . . . . . . . . . . . . . . . . 35

4.2 ProtoHelper class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

5 Results, Examples and future improvements 40

5.1 Testing connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.2 Testing GUIs and commands . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.3 Using received commands in a real project . . . . . . . . . . . . . . . . . . . . 51

5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Conclusion 54
List of Figures

2.1 Activity lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 TCP 3 way handshake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3 UART data transmission diagram [1] . . . . . . . . . . . . . . . . . . . . . . . 10

2.4 UART packet [2] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.1 User interface structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2 Screenshots of fragments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.2 Screenshots of fragments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.3 database entities relation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.1 Parsing command state machine . . . . . . . . . . . . . . . . . . . . . . . . . 39

5.1 TCP/IP connection results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.2 HC-06 Bluetooth module interface with NodeMCU . . . . . . . . . . . . . . . 43

5.3 Bluetooth connection results . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.4 UART connection results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5.6 output from clicking first button . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.7 output from clicking toggle button . . . . . . . . . . . . . . . . . . . . . . . . 50

5.8 output from dragging the red slider . . . . . . . . . . . . . . . . . . . . . . . . 50

5.9 output from Gyroscope sensor block . . . . . . . . . . . . . . . . . . . . . . . 50

5.10 output from joystick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5.11 Commands and GUI for stepper motor example . . . . . . . . . . . . . . . . . 51

Chapter 1

Introduction

1.1 Motivation

Mobile computing technology has evolved to an extent where some smartphones became more
powerful than some regular modern computers. Today’s phones come with very powerful
CPUs, large amounts of RAM and are equipped with a verity of gadgets.

One of the fields that can benefit greatly from these devices is the IoT field (Internet of
things). According to Wikipidia, IoT describes the network of physical objects that are embed-
ded with sensors, software, and other technologies for the purpose of connecting and exchang-
ing data with other devices and systems over the Internet. And since the current smartphones
can connect in a verity of ways (WI-FI, Bluetooth,NFC . . . etc) and are empowered with dif-
ferent sensor, they serve as ideal IoT devices. They can be used as remote controllers, or for
displaying and analyzing information from various other devices or as devices that can provide
information about the environment using the sensors embedded in them.

This project tries to take advantage of these powerful devices and use them to make proto-
typing in electronics and embedded applications easier and faster.

1.2 The problem

Developers who work on products and projects with microcontrollers in them always need a
source that provides various inputs so that they can prototype and test their systems. Some-
times, off the shelf components like potentiometers, switches and push buttons can be used to
create devices that provide inputs/outputs and satisfy such a need. However, when projects get
complicated and other complex inputs and data is needed, then most likely the developer will
need to create some sort of Graphical User Interface (GUI) that connects to a microcontroller
and provides the desired data. However, creating these means while working on a relatively
large project is always time-consuming and often leads to more troubleshooting and may shift
the focus of the developer off the main goal.

1
Chapter 1

1.3 Solution proposed

The goal of this project is to develop an android application and a C++ library which together
allow creating quick graphical user interfaces (GUIs) that can connect and interact with micro-
controller by sending/receiving customized commands.

The android application gives users different components such as Buttons, Switches, Knobs,
LEDs and other I/O components, these components can be linked to user-defined commands
and can be organized to make a graphical user interface. The app also allows the user to connect
to a micro-controller or any other computer via different types of connections such as WI-FI or
Bluetooth and then use the created GUI to send and receive commands.

The other part of this project is the C++ library that: takes care of parsing the commands
received from the android application using an efficient algorithm and, it provides tools for
converting the data-types of the received parameters, and it provides callbacks when commands
are fully received and parsed.

1.4 Features and advantages

This project offers the following features and advantages:

• A single application for creating and using different controllers for multiple projects.
• Ability to use three different types of connections (WI-FI, Bluetooth and USB) to connect
to microcontrollers.
• Ability to use smartphones’ built-in sensors as sources of inputs.
• Reducing cost of a project by reusing the sensors in the mobile device in many projects.
• The library reduces the amount of code needed to process the commands sent to the
microcontroller.
• The library is also general enough to work on almost all types of microcontrollers and
even on normal computers.

1.5 Other use cases

The initial goal of this project was to provide a quick way of making GUIs for prototyping
purposes. However, it turned out that it can be of great use to many other individuals. So, the
tools developed in this project can also be used for:

• Presenting and quickly demonstrating an idea of a product.

• DIY makers and Hobbyists who don’t need to have a background in mobile development
to make interfaces for their projects. And also can be used for Educational purposes.

2
Chapter 1

1.6 Similar solutions

There exist two solutions that are similar to this project. However, each has its own advantages
and its trade-offs.

The first one is a web application and an integrated development environment called MIT
app inventor in which users can create quick mobile applications using a graphical program-
ming language similar to scratch. That web app allows users to drag and drop visual objects
that represent code blocks which allow to create an application that can run on android devices.
The advantage of MIT app inventor is that it gives more flexibility in making the graphical
interfaces. However, it always requires an internet connection to work with, and it requires the
use of a computer because the website is inconvenient to use with a smartphone. In addition,
using connections like Bluetooth and receiving the data in the microcontroller side has to be
handled and re-implemented by the user, which requires a bit more time and troubleshooting.

The other solution is an android application called Blynk. I only discovered this app at the
end of making this project and was surprised by the similarities between them. Similar to this
project, Blynk provides some modular components that can be dragged and dropped to make
quick GUIs as well as it gives the user choice between multiple types of connections. However,
Blynk have the following restrictions:

1. The only program that runs in the micro-controller is only the one that comes with Blynk
app and no other code or logic can run in the microcontroller. That means The hardware
can only be accessed by the components in the GUI which limits the use of this solution
to some small applications. Also, the program they provide is platform specific and can
only be used with specific microcontrollers.

2. The structure of data sent to the micro-controller is predefined and can’t be modified by
the user, i.e. there are no customized commands.

So we can say that our project tries to combine the benefits from both of the solutions mentioned
above while trying to have the minimum amount of trade offs.

3
Chapter 2

Theoretical background and basics

Before diving into the implementation of the project, some fundamental concepts must be es-
tablished first. So, in this chapter we will start by introducing the android framework and its
components. Then, provide some theoretical background behind some communication proto-
cols. And finally talk about some important programming concepts used during development
of the project.

2.1 The Android framework and Android general concepts

Android [8] is an open source, Linux-based software stack created for a wide array of devices
and form factors.

The entire feature-set of the Android OS is available through APIs written in the Java lan-
guage. These APIs form the building blocks that help create Android apps by simplifying the
reuse of core, modular system components and services, which include the following:

• A rich and extensible View System that can be used to build an app’s UI, including lists,
grids, text boxes, buttons, and even an embeddable web browser.

• A Resource Manager, providing access to non-code resources such as localized strings,

graphics, and layout files.

• A Notification Manager that enables all apps to display custom alerts in the status bar

• An Activity Manager that manages the lifecycle of apps and provides a common naviga-
tion back stack

• Content Providers that enable apps to access data from other apps, such as the Contacts
app, or to share their own data.

In this section we will cover some basic building blocks of an android application, as well
as some concepts used in this project

4
Chapter 2

2.1.1 Activity

An activity [4] is a single, focused thing that the user can do. Almost all activities interact with
the user, so the Activity class takes care of creating a window in which a programmer places
its UI with setContentView(View). In fact, when a user opens an application, the first thing
that launches is an Activity called ‘MainActivity’ and while interacting with the application
the user can open many activities that hold different UI elements. While activities are often
presented to the user as full-screen windows, they can also be used in other ways: as floating
windows, Multi-Window mode or embedded into other windows. An activity has its own life
cycle which is demonstrated in Figure 2.1.

Figure 2.1: Activity lifecycle

5
Chapter 2

2.1.2 Fragment

A Fragment [9] is a piece of an application’s user interface that can be placed in an Activity.
Interaction with fragments is done through a class named FragmentManager, which can be
obtained via Activity.getFragmentManager() and Fragment.getFragmentManager().

The Fragment class can be used in many ways to achieve a wide variety of results. In its
core, it represents a particular operation or interface that is running within a larger Activity. A
Fragment is closely tied to the Activity it is in, and can not be used apart from one. Though
Fragment defines its own lifecycle, that lifecycle is dependent on its activity: if the activity is
stopped, no fragments inside it can be started; when the activity is destroyed, all fragments will
be destroyed.

Some android applications use a single activity to switch between multiple fragments, and
each fragment holds a page of the application. This project uses this method since it is very
efficient than starting and ending different activities.

2.1.3 View

A View occupies a rectangular area on the screen and is responsible for drawing and event
handling. In this project the View class is the base class for widgets, which are the modular UI
components that can send commands (buttons, text fields, etc.)

2.1.4 ViewGroups and Containers

A ViewGroup is a special view that can contain other views (called children.) The view group
is the base class for layouts.

A layout defines the structure for a user interface and handles the way views are positioned
inside it. The Android framework provides many layouts such as : Constrain Layout, Relative
layout and linear layout. For instance, Relative layout allows positioning a View relative to
other Views, while Constrain layout positions views based on some constrains defined by the
developer.

The following figure shows an illustration of a view hierarchy, which defines a UI layout:

6
Chapter 2

2.1.5 Manifest file

The android framework requires a special file called AndroidManifest.xml to be included at

the root of the project source set. The manifest file [7] describes essential information about
the app to the Android build tools, the Android operating system, and Google Play.

Among many other things, the manifest file is required to declare the following:

• The app’s package name.

• The permissions that the app needs in order to access protected parts of the system or
other apps. It also declares any permissions that other apps must have if they want to
access content from the app.

• The components of the app, such as activities, broadcast receivers, and content providers.
Each component must define basic properties such as the name of its Java class.

2.1.6 Broadcast Receivers

A Broadcast [6] is an important concept in the android framework, its main use is to send
and receive broadcast messages from the Android system and other Android apps. Similar to
the publish-subscribe design pattern, apps can register to receive specific broadcasts. When
a broadcast is sent, the system automatically routes broadcasts to apps that have subscribed
to receive that particular type of broadcast. For example, in this project, the app needs to
know when a Bluetooth device is disconnected, so it subscribes to a broadcast with action :
ACTION ACL DISCONNECTED.

2.1.7 Intents

An Intent [10] provides a facility for performing late runtime binding between the code in
different applications. Its most significant use is in the launching of activities, where it can
be thought of as the glue between activities. It is basically a passive data structure holding an
abstract description of an action to be performed. It can be used with startActivity() to launch
an Activity, broadcastIntent() to send it to any interested BroadcastReceiver components.

7
Chapter 2

2.2 Theory of some communication protocols

In order to transfer data between the android device and the micro-controller, a connection has
to be established. Fortunately, smartphones can connect to other devices in many
ways : Bluetooth, WI-FI, GSM, NFC ... etc. These types of connections are used with cer-
tain protocols that define the rules, syntax, semantics and synchronization of communication
and possible error recovery methods.

The android application in this project gives users the choice to connect to other devices via
3 types of connections: Bluetooth using the RFCOMM sockets, WI-FI using TCP/IP protocols
and a hardwired Serial connection known as UART.

In this section we see the theory behind these 3 types of connections.

2.2.1 Bluetooth and RFCOMM sockets

Bluetooth [13] is a wireless technology standard used for exchanging data between fixed and
mobile devices over short distances using Ultra High Frequency radio waves (from 2.402 GHz
to 2.48 GHz), and building personal area networks (PANs). It was originally conceived as a
wireless alternative to RS-232 data cables.

A Bluetooth connection can be established through different protocols. The most common
type of Bluetooth socket is RFCOMM (Radio frequency communication), which is the type
supported by the Android APIs. RFCOMM is a connection-oriented, streaming transport over
Bluetooth. It is also known as the Serial Port Profile (SPP).

The Android Bluetooth APIs [5] provide access to the Bluetooth functionality. Using these
APIs, an Android application can perform the following:

• Scan for other Bluetooth devices

• Query the local Bluetooth adapter for paired Bluetooth devices

• Establish RFCOMM channels

• Connect to other devices through service discovery

• Transfer data to and from other devices

• Manage multiple connections.

In android, in order for Bluetooth devices to transmit data between each other, they must
first form a channel of communication using a pairing process. One device makes itself discov-
erable and available for incoming connection requests. Another device finds the discoverable
device using a service discovery process. After the discoverable device accepts the pairing
request, the two devices complete a bonding process where they exchange security keys. The
devices cache these keys for later use. After the pairing and bonding processes are complete,

8
Chapter 2

the two devices exchange information. When the session is complete, the device that initiated
the pairing request releases the channel that had linked it to the discoverable device. The two
devices remain bonded so that they can reconnect automatically during a future session as long
as they’re in range of each other and neither device has removed the bond.

The implementation of this connection is shown in 3.7.1.

2.2.2 TCP/IP protocols

TCP/IP, or Transmission Control Protocol/Internet Protocol, is a set of communication proto-

cols used to interconnect network devices on the internet. TCP/IP can also be used as com-
munication protocols suite in a private computer network, which is the intended use in this
project.

TCP defines how applications create communication channels, manages how a message
should be broken down into packets to be transmitted and received. While IP defines how to
address and route packets for delivery.

A concept that is commonly referred to in this protocol is sockets. A socket is one endpoint
of a two-way communication link between two programs running on the network. A socket is
bound to a port number so that the TCP layer can identify the application that data is destined
to be sent to. An endpoint is a combination of an IP address and a port number.

For two devices to transfer data between each other using TCP/IP, they must run two slightly
different programs where one must act as a server, which accepts incoming connection re-
quests. While the other device acts as a client that requests a connection from a server.

For the two programs to communicate using TCP/IP, each program must create a socket,
and those sockets must be connected. Once such a connection is made, it takes place using
input streams and output streams. Each program has its own input and output stream. Data
written by one program to its output stream is transmitted to the other. There, it enters the input
stream of the program at the other end of the network connection. When that program reads
data from its input stream, it is receiving the data that was transmitted to it over the network.

The TCP handshake

TCP uses a three-way handshake to establish a reliable connection. The connection is full
duplex, and both sides synchronize (SYN) and acknowledge (ACK) each other. The exchange
of these four flags is performed in three steps: SYN, SYN-ACK, and ACK as shown in Figure
2.2.

9
Chapter 2

Figure 2.2: TCP 3 way handshake

The client chooses an initial sequence number, set in the first SYN packet. The server
also chooses its own initial sequence number, set in the SYN/ACK packet shown in Figure
2.2. Each side acknowledges each other’s sequence number by incrementing it; this is the
acknowledgement number. The use of sequence and acknowledgement numbers allows both
sides to detect missing or out-of-order segments. Once a connection is established, ACKs
typically follow for each segment. The connection will eventually end with a RST (reset or tear
down the connection) or FIN (gracefully end the connection).

Implementation of This connection is shown in 3.7.2.

2.2.3 UART and USB

UART or Universal Asynchronous Receiver/Transmitter is not only a communication protocol

like SPI and I2C, but also a physical circuit in a microcontroller, or a stand-alone IC. A UART’s
main purpose is to transmit and receive serial data.

UART only uses two wires to transmit data between devices. It takes bytes of data and
transmits the individual bits sequentially. At the destination, a second UART re-assembles the
bits into complete bytes. Each UART contains a shift register, which is the fundamental method
of conversion between serial and parallel forms. Data flows from the Tx pin of the transmitting
UART to the Rx pin of the receiving UART:

Figure 2.3: UART data transmission diagram [1]

10
Chapter 2

UART transmitted data is organized into packets. Each packet contains 1 start bit, 5 to 9
data bits (depending on the UART), an optional parity bit, and 1 or 2 stop bits:

Figure 2.4: UART packet [2]

UARTs exchanges data asynchronously, this means that there is no additional clock wire.
However, when the receiving UART detects a start bit, it starts to read the incoming bits at a
specific frequency called the baud rate. Baud rate is a measure of the speed of data transfer,
expressed in bits per second (bps). Both UARTs must operate at about the same baud rate. The
baud rate between the transmitting and receiving UARTs can only differ by about 10% before
the timing of bits gets too far off.

Android devices and the android framework already support UART communication. How-
ever, the configuration of a UART chip (parity bit, data bits . . . etc) is different from one man-
ufacturer to another. So we the application has to detect and support many devices or leave the
configuration to the user. To avoid that as maximum as possible we will be using alongside
Android’s USB APIs a library called USBserial [3] which support most of the commonly used
chips, which are as follows:

• CP210X devices Default: 9600,8,1,None,flow off

• CDC devices Default 115200,8,1,None,flow off

• FTDI devices Default: 9600,8,1,None,flow off

• PL2303 devices Default 9600,8,1,None,flow off

• CH34x devices Default 9600,8,1,None,flow off

• CP2130 SPI-USB

Implementation of This connection is shown in details in 3.7.3.

11
Chapter 2

2.3 Design patterns

In this section we will cover some programming concepts and design patterns that were used
during the development of this project.

2.3.1 Interfaces

In Java an interface is a reference type that is similar to class. However, it only provides
a collection of abstract methods. Also, along with abstract methods, an interface may also
contain constants, default methods, static methods, and nested types. Method bodies exist only
for default methods and static methods.

An interface is different from class in the following way :

• You cannot instantiate an interface.

• An interface does not contain any constructors.

• All the methods in an interface are abstract.

• An interface cannot contain instance fields. The only fields that can appear in an interface
must be declared both static and final.

• An interface is not extended by a class; it is implemented by a class.

• A class can implement an interface, thereby inheriting the abstract methods of the inter-
face.

• An interface can extend multiple interfaces.

Interface-based programming, also known as interface-based architecture, is an architec-

tural pattern for implementing modular programming. In fact, interfaces were used quite often
in this project to achieve modularity and to insure the project scalability.

2.3.2 Singleton pattern

In software engineering, the singleton pattern is a software design pattern that restricts the
instantiation of a class to one “single” instance. This is useful when exactly one object is needed
to coordinate actions across the system. The term comes from the mathematical concept of a
singleton.

A simple way of implementing this pattern in Java is as follows :

• Make constructor as private.

12
 Chapter 2

• Make a private static instance (class-member) of this singleton class. But, DO NOT
instantiate it.

• Write a static method called getInstance() that checks the static instance member
for null and creates the instance. At last, it returns an object of the singleton class.

• Use the synchronized keyword to insure thread safety.

• The singleton class is different from a normal Java class in terms of instantiation. For a
normal class, we use a constructor, whereas for singleton class we use the getInstance()
method.

1 public class SingletonClass {

private static SingletonClass SINGLE_INSTANCE = null;
3 private SingletonClass() {}
public static SingletonClass getInstance() {
5 if (SINGLE_INSTANCE == null) {
synchronized(SingletonClass.class) {
7 SINGLE_INSTANCE = new SingletonClass();
}
9 }
return SINGLE_INSTANCE;
11 }
}

This pattern was used in this project with connections classes (3.7), in order to instantiate
a connection once and be able to use the same instance throughout the different parts of the
application.

13
Chapter 3

Android app implementation

In this chapter, we will see how the most important features of the android application were
implemented. To do so, we need to specify the features required from the android app. These
features give the ability for the user to :

• Create custom commands to be sent.

• Choose from a set of modular I/O components and customize them.

• Link those components to commands.

• Ability to create GUIs by positioning those components.

• Ability to save the GUIs in projects.

• Use different types of connections to send and receive commands.

We will start by showing the user interface and how the different screens are interconnected.
Then, we will see how the applications’database is structured. After that we move on to see
how users can create projects, custom commands and modular components. Then, we see how
GUIs can be created and loaded from the database. And finally, we will discuss the way the
different connections were implemented and managed.

3.1 User interface

In this section we will go over the user interface presented by the mobile application. The
following diagram shows how the different app screens are interconnected and gives a short
description about each screen is given.

14
Chapter 3

Figure 3.1: User interface structure

For optimal performance we choose to have a single activity (2.1.1) called ‘MainActivity’
that serves as the host for different fragments, and each fragment represents a screen or a user
interface which the user can navigate to. A screenshot for each individual fragment and its
corresponding class name is shown bellow:

15
Chapter 3

(a) Home Fragment (b) Connection Fragment (c) Terminal Fragment

(d) Projects Fragment (e) New Project Fragment (f) Commands Fragment

Figure 3.2: Screenshots of fragments

16
Chapter 3

(g) GUI Fragment (h) GUI Editor Fragment (i) Catalog Fragment

(j) Widget Builder Fragment

Figure 3.2: Screenshots of fragments

Each fragment has its own Java class and its own layout file.

17
Chapter 3

3.2 Database structure

One of the first things that must be considered when building an android application like the one
of this project is the structure of the database. The structure of the database is very important
as it can effect the performance of the application significantly. Also, if it is well-structured
development will be much easier and scalable in the future.

Our android application uses a local SQL database to store user data that must be loaded
each time the app is opened. An example of this data can be the set of projects created during
the use of the app.

The app needs to store the following :

• User projects : a user project is the top entity that represents the real project that
the user is working on, for example: Stepper motor controller, home automation GUI ...
etc.
• Commands : a command is the data that gets sent whenever the user interacts with the
GUI that he created. commands must be stored in a database because they are defined
by the user.
• Widgets attributes : a widget is a modular component used to make a GUI. Only
the attributes of a widget are stored in the database (coordinates in the screen, color, text,
size . . . etc) as they are the characteristics that allow us to reconstruct the widget later.

Those elements of the database are called entities and are linked to each other in the fol-
lowing manner:

Figure 3.3: database entities relation

3.3 User defined Projects

Users can create many Projects within the app. Each Project contains a set of commands
and a Graphical User Interface. The list of projects can be found by navigating to Projects
Fragment (See 3.2d).

18
 Chapter 3

In order to store Projects in a local SQL database we need to represent it as a Java object
as follows:
public class Project
2 int id;
String name;
4 public Project(String name);
public int getId() ;
6 public void setId(int id);
public String getName();
8 public void setName(String name);
}

The name is defined by the user, while the id is auto-generated.

The id is used by other entities as a foreign key in order to identify which project they
belong to.

In the next sections we will see how commands and widgets are represented and stored in
the database.

3.4 Custom commands

Commands are the most crucial part in this project, they are the link between the android
app and the micro-controller. Commands are sent when a user interacts with the created GUI:
Button clicks, Knob turns . . . etc.

Each Command has a name and a set of parameters. The command name is used as
a keyword that identifies the command. While Parameters hold the specific data to be sent.
Those parameters can be divided into two types:

1. Variable Parameters: these must be used with components/widgets that produce a range
of values, such as a knob or a slider. The order and the number of variable parameters in
a command is dictated by the component which is linked to a command to ensure that all
variable parameters in a command are used correctly.

2. Constant Parameters: this type of parameters has a fixed value that is defined by the
user. They are generally used as constants to indicate different actions when there are
multiple commands that use the same name, but have different parameters. An example
of that is shown in 5.3.

19
 Chapter 3

In order to store commands in the app’s database, we represent them as Java objects as
follows:
1 public class Command {

3 int id;
int projectID;
5 String name;
Parameter[] params;
7
public Command(int projectID, String name, Parameter[] params) ;
9
//setters and getters ...
11 }

• id : is a unique identifier for each command. It is used to link widgets to a specific

command.

• projectID : used to determine which project does the current command belong to.

• name : is a keyword for identifying the command for example: MOVE, TURN, SWITCH
. . . etc. Note that multiple commands can have same name but different parameters.

• Params[] :An array of Parameter objects. Each Parameter object holds details
about the actual parameter : its data type, default value, and whether it is variable or
constant.

Also, we decided to represent parameters in a class because that allows providing more details
about their nature and allows easy recognition of variable parameters.

In order to load those Parameter objects to/from the database, we convert them first to a
JSON string using the GSON library [11] and store them in a single row of a command entity.

A parameter is represented in the following way :

1 public class Parameter {
String defValue;
3 String dataType;
int isVariable;
5
//setters and getters ...
7 }

Listing 3.1: Parameter class

20
Chapter 3

Format of transmitted commands:

When a user performs an action on the GUI, a command will be generated and sent in the
following format:

>CommandName,Parameter1,Parameter2,. . . ,ParameterN<

• ‘>’ indicates the start of a new command.

• ‘<’ indicates the end of a command.
• Parameters and command name are separated by a comma.
• Parameters are stored in a string regardless their datatype.

The micro-controller receives the command with the above format and with the aid of
the Helper library it will know how to interpret those symbols and be able to reconstruct a
command object from it.

Also, the different parameters are converted and sent in string format even if they hold num-
bers. That method is commonly used in data transmission where numbers are being transmitted
over a communications link. These parameters can later be converted correctly to their original
data type. In fact, the conversion is taken care of by the Helper library which is discussed in
details in Section 4.2.

3.5 Widgets (modular components)

A widget is any modular component, such as a Button, slider, knob, joystick and even a sensor
block. The android application provides a verity of widgets that the user can choose from and
add to the GUI that he intends to build. The user can add a new widget to his GUI from the
CatalogFragment (see fig 3.2i). In this section we will see how those built-in widgets were
implemented and what design decisions were taken to insure the projects’ salability.

To implement these widgets and to keep the project scalable, such that new widgets can be
easily added to the project at any given time. The following decisions were taken to create any
new widget class :

1. Each widget class must extend the View class (see 2.1.3). This is in order for the
system to be able to draw it on the screen, listen to its touch and click events and so that
the widget can be a child of any container or layout (see 2.1.4).
2. In order to store the widgets created by the user in the app’s database and be able to
load them in a later time we need another class called ‘WidgetAttributes’. This
class has all the attributes that are needed to recreate the widget. The attributes : posX,
posY, linkedCommandID, ProjectID and viewType are always used while
others such as: text, color, size, orientation . . . etc are optional.

21
 Chapter 3

3. Each widget class must have an instance of ‘WidgetAttributes’. The instance

should be updated each time the user chooses to change some attributes of the widget.
4. A widget class should also implement a special interface called ‘AbstractWidget’.
That interface serves as the blueprint for any widget class. It ensures that all the com-
mon functionalities af a widget are implemented. It also expands the functionalities of a
widget beyond what a normal View can do, for example it allows widgets to be linked to
commands and be able to send them whenever a specific event happens.
5. During development of the project, any widget instance must be referenced to ‘AbstractWidget’
instead of its original class. That is to take advantage of polymorphism to produce code
that is modular and scalable. For instance, to add a new widget class to the project we
only need to implement the functions specified by the AbstractWidget interface and
no additional code or alternations on other parts of the project are required, the new wid-
get will work exactly as intended because it implements all the functionalities required
by the AbstractWidget interface.

Here is how a widget class should be declared:

1 class ExampleWidget extends View implements AbstractWidget{
// widget logic ...
3 }

Here is the code snippet for AbstractWidget’ interface:

1 public interface AbstractWidget {

3 void setAttributes(WidgetAttributes attrs);

WidgetAttributes geFinaltAttrs();
5
void setOnCommandExecutedListener(OnCommandExecutedListener
listener);
7 interface OnCommandExecutedListener {
void onCommandExecuted(Command command);
9 }

11 void setPrimaryColor(int color);

void setSecondaryColor(int color);
13 void setPrimaryText(String text);
void setOrientation(int angle);
15 void setInPreviewMode();
void setSize(int mode, int size);
17
}

Listing 3.2: AbstractWidget interface

Here is a snippet of the ‘WidgetAttributes’ class:

22
 Chapter 3

public class WidgetAttributes {

2
int id;
4 int projectID;
int linkedCommandID;
6
//attrs
8 float posX;
float posY;
10 int viewType;
String primaryText;
12 int primaryColor;
int orientation;
14 int size;
int maxProgress;
16 int minProgress;
boolean showProgress;
18
//setters and getters ...
20 }

User created widgets

Users can create customized versions of the built-in widgets. That is done by selecting a widget
from the WidgetsCatalog (3.2i). When a widget is selected, another dialog called WidgetBuilder
(3.2j) will be opened to allow customizing and linking the widgets to commands. The Widget-
Builder dialog uses the setters implied by the interface in Listing 3.2 to change the look of the
default widget by setting colors, text, size . . . etc.

When the save button is clicked an instance of WidgetAttributes will be generated

based on : the chosen widget, chosen attributes and command specified by the user from the
dialog. That instance is passed to GUI Editor fragment 3.2h and the Widget Builder dialog is
closed. Now the desired widget can be positioned to make a GUI.

Recreating Widgets from their attributes

Before we see how widgets can be positioned inside a layout, we should first see how they
can be recreated from a WidgetAttributes instance. To do so, we created a function
called getWidgetFromAttrs(attrs). The desired attributes are passed to this function as a
parameter and a View instance representing the widget will be returned. That instance can be
later included inside a container/layout and be positioned by the user in the desired manner.

23
Chapter 3

Arranging widgets

As stated earlier, widgets added from Widget Builder dialog are passed to GUI Editor fragment
as instances of WidgetAttributes class. Then, using the getWidgetFromAttrs(attrs)
function the widget is recreated and placed inside a RelativeLayout container (2.1.4) using the
addView() function.

Since, widgets extend the View class, we can use setOnTouchListener() on all the added
widgets to get a callback when the user performs a drag movement on one of them. That allows
implementing the functionality to position and arrange those widgets.

The onTouch(View view, MotionEvent event)() function is the callback for the touch
listener, so we need to override it in the GUI Editor fragment class. Two parameters are
provided by the callback, the view/widget that the user selected and a MotionEvent. The Mo-
tionEvent instance provides the coordinates of the user touch by using it getters event.getX()
and event.getY(). Those coordinates are used to position the widget using view.setX() and
view.setY() functions.

After all the widgets are arranged in the desired manner, we loop through all of them and
using the getFinalAttrs() from interface 3.2 to extract the attributes and coordinates of each
widget, and we save those attributes to the local database.

24
Chapter 3

3.6 Graphical user interfaces (GUIs)

After we have seen how the built-in modular components (widgets) are implemented and how
the user can create customized versions of them and position them, now we see the implemen-
tation behind how users can:

• Load widgets from a database to a Graphical user interface.

• And what happens when the user interacts with these widgets to send commands.

Loading GUIs from the database

A Graphical user interface is basically a combination of widgets that are positioned inside a
certain container or a layout. Since each project has a single GUI in it, we don’t need to repre-
sent it as a class and save it in a database. Instead, each time we want to load a GUI of a certain
project we just load its related widgets by doing an SQL Query in widgetAttributes
table for widgets having the same projectID as the current opened project.

The SQL Query will return a list of widgetAttributes instances that can be used to recreate
the actual widgets. We use the function getWidgetFromAttrs(attrs) on each item of that list
to get another list that contains the desired.

In order to position those widgets on the screen we use a layout called RelativeLayout
(see 2.1.4). This layout allows positioning its children or Views relative to each other or by
specifying their coordinates in pixels. We will use the second method which is positioning
by coordinates. The coordinates of a widget are automatically set from the widgetAttributes
that we retrieved earlier, so the only thing that we need to do is to use the addView(view)
function provided by the layout to place the loaded widgets as its children views and they will
be arranged automatically.

After the GUI is loaded, the user can interact with its widgets to send and receive com-
mands. The following flowchart summarizes the above :

25
Chapter 3

Interacting with widgets and sending commands

Each time a widget is generated and returned by getWidgetFromAttrs(attrs) function and

is loaded to the RelativeLayout container, it also subscribes to a specific callback using se-
tOnCommandExecutedListener() specified by the AbstarctWidget interface (3.2). The same
callback is used for all the loaded widgets. After the GUI finishes loading, whenever the user
interacts with a widget, it generates the command to be executed and triggers the callback On-
CommandExecuted(Command command). That callback is later going to be used to send
the command to a micro-controller via a connection.

In the next section we will discuss how the different types of connections were implemented
and how they can be managed.

26
 Chapter 3

3.7 Implementing & Managing connections

In this section we will see how connection classes were implemented and discuss the decisions
taken during their development.

Before we see how any type of connection is implemented, we first need to define an ab-
straction that will serve as the blueprint of any connection class, and to also ensures future
project scalability. To do so we create an interface called BaseConnection. This interface
lists all the mandatory functions that any type of connection class needs :

• A function for sending data : write(data)

• A callback for receiving data and its setter function : setDataInCallback(callback)
• A function for canceling the current connection : disconnect()
• An enum for representing the type of connection

Here is a snippet of the BaseConnection interface:

public interface BaseConnection {

2
void write(String data);
4 void setDataInCallback(DataInCallback callback);
void disconnect();
6 enum ConnectionType {BT,USB, TCP,NO_CONNECTION}

8 interface DataInCallback {
void onReceive(String message);
10 }
}

In the different parts of this project, any connection instance is referenced to the above
interface like the following example :

1 BaseConnection connection = BluetoothConnection.getInstance();

This is very powerful since all connections are represented using a single type. That allows
us to avoid using conditional statements everywhere, for example to decide which connection
to use to send the data or to receive it.

This also allows us to add support for new connection types (like GSM or Ethernet) without
having to modify the whole code of the project. A new class that implements the above interface
and some small modifications to indicate a new type of connection is added is sufficient.

Now we will see how the different types of connections were implemented.

27
 Chapter 3

3.7.1 Bluetooth Connection

1- Permissions:

In order to use Bluetooth functionalities, some permissions must be declared in the manifest
file in the following manner (see 2.1.5) :
1 <manifest ... >
<uses-permission android:name="android.permission.BLUETOOTH" />
3 <uses-permission android:name="android.permission.BLUETOOTH_ADMIN" />
...
5 </manifest>

2- Initiating the Bluetooth adapter:

A Bluetooth adapter represents the local device Bluetooth adapter. The BluetoothAdapter lets
you perform fundamental Bluetooth tasks. Since there is only a single physical adapter in the
device, an instance of BluetoothAdapter is acquired using the getDefaultAdapter() method :
1 BluetoothAdapter bluetoothAdapter = BluetoothAdapter.getDefaultAdapter();

3- Enabling Bluetooth:

The bleutoothAdapter instance allows verifying if BT is enabled. If not we display a dialog

requesting user permission to enable Bluetooth using startActivityForResult() (see 2.1.7).
1 if (!bluetoothAdapter.isEnabled()) {
Intent enableBtIntent = new Intent(BluetoothAdapter.
ACTION_REQUEST_ENABLE);
3 startActivityForResult(enableBtIntent, REQUEST_ENABLE_BT);
}

4- Finding devices

We can either start a discovery service for finding all nearby discoverable Bluetooth devices , or
we can just query the list of already paired devices. To query the list of already paired devices
we use the following code :

28
 Chapter 3

1 if (!bluetoothAdapter.isEnabled()) {
Set<BluetoothDevice> pairedDevices = bluetoothAdapter.getBondedDevices
();
3 if (pairedDevices.size() > 0) {
// There are paired devices. Get the name and address of each paired
device.
5 for (BluetoothDevice device : pairedDevices) {
String deviceName = device.getName();
7 String deviceHardwareAddress = device.getAddress(); // MAC address
}
9 }
}

To start discovering devices, we simply call startDiscovery(). The process is asynchronous

and returns a boolean value indicating whether discovery has successfully started. The discov-
ery process usually involves an inquiry scan of about 12 seconds, followed by a page scan of
each device found to retrieve its Bluetooth name.

In order to receive information about each device discovered, the application must regis-
ter a BroadcastReceiver (see 2.1.6) for the ACTION FOUND intent. The system broadcasts
this intent for each device. The intent contains the extra fields EXTRA DEVICE and EX-
TRA CLASS, which in turn contain a BluetoothDevice and a BluetoothClass, respectively.
The following code snippet shows how we can register to handle the broadcast when devices
are discovered:

protected void onCreate(Bundle savedInstanceState) {

2 // Register for broadcasts when a device is discovered.
IntentFilter filter = new IntentFilter(BluetoothDevice.ACTION_FOUND);
4 registerReceiver(receiver, filter);
}
6
// Create a BroadcastReceiver for ACTION_FOUND.
8 private final BroadcastReceiver receiver = new BroadcastReceiver() {
public void onReceive(Context context, Intent intent) {
10 String action = intent.getAction();
if (BluetoothDevice.ACTION_FOUND.equals(action)) {
12 // Discovery has found a device. Get the BluetoothDevice
// object and its info from the Intent.
14 BluetoothDevice device = intent.getParcelableExtra(
BluetoothDevice.EXTRA_DEVICE);
String deviceName = device.getName();
16 String deviceHardwareAddress = device.getAddress(); // MAC
address
}}};
18
protected void onDestroy() {
20 super.onDestroy();
unregisterReceiver(receiver);
22 }

29
 Chapter 3

5- Connecting as a server

After discovering nearby devices and selecting the device that we want to connect to, we es-
tablish a connection between the two devices, the phone acts as a server by holding an open
BluetoothServerSocket. The purpose of the server socket is to listen for incoming connection
requests and provide a connected BluetoothSocket after a request is accepted. When the Blue-
toothSocket is acquired from the BluetoothServerSocket, the BluetoothServerSocket will be
closed, since we need only a single connection.

So we need to go through the following steps :

1. Get a BluetoothServerSocket by calling listenUsingRfcommWithServiceRecord()

2. Start listening for connection requests by calling accept(). The accept() is a blocking
call and must be called within a thead.

3. call serverSocket.close () after connecting and aquiring a BluetoothSocket.

Here is the code snippet that performs the above steps :

private class AcceptThread extends Thread {
2 private final BluetoothServerSocket mmServerSocket;

4 public AcceptThread() {
// Use a temporary object that is later assigned to mmServerSocket
6 // because mmServerSocket is final.
BluetoothServerSocket tmp = null;
8 try {
// MY_UUID is the app’s UUID string, also used by the client code.
10 tmp = bluetoothAdapter.listenUsingRfcommWithServiceRecord(NAME,
MY_UUID);
} catch (IOException e) {
12 Log.e(TAG, "Socket’s listen() method failed", e);
}
14 mmServerSocket = tmp;
}
16
public void run() {
18 BluetoothSocket socket = null;
// Keep listening until exception occurs or a socket is returned.
20 while (true) {
try {
22 socket = mmServerSocket.accept();
} catch (IOException e) {
24 Log.e(TAG, "Socket’s accept() method failed", e);
break;
26 }

28 if (socket != null) {
// A connection was accepted. Perform work associated with
30 // the connection in a separate thread.
manageMyConnectedSocket(socket);
32 mmServerSocket.close();
break;

30
 Chapter 3

34 }
}
36 }

38 // Closes the connect socket and causes the thread to finish.

public void cancel() {
40 try {
mmServerSocket.close();
42 } catch (IOException e) {
Log.e(TAG, "Could not close the connect socket", e);
44 }
}
46 }

Finally, After a connection is established. We can use the resulting BluetoothSocket instance to
send and receive data.

3.7.2 TCP/IP connection

In order to implement a connection using TCP/IP in our application we will use the JAVA
networking APIs.

1- Permissions

Before implementing the TCP/IP connection in java, a permission has to be declared in the
manifest file as follows :
<manifest>
2 <application>
...
4 </application>
<uses-permission android:name="android.permission.INTERNET" />
6 </manifest>

2- Connecting as a client

In contrast to a Bluetooth connection, this time the android device will act as a client. We chose
that because if we use the mobile phone as a server then its IP address will change every time
the WI-FI adapter is turned ON. This means that the IP in the code of the client device has to be
changed whenever the mobile phone changes its IP. So, as a solution, the user creates a server
with a constant/static IP and a certain port and use the phone as a client that connects to it.

The program for a TCP client is very straightforward. We only need to create a socket and
pass to it the IP address and port number opened by the server. The client automatically

31
 Chapter 3

sends a resquest to the server to connects and if the request is not accepted an Exception will
be returned and must it be handled correctly. The following java code is used to achieve that:
try{
2 Socket socket = new Socket("IP_ADDRESS", PORT_NBR);//send request
to a server woth IP_ADDRESS and PORT_NBR
}
4 catch(IOexception e){
//handle exception...
6 }

To send data to the server, get the OutputStream object from the socket first and use the
write(bytes) to send the data:
OutputStream output = socket.getOutputStream();
2 //...
output.write(bytes);

3.7.3 UART connection

We choose UART instead of standard USB since its far simpler and can be implemented with
most microcontrollers. Also, as mentioned earlier we will be using the USBserial library [3]
to provide automatically the UART protocol configuration for different UART chips.

1- Permissions:

In order to be able to use the USB port we need to declare a permission in the manifest file.
Also to be able to tell the app if a USB device is connected or disconnected we need to declare
a Broadcast reciever (2.1.6) with two intent filters. This is how we do that in the manifest file :

1 <manifest ... >

<uses-permission android:name="com.google.android.things.permission.
USE_PERIPHERAL_IO" />
3 ...
<application
5 ...
<receiver
7 android:name=".Connections.ConnectionDisconnectReceiver"
android:enabled="true">
9 <intent-filter>
<action android:name="android.bluetooth.device.action.
ACL_DISCONNECTED" />
11 <action android:name="android.hardware.usb.action.
USB_DEVICE_DETACHED" />
<action android:name="android.hardware.usb.action.
USB_DEVICE_ATTACHED"/>
13 </intent-filter>

32
 Chapter 3

</receiver>
15 </application>
</manifest>

Discovering device connected to USB port

To discover devices connected to the USB port first we get an instance of UsbManager. The
UsbManager can be used to get the list of connected devices using the getDeviceList() which
returns a HashMap of key-value pairs, the key is the name of the port name to which the USB
device is connected and the value is a UsbDevice instance that can be used to start a serial
connection.
//get UsbManager instance
2 UsbManager usbManager = (UsbManager) context.getSystemService(Context.
USB_SERVICE);
//load list of devices
4 HashMap<String, UsbDevice> usbDevices = usbManager.getDeviceList();

Most generally there will be only a single device attached to the USB port. So, we take the
first device in that list and use the USBserial library to create an instance of UsbSerialDevice.
That instance can later be used to send and receive serial data.
1 UsbSerialDevice serial = UsbSerialDevice.createUsbSerialDevice(device,
usbConnection);

To start a serial connection we use the followed code :

1 serial.open();
serial.setBaudRate(115200);
3 serial.setDataBits(UsbSerialInterface.DATA_BITS_8);
serial.setParity(UsbSerialInterface.PARITY_ODD);

Here we have set the baud rate to 115200 bps, 8 data bits and the parity bit to be odd. In the
application the user can specify these parameters when starting a new UART connection.

In order to send serial data we use usbSerialDevice.write(bytes) . And to re-

ceive data, we subscribe to a callback using usbSerialDevice.read(serialCallback)
that triggers each time some bytes are received, this requires no pooling like in other connec-
tions.

33
Chapter 3

Remark:

An important thing to note is that all the connection classes above were implemented using the
singleton pattern 2.3.2. This is very important, since a connection should be instantiated only
once then is used multiple times within the different parts of the application.

34
Chapter 4

The helper library implementation

In this chapter we will discuss the implementation of the second half of the project which is the
Helper Library. This library is intended to provide developers with an efficient mean to parse
the commands received from the android device, a convenient API to extract their parameters
and convert them into the desired data types, and it provides a callback that triggers when a
command is fully received and parsed.

The programming language used to develop this library is C++. It was chosen because it
is very light on micro-controllers, allows custom memory management and, it is also used by
most developers.

This library is general enough to work on all types of microcontrollers and even computers
which extends further the use of this project !

4.1 Command and parameters representation

Command class

First we will see how a command and its parameters are represented in C++. A command
is represented as an object, mainly to keep code compact and cleaner by giving access to both
command name, parameters and params size.

The command class has three private members, command name which is stored in a
CString, an array of Parameter objects and params size that holds the number of param-
eters. These private members can only be accessed from a special class called ProtoHolper
which we will discuss later. Public getters are also provided to give the user access to these
entities.

It is also important to note that this class implements its own copy constructor, deconstructor
and assignment operator. This is because manual memory management is required for the two
first class members.

35
 Chapter 4

This is a snippet of the command class:

2 #define MAX_PARAMS_SIZE 15
#include "Parameter.h"
4
class ProtoHelper;
6 class Command
{
8 protected:
Parameter* params[MAX_PARAMS_SIZE];
10 char* command_name = nullptr;
int params_size = 0;
12
public:
14 Command();
Command(const Command& command);
16 ˜Command();
Command& operator=(const Command&);
18 Parameter* getParam(uint8_t index);
const char* getCommandName();
20 int getParamsSize();

22 private:
void setCommandName(char* name, uint8_t size);
24 void addParam(Parameter* param);

26 friend class ProtoHelper;

};

Parameter class

Parameters have their own class too. This is mainly to provide a convenient way to convert
the parsed parameters to the desired datatype by providing many getter functions that return
different data types.

The Parameter class has two members, a CString to hold the extracted parameter
and an integer variable that holds the number of characters of the previous CString. This is the
blueprint for that class:

#include<stdint.h>
2
class Parameter {
4
private:
6 char* data = nullptr;
uint8_t size;
8
public:
10 Parameter(char* bytes, uint8_t size);
˜Parameter();
12 Parameter(const Parameter&);

36
 Chapter 4

Parameter& operator=(const Parameter& src);

14
float getFloatValue();
16 char* getStringValue();
int getIntValue();
18 double getDoubleValue();
long getLongValue();
20
};

The different getters use functions from the standard library ‘stdlib’ for a safe and effi-
cient string to other datatype conversion. These getters are implemented as follows:

1 int Parameter::getIntValue(){
char* c = data;
3 return atoi(c);
}
5 float Parameter::getFloatValue(){
char* c = data;
7 return atof(c);
}
9 long Parameter::getLongValue(){
char* c = data;
11 return atol(c);
}
13 double Parameter::getDoubleValue(){
char* c = data;
15 return atof(c);
}

The data variable is buffered, because the converter functions are mutating and will cause
memory problems if data is passed directly as a parameter.

4.2 ProtoHelper class

The following singleton class is responsible for acquiring received bytes, extracting commands
parameters and triggering a callback whenever a command is fully received and parsed.

This class has the following 5 members:

• an instance of the class itself to implement the singleton pattern

• a pointer to the callback function
• an instance of the current command that is being parsed
• a variable that holds the current state of a state machine, the state machine will be dis-
cussed later in details

37
 Chapter 4

This is s snippet of the class:

#include "Command.h"
2
class ProtoHelper
4 {
private:
6 static ProtoHelper* instance_;
void (*callback)(Command* command);
8 enum state {BEGIN ,OPCODE_CONCAT , PARAM_CONCAT};
state current_state = BEGIN;
10 Command* current_command;
public:
12
//Singletons can not be cloneable
14 ProtoHelper(ProtoHelper& other) = delete;

16 //Singletons can not be assigned

void operator=(const ProtoHelper&) = delete;
18
static ProtoHelper* getInstance();
20 void LoadNewByte(char byte);
void setOnCommandRecievedCallabck(void (*callback)(
Command* command));
22 ˜ProtoHelper();

24 protected:
ProtoHelper();
26 };

Parsing the received commands

We have already seen in 3.4 the format of sent commands. In order to parse these commands
and separate the different parameters efficiently, we use the state machine illustrated in Listing
4.1. That state machine takes one of the following decisions whenever a byte is received and
loaded using the LoadNewByte(byte) :

• wait for new command

• concatenate command name

• concatenate parameters

• save parameters

• trigger a callback if command is fully received

This method is very powerful, since there is no need to store the received command in a
string and then parse it. Instead, the command and its parameters are extracted at the same time
as they are received which saves RAM and Processing time.

38
Chapter 4

This is a diagram of the state machine:

charIn != ‘>’
charIn != ‘,’
charIn != ‘,’

charIn == ‘>’ charIn == ‘,’ concat

start wait command concat name
parameter

charIn != ‘<’ charIn == ‘,’

save command save

& notify user parameter

charIn == ‘<’

Figure 4.1: Parsing command state machine

This state machine is implemented within the loadNewByte(byte) function. That function
should be used in the beginning of the big loop of the program that runs in the microcontroller
and should be called as frequently as possible.

39
Chapter 5

Results, Examples and future

improvements

In this chapter we will be testing the features of this project and showing the final results. First
we will start by testing if the different types of connections provided by the android app. Then,
we build an example GUI and check if it sends commands correctly and at the same time test
the Helper library if it can parse the incoming commands correctly. And finally, we will see a
real example of using this project by creating a GUI that controls the speed and angle of two
stepper motors.

In order to test the project we used a development board called ’NodeMCU V3’ alongside
with an android device (OnePlus 6) running android 10.0.

We chose the NodeMCU V3 board because it is based on the ESP8266 microcontroller,

which comes with a complete and self-contained Wi-Fi networking capabilities and support
for UART which allows us to test all connections types provided by the android app. And
programming it will be simple because the board is supported by the Arduino platform.

ESP8266 NodeMCU V3

40
 Chapter 5

5.1 Testing connections

The android app provides a dedicated Terminal to test connections. So in order to test any
connection we will send a message and wait the microcontroller to send it back. If it is not sent
back then we know we have a problem.

TCP/IP connection test:

The first connection we will be testing is going to be a WI-FI connection using the TCP/IP
protocol. The android device will act as a client and the microcontroller will be the server. To
do so, we go through the following steps:

1. We connect both devices to the same WI-FI access point which is going to be created
from the mobile phone.

2. We upload the code in listing 5.1 to our Microcontroller. Basically this code is going to
establish a Wi-Fi connection by connecting to the access point of the mobile phone. Then
it initiates a TCP server socket on port 1027 and IP address 192.168.43.244. Then, it
keeps waiting for a client to connect. If a client connects, the MCU will send back any
message it receives from the client.

3. After uploading the previous code, we keep the dev-board connected to power, and we
open the android application and navigate to Connection Fragment 3.2b.

4. We click the TCP/IP button and enter the port 1027 and IP address 192.168.43.244
and click connect (see fig 5.1a). We can see from the screenshot in fig 5.1b that the
connection was successful.

5. Now we navigate to Terminal fragment 3.2c and send the messages : “TCP test”, “TCP
test 2”, “TCP test 3”.

6. We can see from fig 5.1c that the messages were sent and received back successfully !

#include <ESP8266WiFi.h>
2 #include <WiFiClient.h>

4 // Hardcode WiFi parameters as this isn’t going to be moving around.

const char* ssid = "Oneplus 6";
6 const char* password = "Pass1234567";

8 // Start a TCP Server on port 1027

static WiFiServer server(1027);
10 static WiFiClient client ;

12
void setup() {
14 Serial.begin(115200);
WiFi.begin(ssid,password);
16 Serial.println("");
//Wait for connection

41
 Chapter 5

18 while(WiFi.status() != WL_CONNECTED) {
delay(500);
20 Serial.print(".");
}
22 Serial.print("Connected to "); Serial.println(ssid);
Serial.print("IP Address: "); Serial.println(WiFi.localIP());
24
// Start the TCP server
26 server.begin();
}
28
void loop() {
30 if (!client.connected()) {
// try to connect to a new client
32 client = server.available();
} else {
34 // read data from the connected client
if (client.available() > 0) {
36 client.write(client.read());
}
38 }
}
40

Listing 5.1: Code for testing TCP/IP (microcontroller side)

(a) Starting TCP/IP connection (b) Connection successful (c) Test Messages

Figure 5.1: TCP/IP connection results

42
Chapter 5

Bluetooth connection test:

Now we test the Bluetooth connection. We used the HC-06 Bluetooth module with the micro-
controller.

Figure 5.2: HC-06 Bluetooth module interface with NodeMCU

The HC-06 is a Bluetooth module designed for establishing short range wireless data com-
munication between two microcontrollers or systems. The module works on Bluetooth 2.0
communication protocol and, it can only act as a slave device. The communication with this
HC-06 module is done through UART interface, that’s why we can connect it directly to the
RX and TX pins of our micro-controller. Also, by default this module uses a baud-rate of 9600
bps.

Now we go through the following steps to test the Bluetooth connection:

1. Attach the module to the microcontroller as shown in fig 5.2.

2. Upload the code in listing 5.2 to the microcontroller. The code is very simple, it initiates
a serial connection with a baud-rate of 9600 bps and keeps waiting for any incoming
bytes. Any message that is received will be sent back.

3. After uploading the previous code, we keep the dev-board connected to power and, we
open the android application and navigate to Connection Fragment 3.2b.

4. We click the Bluetooth button and make sure to fulfill the requirements implied by the
android OS. After finishing that, click discover devices button (see fig 5.3a).

5. A device will appear in the list of discovered devices as in fig 5.3b. We click on that
device to connect. We can see from the screenshot in fig 5.3c that the connection was
successful.

6. Now we navigate to Terminal fragment 3.2c and send the messages : BT test”, BT test
2”, BT test 3”.

7. We can see from fig 5.3d that the messages were sent and received back successfully !

43
 Chapter 5

1 #include <SoftwareSerial.h>

3 SoftwareSerial mySerial(D7,D8);

5 void setup() {
mySerial.begin(9600);
7 mySerial.println("init");
}
9
void loop() {
11
if(mySerial.available()){
13 mySerial.write(mySerial.read());
}
15 }

Listing 5.2: MCU code for testing Bluetooth connection

(a) Bluetooth requirements (b) Discovered devices (c) Connection successful

44
Chapter 5

(d) Test Messages

Figure 5.3: Bluetooth connection results

UART connection test:

Now we test the UART connection. To do so we go through the following steps:

1. Connect the NodeMCU to the mobile phone with a USB cable.

2. Upload the code in listing 5.3 to the microcontroller. The code functionality is identical
to the one of Bluetooth.

3. After uploading the previous code, we open the android application and navigate to Con-
nection Fragment 3.2b.

4. Click the USB button and wait till the app detects the USB device attached to the phonese
USB port. When one is found, the app will automatically connect to it. In our case the
app detected a device at /dev/bus/usb/001/002 and successfully connected to it (see fig
5.4a).

5. Now we navigate to Terminal fragment 3.2c and send the messages : USB test”, Usb test
2”, usb test 3”.

6. We can see from fig 5.4b that the messages were sent and received back successfully !

45
 Chapter 5

1
void setup() {
3 Serial.begin(9600);
Serial.println("init");
5 }

7 void loop() {
if(Serial.available()) Serial.write(mySerial.read());
9 }

Listing 5.3: MCU code for testing USB connection

(a) USB Connection successful (b) USB Test Messages

Figure 5.4: UART connection results

46
Chapter 5

5.2 Testing GUIs and commands

In this section we will build a simple GUI to test whether the android app works properly, i.e.
interacting with widgets will send commands in the desired manner. It will also be used to
test some special components like sensor blocks and whether the Helper library can parse the
commands sent from that GUI correctly.

The GUI that we will be testing will have the following widgets :

• widget 1: A button that sends the message “CLICK” whenever its clicked

• widget 2: A toggle button (similar to a switch) that keeps toggling between 0 and 1 each
time its clicked.

• widget 3: A slider that sends a range of values from 0 to 100

• widget 4: A Gyroscope sensor block that sends angular velocity on X, Y and Z axis

• widget 5: A Joy stick that sends the balls XY coordinates and the angle.

According to the widgets listed above we will need to create and position the widgets in fig
5.5b and link them to the commands in fig 5.5a:

(a) Commands for this example (b) GUI for this example

47
 Chapter 5

• command 1: has command name = “CLICK” and 0 paramaters. And, is linked to

the first button.

• command 2: has command name = Toggle and a single variable parameter of type
int to store 0 or 1. And is linked to the toggle button.

• command 3: has command name = “SLIDE” and a single variable parameter of type
int to store the slider value (0 to 100). And is linked to the red slider.

• command 4: has command name = “GYRO” and 3 variable parameters of type dou-
ble to store the angular velocities about XYZ. And is linked to the Gyroscope block.

• command 5: has command name = “JOY” and 3 variable parameters of type double
to store the XY location of the joystick’s ball and its angle from the x-axis.

The following code will be used in order to test the functionality of the GUI in fig 5.5b :
1 #include <Arduino.h>
#include <ESP8266WiFi.h>
3 #include <WiFiClient.h>

5 #include "ProtoHelper.h"
#include "Command.h"
7
// Hardcode WiFi parameters as this isn’t going to be moving around.
9 const char* ssid = "Oneplus 6";
const char* password = "Pass1234567";
11
// Start a TCP Server on port 1027
13 static WiFiServer server(1027);
static WiFiClient client ;
15
//Helper library
17 ProtoHelper* helper;

19 //callback for received commands

void onReceive(Command *command);
21
void setup() {
23
Serial.begin(115200);
25 WiFi.begin(ssid,password);
Serial.println("");
27
//Wait for connection
29 while(WiFi.status() != WL_CONNECTED)
{
31 delay(500);
Serial.print(".");
33 }
Serial.print("Connected to "); Serial.println(ssid);
35 Serial.print("IP Address: "); Serial.println(WiFi.localIP());

37 // Start the TCP server

server.begin();
39

48
 Chapter 5

//get an instance of the helper class

41 helper = ProtoHelper::getInstance();

43 //set the callback when a command is received

helper->setOnCommandRecievedCallabck(onReceive);
45 }

47 void loop() {
if (!client.connected()) {
49 // try to connect to a new client
client = server.available();
51 } else {
// read data from the connected client
53 if (client.available() > 0) {
char a = client.read();
55 helper->LoadNewByte(a);
}
57 }
}
59

61 void onReceive(Command *command){

63 Serial.print(command->getOpcode());Serial.print(" ");

65 for (int i = 0; i < command->getParamsSize(); i++)

{
67 Serial.print(command->getParam(i)->getStringValue()); Serial.print(
" ");
}
69
Serial.println("");
71
}

The above code allows connecting the android device to the ESP8266 via TCP/IP in the
same ways as in 5.1. And the Helper library is used to parse the commands received from the
mobile device in order to display them in the serial monitor.

Outputs:

Clicking the first Button results in the following output :

Figure 5.6: output from clicking first button

49
Chapter 5

Interacting with the toggle Button results in the following output :

Figure 5.7: output from clicking toggle button

Dragging the slider results in the following output :

Figure 5.8: output from dragging the red slider

Enabling the Gyroscope sensor block results in the following output :

Figure 5.9: output from Gyroscope sensor block

Moving the ball of the joystick results in the following output :

Figure 5.10: output from joystick

50
Chapter 5

Next, we will see how the received commands can be used in a real project.

5.3 Using received commands in a real project

In this section we show an example where we create a GUI and use it in a real project. We will
be controlling two stepper motors by moving them with a certain angle in different directions
and be able to vary their speed. Also, we are going to demonstrate how two commands can
have the same command name but different parameters.

We will need the following widgets:

• 2 sliders for setting the motors angles

• 2 knobs for setting the motors speeds

• A button for sending a signal to spin both motors

• A button to stop both motors from spinning

Figure 5.11: Commands and GUI for stepper motor example

51
 Chapter 5

According to those widgets, we created the GUI and commands in figure 5.11.

The first and second commands have: command name = ‘R’ which stands for Rotate,
a constant integer parameter for distinguishing which motor to rotate and lastly a variable
integer parameter for holding the rotation angle.

The third and fourth commands have: command name = ‘S’ which stands for Speed, a
constant integer parameter for distinguishing which motor to change its’ speed and lastly a
variable integer parameter for holding the speed value.

The fifth and sixth commands are single character commands. ‘P’ is a signal for spinning
both motors and ‘X’ is a signal for stopping them.

Remark:

We have used a single character for command names instead of a full string because of reasons
that we will mention in the discussion section.

For controlling the stepper motors we used a library called TinyStepLib [12]. It basically
allows for creation of many instances of stepper motors and control them using the following
functions :
1 stepper_n.moveWithAngle(angle); //for rotating with a positive angle.
stepper_n.setDirection(angle); //for setting the direction of stepping
3 stepper_n.setSpeed(angle); //for setting the speed of the motor

This time the callback function in the code implementation in the micro-controller side is
as follows:
1 void callback(Command* command){
char commandName = *(command->getOpcode());
3
switch (commandName) {
5 case ’X’://stop both motors
stepper_1->stop();
7 stepper_2->stop();
break;
9
case ’P’://spin both motors
11 stepper_1->spin(360);//360 degrees per sec
stepper_2->spin(360);
13 break;

15 case ’R’://rotate motors

//param 0 is the motor
17 if (command->getParam(0)->getIntValue() == 1)
stepper_1->moveWithAngle(command->getParam(1)->getIntValue());
19 else
stepper_2->moveWithAngle(command->getParam(1)->getIntValue());
21
break;

52
 Chapter 5

23
case ’S’://set speed of motors
25 int speed = map(command->getParam(1)->getIntValue(),0,37,0,360);
//param 0 is the motor
27 if (command->getParam(0)->getIntValue() == 1)
stepper_1->setSpeed(speed);
29 else
stepper_2->setSpeed(speed);
31 break;

33 }
}

We have recorded a video to demonstrate how everything worked together. The following
link can be used to watch it : http://bit.ly/ProtoPlusDemo.

5.4 Discussion

We mentioned earlier that we have used only a single character for a command name, this
is because if we represent a command with a complete sting, then we will have to do a string
comparison instead of a simple character comparison. A sting comparison requires more pro-
cessing power and consumes time. For instance, in a project that requires 15 commands, 15
strings will have to be compared each time a command is received which is highly inefficient.
A good practice would be to use a single character for representing a command, plus it is pos-
sible use up to 120 different characters to represent different command names which is very
sufficient and generally doesn’t create any problem and requires less time to transmit. Also, the
same thing would be applied for constant parameters that are not numbers.

From the video of the previous example we see that the motors can be controlled from the
GUI exactly as intended. The great thing is that the time taken to create the Graphical user
interface and use it to test that stepper motor library did not exceed 10 minutes which shows
how much time can be gained by using our proposed solution.

53
Conclusion

The objectives of this project were met: The android application can provide modular and cus-
tomizable components that can be linked to user-defined commands and dragged and dropped
to make graphical user interfaces. In addition, the app is very intuitive and has a simple inter-
face which makes it usable not only by developers and professionals but also by beginners and
hobbyists. The C++ library also worked as intended, the received data was parsed rapidly and
efficiently and the API provided by the library allowed handing that data in a concise manner.

This project can be improved in many aspects, the most important additions that we consider
are as follows:

• Currently, the only built-in widgets available are : button, toggle button, slider, knob,
joystick and gyro sensor block. So far only these components were created for the pur-
pose of demonstrating the project. Many more components and sensor blocks should be
added in the future.

• More connection methods can be supported such as GSM and Ethernet and other different
protocols like SPI and I2C.

• Even though commands are sent efficiently, that aspect can be further improved to reduce
transmission and processing time.

• Adding other widgets that can serve as outputs such as LEDs instead of inputs would be
very useful.

• Extend the GUI screen by allowing scroll.

Hopefully in the next few months the final version of this project will be released to public
as a product by uploading the android application to the Google Play Store. The application
can be downloaded from the following link bit.ly/protoplus.

54
Bibliography

[1] circuitbasics. UART data transmission diagram. URL: https://www.circuitbasics.

com/wp-content/uploads/2016/01/Introduction-to-UART-Data-
Transmission-Diagram.png.
[2] circuitbasics. UART packet. URL: https : / / www . circuitbasics . com / wp -
content/uploads/2016/01/Introduction-to-UART-Packet-Frame-
and-Bits-2.png.
[3] felHR85. UsbSerial Library. URL: https://github.com/felHR85/UsbSerial.
[4] Google. Activity class. URL: https://developer.android.com/reference/
android/app/Activity.
[5] Google. Android Bluetooth APIs. URL: https://developer.android.com/
guide/topics/connectivity/bluetooth.
[6] Google. Android Broadcasts overview. URL: https : / / developer . android .
com/guide/components/broadcasts.
[7] Google. Android manifest. URL: https://developer.android.com/guide/
topics/manifest/manifest-intro.
[8] Google. Android platform. URL: https://developer.android.com/guide/
platform.
[9] Google. Fragment class. URL: https://developer.android.com/reference/
androidx/fragment/app/Fragment.html.
[10] Google. Intent class. URL: https://developer.android.com/reference/
android/content/Intent.
[11] google. GSON Library. URL: https://github.com/google/gson.
[12] Zakaria madaoui. TinyStepLib Library. URL: https://github.com/zakimadaoui/
TinyStepLib.
[13] Wikipidia. Bluetooth. URL: https://en.wikipedia.org/wiki/Bluetooth.

55