Enabling Smart Home Energy Management

through Gesture-Based Control and IoT Technology

Khairul Hawani Rambe* and Chia-Ho Ou†*

*
International Master Program in Information Technology and Applications
†
Department of Computer Science and Information Engineering
National Pingtung University, Taiwan
Email: [email protected], [email protected]

Abstract— The Internet of Things (IoTs) have become monitoring and control of smart home devices using an
ubiquitous in all aspects of public needs today. The application Android app [6]. The Kinect sensor detects hand movements
of IoT technology is crucial for promoting energy-saving to control smart home devices [7]. Another system developed
behavior. This paper presents a control system used for smart an IoT-based system that uses hand gestures to control smart
home devices based on hand gesture control and Android devices. The system uses hand gestures to control smart
applications. The hardware design uses the ESP32 household appliances, such as lights, fans, and air
microcontroller, which has an in-chip Wi-Fi module, making it conditioners. [8]. Utilizes an Android smartphone and a
very suitable for creating IoT application systems. The software gesture APDS 9960 sensor for gesture recognition in smart
was designed using the Arduino IDE, and the application
home energy management systems [9]. Development of a
display design was developed using Kodular Creator, which is
used to control the lamps and switches. The experiment results
system that enables gesture-based control for smart homes
show that the electrical equipment can be activated using the using an Android device and a Bluetooth module for
Android application with gesture sensors by waving down, up, communication with smart devices [10]. All systems of them
left, and right. can achieve real-time control of smart home devices and are
proven to be effective in reducing energy consumption.
Keywords—Internet of Things, smart home, android, gesture.
III. SYSTEM IMPLEMENTATION
I. INTRODUCTION The proposed control system is used for smart homes
Smart homes are becoming popular to reduce energy and based on hand gesture control and Android applications. The
improve living standards. The times have made smart homes hardware design uses the ESP32 microcontroller, which has
and the Internet of Things (IoTs) affect our lives through an in-chip Wi-Fi module, making it very suitable for creating
connected services anywhere and anytime [1]. The Internet of IoT application systems. Hand gesture serves as a signal for
Things (IoT) provides a network of interconnected devices the gesture sensor, and if there is a hand gesture as a gesture
that can be controlled remotely[2]. Strong internet signal, it will be input and processed by the ESP32 Wi-Fi
connectivity and appropriate facilities are necessary to use IoT microcontroller to produce an output that is used as an input
technology. This technology has the potential to provide a relay module to turn on or turn off the lamps and switches.
more effective and user-friendly way to control smart home The process block consists of the ESP32 Wi-Fi module as
devices, which can result in more efficient energy the command receiver control center and feedback sender.
management such as turning off electrical equipment when The output block consists of 4 relays which are used as
not in use. Android can be used to remotely turn on or off automatic switches for lamp 1, lamp 2, switch 1 and switch 2.
electronics, such as lamps, fans, and televisions, and promote Figure 1 shows the system block diagram used in this study.
energy-efficient behaviors. Another reason for implementing
IoT is to control the use of electronic devices, which can save
on electricity bills. In this study, gesture sensors detect hand
gestures, and gesture signals are used as a switch to turn lights
on or off. The ESP32 Wi-Fi microcontroller processes the
gesture signals and produces output, which is used as input for
the relay module. Digital output data is generated by hand
gestures, such as moving up, down, left, and right on the top
of the gesture sensor, instead of touch or push. An Android
application was created using Kodular Creator.
II. RELATED WORK Fig. 1: System block diagram.
The importance of Internet usage has become ubiquitous In terms of software design, the software was designed
in all aspects of public needs today[3]. Gesture-based control using the Arduino IDE, and the application display design was
has been extensively studied and several techniques have been developed using Kodular Creator.
proposed to detect and recognize hand gestures[4]. IoT
technology has also been studied for smart home energy To control the lamps and switches, it is necessary to
management. IoT devices can be used to remotely control determine the logic to be applied to the control system and
smart home devices, enabling users to manage their energy make the algorithm, which is then implemented using the
usage[5]. One of the systems is a combination of Zigbee and Arduino and IDE Kondular Creator. The flowchart for the
WIFI communication technologies that enable remote proposed algorithm on the device will be planted inside the
 Arduino UNO microcontroller and Kondular Creator. Figure ESP32 so that the output will turn on relay 4 and turn on or
2 shows the flowchart for our system design presented in this turn off switch 2.
study.

Fig. 4: Android Application

Figure 4 shows the Android Application implemented in
the study. Once the Application is connected to the Internet, it
displays the status of the device on a thumbnail screen, and
several buttons are available to control multiple outputs. If you
press the button on the application menu, it will turn on the
relay and turn on or turn off the lamp and switch.
Based on the experiment results it was found that electrical
equipment can be activated using the Android application and
gesture sensors by waving down, up, left, and right.
ACKNOWLEDGMENT
This work was supported by the National Science and
Technology Council of Taiwan under Contract MOST 111-
2221-E-153-004 and the Ministry of Education STEM
Women's Talent Development Program.
Fig. 2: System Design Flowchart
REFERENCES
[1] W. Choi, J. Kim, S. Lee, and E. Park, “Smart home and Internet of
Things: A bibliometric study,” Journal of Cleaner Production, p.
IV. RESULTS AND DISCUSSION 126908.
To assess the capabilities of the proposed system, testing [2] A. A. Laghari, K. Wu, R. A. Laghari, M. Ali, and A. A. Khan, “A
is very important. Initially, the ESP32 connection is tested to Review and State of Art of Internet of Things (IoT),” Archives of
Computational Methods in Engineering, Jul. 2021
ensure connectivity between the ESP32, the gesture sensor,
[3] S. Kashan Ali Shah and W. Mahmood, “Smart Home Automation
and the Application. Figure 3 illustrates the testbed system Using IOT and its Low-Cost Implementation,” International Journal of
implemented in the study. Engineering and Manufacturing, vol. 10, no. 5, pp. 28–36, Oct. 2020
[4] C. Li, K. Dong, F. Jin, J. Song, and W. Mo, “Design of Smart Home
Monitoring and Control System Based on Zigbee and WIFI,” 2019
Chinese Control Conference (CCC), Guangzhou, China, 2019, pp.
6345-6348.
[5] S. A. Ayubi, D. W. Sudiharto, E. M. Jadied, and E. Aryanto, “The
Prototype of Hand Gesture Recognition for Elderly People to Control
Connected Home Devices,” Journal of Physics: Conference Series, vol.
1201, p. 012042, May 2019
[6] W. A. Jabbar et al., “Design and Fabrication of Smart Home With
Internet of Things Enabled Automation System,” IEEE Access, vol. 7,
Fig. 3: System testbed pp. 144059–144074, 2019
[7] R. Raza, M. A. Shah, S. A. Malik, and S. H. Ahmed, "Smart home
If a signal is received from the motion sensor located on automation using hand gesture recognition through Kinect sensor,"
the ESP32 microcontroller, the status of Relay 1 will be 2018 International Conference on Computing, Mathematics and
checked. If the gesture is upward, the gesture sensor will Engineering Technologies (iCoMET), Sukkur, Pakistan, 2018, pp. 1-6.
detect the gesture and be processed by ESP32 so that the [8] Lai, C. F., Lee, C. S., and Huang, Y. M.. “Design and Implementation
output will turn on relay 1 and turn on or turn off lamp 1. If of a Gesture-Based Smart Home Control System Using Internet of
the downward gesture, the gesture sensor will detect the Things Technology”. International Journal of Distributed Sensor
Networks, 14(9), 1550147718801415, 2018.
gesture and be processed by ESP32 so that the output will turn
on relay 2 and turn on or turn off lamp 2. If a rightward gesture [9] Zhang, X., Zhang, J., & Guo, W. “Smart Home Energy Management
Based on Android and Gesture APDS 9960 Recognition”. International
is presented, the gesture sensor will detect the gesture and be Journal of Smart Grid and Clean Energy, 9(6), 323-331, 2020.
processed by ESP32 so that the output will turn on relay 3 and [10] Almohammed, Z., Alrajhi, M., Alsaleh, M., & Almuhanna, R.
turn on or turn off switch 1. If the rightward gesture is shown, “Gesture-based control system for smart homes using android”.
the gesture sensor will detect the gesture and be processed by International Conference on Computer Applications & Information
Security (ICCAIS) (pp. 1-6). IEEE, 2019. .