A Reinforcement Learning Routing Protocol for UAV Aided Public Safety Networks
Abstract
:1. Introduction
- We first analyze the impact of different clustering schemes and a UAV presence on the performance of multihop routing in a disaster scenario. We then present a RL approach to ensure end-to-end connectivity and improve Energy Efficeincy (EE) of PSNs.
- We consider the mobility of UAVs around the disaster area. Multiple UAV trajectories are devised in order to improve the coverage of clusters in the disaster area while ensuring EE.
2. System Model
2.1. Network Throughput and Delay
2.2. Energy Model
2.3. Problem Formulation
3. Reinforcement Learning-Based Routing
3.1. Clustering
3.1.1. Clustering-Energy
3.1.2. Clustering-Kmean
3.2. Route Discovery
3.3. Routing
Algorithm 1: Reinforcement Learning Algorithm. |
|
3.4. Control Overhead
- Beacon messages overhead: These beacon messages are sent by the nodes to find the information about their respective neighbors. The number of beacon messages sent are dependent on the number of nodes, N, in the environment. The nodes which are in its vicinity will reply. These beacon messages are resent after every 10 s to renew the neighbor’s information. Since there is no mobility in our scenario the neighbor change is only possible due to DNs.
- Clustering overhead: To calculate the clustering overhead, we categorize the clustering schemes as centralized and decentralized schemes. The schemes that employ K mean clustering are centralized schemes because they require the location information of all the nodes. We assume that after the exchange of beacon messages, location information of all the nodes is forwarded/relayed to the CC. The CC performs K mean clustering and inform the nodes about their clusters and CHs. Here, we do not consider the overhead of passing location information to CC and K mean information back to the nodes from CC. This approximate overhead can be readily found from the achievable capacity [52]. The schemes that employ clustering energy are distributed schemes. The clustering overhead is calculated when a node broadcasts control packets to declare itself as the CH. The receiving nodes will reply accordingly as discussed in Section 3.1.1. These control packets depend on the number of clusters i and the number of CMs in each cluster.
- Routing overhead: Once the CHs are formed, this step includes the amount of control overhead involved in the discovery of neighboring CHs and the routing path. It is assumed that the routing tables are only maintained at the CHs, which will reduce the overall routing overhead. For route discovery, the CHs will send the control packets to the neighboring CHs which will then forward the control packets all the way to the CC. The CC will confirm the routing path for each CH through a reverse response as discussed in Section 3.2.
4. Performance Analysis
4.1. Reinforcement-Based Routing
4.2. Combined RL and UAV(s) Trajectory Optimization
- Two UAVs on Opposite Axis and Same direction (TUOAS): In this scheme, the two UAVs are placed at the edge of the building. Both the UAVs are placed on the opposite corners of the building. They start moving from the same side of the building, as shown in Figure 16. Both these UAVs are moving in parallel to each other but the direction they are following is the same. The trajectory they are following is along the straight line alongside the building. When they reach the opposite corner of the building, they will follow the same path backwards. The CHs that are in the range of any of these UAVs will send there packets through the respective UAV. Thus, the Equation (15) for the two UAVs will be modified as:Here is the cost associated with UAV in which and are the sender and receiver nodes associated to the UAV:Similarly is the associated cost with the UAV. While and are the sender and receiver nodes associated to the UAV. In the case, sender m is in the direct range of CC, the cost will be calculated using Equation (15). The above equations can also be used for all other two UAV schemes presented below.
- Two UAVs on Opposite Axis and Opposite Direction (TUOAO): In this scheme two UAVs are placed at the opposite corner of the building as shown in Figure 16. Both UAVs moves along a straightline alongside the building towards their respective direction. By moving in this way they will help in maximize the coverage area of the building affected by the disaster. When a UAV reaches the edge of the building it will follow the same path backwards and it keeps on doing this till the end of the simulation.
- Two UAVs moving on Same Axis (TUSA): In this scheme, both the UAVs were placed on the same axis separated by 60 m, as shown in Figure 16. Both the UAVs move in the same direction and on reaching there respective endpoint they follow the same path backwards. The maximum separation between them remains the same.
- Single UAV in motion (SU): In this scheme, we placed a single UAV at the corner of the building. The UAV moves alongside the building and traverses the same path on its way back from the end of the building.
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Zakaria, M.; Jun, W.; Ahmed, H. Effect of terrorism on economic growth in Pakistan: An empirical analysis. Econ. Res. Ekonomska IstražIvanja 2019, 32, 1794–1812. [Google Scholar] [CrossRef] [Green Version]
- Qureshi, R.; Gulraiz, A.; Shahzad, Z. An Analysis of Media’s Role: Case Study of Army Public School (APS) Peshawar Attack. Soc. Commun. 2016, 2, 20–30. [Google Scholar] [CrossRef] [Green Version]
- Masood, A.; Scazzoli, D.; Sharma, N.; Moullec, Y.L.; Ahmad, R.; Reggiani, L.; Magarini, M.; Alam, M.M. Surveying pervasive public safety communication technologies in the context of terrorist attacks. Phys. Commun. 2020, 41, 101109. [Google Scholar] [CrossRef]
- Ali, K.; Nguyen, H.X.; Shah, P.; Vien, Q.T.; Bhuvanasundaram, N. Architecture for public safety network using D2D communication. In Proceedings of the 2016 IEEE Wireless Communications and Networking Conference Workshops (WCNCW), Doha, Qatar, 3–6 April 2016; pp. 206–211. [Google Scholar]
- Shaikh, F.S.; Wismüller, R. Routing in multi-hop cellular device-to-device (D2D) networks: A survey. IEEE Commun. Surv. Tutor. 2018, 20, 2622–2657. [Google Scholar] [CrossRef]
- Muraoka, K.; Shikida, J.; Sugahara, H. Feasibility of capacity enhancement of public safety LTE using device-to-device communication. In Proceedings of the 2015 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Korea, 28–30 October 2015; pp. 350–355. [Google Scholar]
- Anupama, M.; Sathyanarayana, B. Survey of cluster based routing protocols in mobile adhoc networks. Int. J. Comput. Theory Eng. 2011, 3, 806. [Google Scholar] [CrossRef] [Green Version]
- Ephremides, A.; Wieselthier, J.E.; Baker, D.J. A design concept for reliable mobile radio networks with frequency hopping signaling. Proc. IEEE 1987, 75, 56–73. [Google Scholar] [CrossRef] [Green Version]
- Chiang, C.C.; Wu, H.K.; Liu, W.; Gerla, M. Routing in clustered multihop, mobile wireless networks with fading channel. In Proceedings of the IEEE SICON, Singapore, 14–17 April 1997; Volume 97, pp. 197–211. [Google Scholar]
- Fodor, G.; Parkvall, S.; Sorrentino, S.; Wallentin, P.; Lu, Q.; Brahmi, N. Device-to-device communications for national security and public safety. IEEE Access 2014, 2, 1510–1520. [Google Scholar] [CrossRef]
- Lu, Q.; Miao, Q.; Fodor, G.; Brahmi, N. Clustering schemes for D2D communications under partial/no network coverage. In Proceedings of the 2014 IEEE 79th Vehicular Technology Conference (VTC Spring), Seoul, Korea, 18–21 May 2014; pp. 1–5. [Google Scholar]
- Guizani, B.; Ayeb, B.; Koukam, A. Hierarchical cluster-based link state routing protocol for large self-organizing networks. In Proceedings of the 2011 IEEE 12th International Conference on High Performance Switching and Routing, Cartagena, Spain, 4–6 July 2011; pp. 203–208. [Google Scholar]
- Laha, A.; Cao, X.; Shen, W.; Tian, X.; Cheng, Y. An energy efficient routing protocol for device-to-device based multihop smartphone networks. In Proceedings of the 2015 IEEE International Conference on Communications (ICC), London, UK, 8–12 June 2015; pp. 5448–5453. [Google Scholar]
- Chang, T.C.; Wei, C.; Hsu, M.; Lin, C.; Su, Y.T. Distributed clustering and spectrum-based proximity device discovery in a wireless network. In Proceedings of the 2016 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), Nara, Japan, 1–3 June 2016; pp. 1–4. [Google Scholar]
- Islam, N.; Dey, S.; Sampalli, S. Energy-Balancing Unequal Clustering Approach to Reduce the Blind Spot Problem in Wireless Sensor Networks (WSNs). Sensors 2018, 18, 4258. [Google Scholar] [CrossRef] [Green Version]
- Minhas, H.I.; Ahmad, R.; Ahmed, W.; Alam, M.M.; Magarani, M. On the impact of clustering for Energy critical Public Safety Networks. In Proceedings of the 2019 International Symposium on Recent Advances in Electrical Engineering (RAEE), Islamabad, Pakistan, 28–29 August 2019; Volume 4, pp. 1–5. [Google Scholar]
- Echoukairi, H.; Kada, A.; Bouragba, K.; Ouzzif, M. A novel centralized clustering approach based on K-means algorithm for wireless sensor network. In Proceedings of the 2017 Computing Conference, London, UK, 18–20 July 2017; pp. 1259–1262. [Google Scholar]
- Gupta, A.; Shekokar, N. A novel K-means L-layer algorithm for uneven clustering in WSN. In Proceedings of the 2017 International Conference on Computer, Communication and Signal Processing (ICCCSP), Chennai, India, 10–11 January 2017; pp. 1–6. [Google Scholar]
- Razzaq, M.; Devi Ningombam, D.; Shin, S. Energy efficient K-means clustering-based routing protocol for WSN using optimal packet size. In Proceedings of the 2018 International Conference on Information Networking (ICOIN), Chiang Mai, Thailand, 10–12 January 2018; pp. 632–635. [Google Scholar]
- Bholowalia, P.; Kumar, A. Article: EBK-Means: A Clustering Technique based on Elbow Method and K-Means in WSN. Int. J. Comput. Appl. 2014, 105, 17–24. [Google Scholar]
- Arbia, D.B.; Alam, M.M.; Attia, R.; Hamida, E.B. ORACE-Net: A novel multi-hop body-to-body routing protocol for public safety networks. Peer-Peer Netw. Appl. 2017, 10, 726–749. [Google Scholar] [CrossRef]
- Yuan, H.; Guo, W.; Wang, S. Emergency route selection for D2D cellular communications during an urban terrorist attack. In Proceedings of the 2014 IEEE International Conference on Communications Workshops (ICC), Sydney, Australia, 10–14 June 2014; pp. 237–242. [Google Scholar]
- Hassan, A.; Ahmad, R.; Ahmed, W.; Magarini, M.; Alam, M.M. UAV and SWIPT Assisted Disaster Aware Clustering and Association. IEEE Access 2020, 8, 204791–204803. [Google Scholar] [CrossRef]
- Li, X.; Guo, D.; Grosspietsch, J.; Yin, H.; Wei, G. Maximizing mobile coverage via optimal deployment of base stations and relays. IEEE Trans. Veh. Technol. 2015, 65, 5060–5072. [Google Scholar] [CrossRef] [Green Version]
- Zeng, Y.; Zhang, R.; Lim, T.J. Wireless communications with unmanned aerial vehicles: Opportunities and challenges. IEEE Commun. Mag. 2016, 54, 36–42. [Google Scholar] [CrossRef] [Green Version]
- Mozaffari, M.; Saad, W.; Bennis, M.; Debbah, M. Unmanned aerial vehicle with underlaid device-to-device communications: Performance and tradeoffs. IEEE Trans. Wirel. Commun. 2016, 15, 3949–3963. [Google Scholar] [CrossRef]
- Al-Hourani, A.; Kandeepan, S.; Lardner, S. Optimal LAP altitude for maximum coverage. IEEE Wirel. Commun. Lett. 2014, 3, 569–572. [Google Scholar] [CrossRef] [Green Version]
- Košmerl, J.; Vilhar, A. Base stations placement optimization in wireless networks for emergency communications. In Proceedings of the 2014 IEEE International Conference on Communications Workshops (ICC), Sydney, Australia, 10–14 June 2014; pp. 200–205. [Google Scholar]
- Hassan, A.; Ahmad, R.; Ahmed, W.; Magarini, M.; Alam, M.M. Managing Critical Nodes in UAV assisted Disaster Networks. In Proceedings of the 2020 17th Biennial Baltic Electronics Conference (BEC), Tallinn, Estonia, 6–8 October 2020; pp. 1–5. [Google Scholar] [CrossRef]
- Alam, M.M.; Le Moullec, Y.; Ahmad, R.; Magarini, M.; Reggiani, L. A Primer On Public Safety Communication in the Context of Terror Attacks: The NATO SPS “COUNTER-TERROR” Project. In Advanced Technologies for Security Applications; Palestini, C., Ed.; Springer: Dordrecht, The Netherlands, 2020; pp. 19–34. [Google Scholar]
- Valente Klaine, P.; Nadas, J.; Souza, R.; Imran, M. Distributed Drone Base Station Positioning for Emergency Cellular Networks Using Reinforcement Learning. Cogn. Comput. 2018, 10. [Google Scholar] [CrossRef] [Green Version]
- Hydher, H.; Jayakody, D.N.K.; Hemachandra, K.T.; Samarasinghe, T. Intelligent UAV deployment for a disaster-resilient wireless network. Sensors 2020, 20, 6140. [Google Scholar] [CrossRef]
- Lin, N.; Fu, L.; Zhao, L.; Min, G.; Al-Dubai, A.; Gacanin, H. A Novel Multimodal Collaborative Drone-Assisted VANET Networking Model. IEEE Trans. Wirel. Commun. 2020, 19, 4919–4933. [Google Scholar] [CrossRef]
- Zhao, L.; Yang, K.; Tan, Z.; Li, X.; Sharma, S.; Liu, Z. A Novel Cost Optimization Strategy for SDN-Enabled UAV-Assisted Vehicular Computation Offloading. IEEE Trans. Intell. Transp. Syst. 2020. [Google Scholar] [CrossRef]
- Zhang, S.; Zhang, H.; Song, L. Beyond D2D: Full Dimension UAV-to-Everything Communications in 6G. IEEE Trans. Veh. Technol. 2020, 69, 6592–6602. [Google Scholar] [CrossRef] [Green Version]
- Wang, L.; Wang, K.; Pan, C.; Xu, W.; Aslam, N.; Hanzo, L. Multi-Agent Deep Reinforcement Learning Based Trajectory Planning for Multi-UAV Assisted Mobile Edge Computing. IEEE Trans. Cogn. Commun. Netw. 2020. [Google Scholar] [CrossRef]
- Atif, M.; Ahmad, R.; Ahmad, W.; Zhao, L.; Rodrigues, J.J.P.C. UAV-Assisted Wireless Localization for Search and Rescue. IEEE Syst. J. 2021, 1–12. [Google Scholar] [CrossRef]
- Arafat, M.Y.; Habib, M.A.; Moh, S. Routing Protocols for UAV-Aided Wireless Sensor Networks. Appl. Sci. 2020, 10, 4077. [Google Scholar] [CrossRef]
- Zhan, C.; Zeng, Y.; Zhang, R. Energy-Efficient Data Collection in UAV Enabled Wireless Sensor Network. IEEE Wirel. Commun. Lett. 2018. [Google Scholar] [CrossRef] [Green Version]
- Gomez, J.M.; Wiedemann, T.; Shutin, D. Unmanned Aerial Vehicles in Wireless Sensor Networks: Automated Sensor Deployment and Mobile Sink Nodes. In Proceedings of the International Conference on Intelligent Autonomous Systems, Baden-Baden, Germany, 11–15 June 2018. [Google Scholar]
- Uddin, M.A.; Mansour, A.; Jeune, D.L.; Ayaz, M.; Aggoune, E.H.M. UAV-Assisted Dynamic Clustering of Wireless Sensor Networks for Crop Health Monitoring. Sensors 2018, 18, 555. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zema, N.R.; Mitton, N.; Ruggeri, G. Using location services to autonomously drive flying mobile sinks in wireless sensor networks. In Proceedings of the International Conference on Ad Hoc Networks, San Remo, Italy, 1–2 September 2015; pp. 180–191. [Google Scholar]
- Villas, L.A.; Guidoni, D.L.; Maia, G.; Pazzi, R.W.; Ueyama, J.; Loureiro, A.A. An energy efficient joint localization and synchronization solution for wireless sensor networks using unmanned aerial vehicle. Wirel. Netw. 2015, 21, 485–498. [Google Scholar] [CrossRef]
- Albu-Salih, A.T.; Seno, S.A.H. Energy-efficient data gathering framework-based clustering via multiple UAVs in deadline-based WSN applications. IEEE Access 2018, 6, 72275–72286. [Google Scholar] [CrossRef]
- Dong, M.; Ota, K.; Lin, M.; Tang, Z.; Du, S.; Zhu, H. UAV-assisted data gathering in wireless sensor networks. J. Supercomput. 2014, 70, 1142–1155. [Google Scholar] [CrossRef]
- Okcu, H.; Soyturk, M. Distributed clustering approach for UAV integrated wireless sensor networks. Int. J. Hoc Ubiquitous Comput. 2014, 15, 106–120. [Google Scholar] [CrossRef]
- Hashima, S.; Hatano, K.; Mohammed, E. Multiagent Multi-Armed Bandit Schemes for Gateway Selection in UAV Networks. In Proceedings of the 2020 IEEE Globecom Workshops (GC Wkshps), Taipei, Taiwan, 7–11 December 2020; pp. 7–11. [Google Scholar]
- Mohamed, E.M.; Hashima, S.; Aldosary, A.; Hatano, K.; Abdelghany, M.A. Gateway Selection in Millimeter Wave UAV Wireless Networks Using Multi-Player Multi-Armed Bandit. Sensors 2020, 20, 3947. [Google Scholar] [CrossRef]
- Hodge, V.J.; Hawkins, R.; Alexander, R. Deep reinforcement learning for drone navigation using sensor data. Neural Comput. Appl. 2020, 33, 2015–2033. [Google Scholar] [CrossRef]
- Bultitude, Y.D.J.; Rautiainen, T. IST-4-027756 WINNER II D1. 1.2 V1. 2 WINNER II Channel Models. In EBITG, TUI, UOULU, CU/CRC, NOKIA; Tech. Rep; 2007; Available online: http://www.ero.dk/93F2FC5C-0C4B-4E44-8931-00A5B05A331B?frames=no& (accessed on 10 March 2021).
- Kiani, F.; Amiri, E.; Zamani, M.; Khodadadi, T.; Abdul Manaf, A. Efficient intelligent energy routing protocol in wireless sensor networks. Int. J. Distrib. Sens. Netw. 2015, 11, 618072. [Google Scholar] [CrossRef] [Green Version]
- Gupta, P.; Kumar, P.R. The capacity of wireless networks. IEEE Trans. Inf. Theory 2000, 46, 388–404. [Google Scholar] [CrossRef] [Green Version]
- Johnson, D.B. A note on Dijkstra’s shortest path algorithm. J. ACM 1973, 20, 385–388. [Google Scholar] [CrossRef]
Name | Symbol | Name | Symbol |
---|---|---|---|
Number of nodes | N | Number of clusters | i |
Cluster radius | CH Tx radius | ||
K value | K | Average distance of each K | |
Distance | d | Transmissions | T |
Reward function | Learning rate | V | |
Discount factor | Path loss | ||
Carrier frequency | Number of walls | ||
Number of floors | Floor loss | ||
Throughput | r | Path | |
Routing matrix | H | Packet size | L |
Energy Efficiency | Delay | ||
Transmitting power | Bandwidth | B | |
Noise power | Packet size | L | |
Beta | Alpha |
Scheme Name | Clustering Type | Clustering Overlapping | Gate Way | Location Awareness |
---|---|---|---|---|
Clustering-energy | Energy based | High | No | Not required |
Clustering-energy-GW | Energy based | High | Yes | Not required |
Clustering-k mean | K mean | Low | No | Required |
Clustering-k mean-GW | K mean | Low | Yes | Required |
Parameter | Values |
---|---|
Number of Devices (N) | 100 |
Network Grid | 100 m × 100 m |
CC Placement | m |
UAV Placement (Initial) | m |
Size of Data Packet (L) | 1024 bytes |
Header Size | 40 bytes |
Initial Power Level | 0 to 1 J |
50 nJ/bit | |
50 nJ/bit | |
Threshold | mJ |
Cluster Range, | 30 m |
CH Tx Range, | 45 m |
Distance b/w UAV and CC | 60 m |
Max Transmissions in a Round () | 5 |
1024 | |
Discount Factor () | |
Learning Rate (V) |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Minhas, H.I.; Ahmad, R.; Ahmed, W.; Waheed, M.; Alam, M.M.; Gul, S.T. A Reinforcement Learning Routing Protocol for UAV Aided Public Safety Networks. Sensors 2021, 21, 4121. https://doi.org/10.3390/s21124121
Minhas HI, Ahmad R, Ahmed W, Waheed M, Alam MM, Gul ST. A Reinforcement Learning Routing Protocol for UAV Aided Public Safety Networks. Sensors. 2021; 21(12):4121. https://doi.org/10.3390/s21124121
Chicago/Turabian StyleMinhas, Hassan Ishtiaq, Rizwan Ahmad, Waqas Ahmed, Maham Waheed, Muhammad Mahtab Alam, and Sufi Tabassum Gul. 2021. "A Reinforcement Learning Routing Protocol for UAV Aided Public Safety Networks" Sensors 21, no. 12: 4121. https://doi.org/10.3390/s21124121