Internet of Things 25 (2024) 101053

Contents lists available at ScienceDirect

Internet of Things
journal homepage: www.elsevier.com/locate/iot

Review article

Current testing and performance evaluation methodologies of LoRa

and LoRaWAN in IoT applications: Classification, issues, and future
directives
Melchizedek Alipio a,b ,∗, Miroslav Bures a
a
System Testing IntelLigent Lab (STILL), Department of Computer Science, Faculty of Electrical Engineering, Czech Technical University in
Prague, Karlovo Namesti 13, Prague, 121 35, Czech Republic
b
Department of Electronics and Computer Engineering, De La Salle University, 2401 Taft Avenue, Manila, 1101, Philippines

ARTICLE INFO ABSTRACT

Keywords: Long Range (LoRa) and Long Range Wide Area Network (LoRaWAN) are emerging technologies
Internet of Things essential in connecting and managing a wide range of devices in various Internet of Things
LoRa (IoT) systems. Testing and evaluation methods play a crucial role in assessing and optimizing
LoRaWAN
the performance of these technologies before their deployment in real-world IoT applications.
Simulation
Previous review studies focused mainly on the comparison of current Low-Power Wide Area
Testbed
Testing
Networks (LPWAN) technologies, evaluating the performance of LoRa and LoRaWAN platforms
for a general or specific application, or in a single testing methodology. However, the literature
does not include any articles dedicated to comprehensively reviewing the current testing
scenarios and performance evaluation methodologies used in LoRa-based or LoRaWAN-based
networks deployed for IoT systems. Hence, this paper aims to review the state-of-the-art studies
on LoRa and LoRaWAN test and evaluation methods in various IoT applications. In this paper,
these studies are critically reviewed and classified according to their test parameters, test
architectures, and performance evaluation methodologies. Additionally, a summary and unified
view of test and evaluation methodologies to assess the performance characteristics of LoRa
and LoRaWAN in IoT-driven applications is presented. Lastly, the issues and challenges behind
these test cases and evaluation methods are identified, and the possible future directions of this
research domain are discussed.

1. Introduction

LoRa and LoRaWAN are wireless communication technologies specifically designed for IoT applications. LoRa is a modulation
technique used to achieve long-range communication in unlicensed radio frequency (RF) bands. It operates in the sub-GHz frequency
bands, typically 433 MHz, 868 MHz, or 915 MHz, which allows signals to travel long distances while consuming very little power.
This makes it well-suited for IoT devices that may need to operate on batteries for extended periods [1,2]. On the other hand,
LoRaWAN is a network protocol built on top of the LoRa technology. It is an open standard and is managed by the LoRa Alliance,1
a nonprofit organization. LoRaWAN defines the communication protocol and system architecture for managing communication
between LoRa-enabled devices and the back-end infrastructure, usually referred to as the network server [3].

∗ Corresponding author at: System Testing IntelLigent Lab (STILL), Department of Computer Science, Faculty of Electrical Engineering, Czech Technical
University in Prague, Karlovo Namesti 13, Prague, 121 35, Czech Republic.
E-mail addresses: [email protected], [email protected] (M. Alipio), [email protected] (M. Bures).
1 https://lora-alliance.org/

https://doi.org/10.1016/j.iot.2023.101053
Received 14 August 2023; Received in revised form 6 November 2023; Accepted 24 December 2023
Available online 30 December 2023
2542-6605/© 2023 Elsevier B.V. All rights reserved.
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

LoRa and LoRaWAN play crucial roles in connecting and managing a wide range of devices. IoT devices, such as sensors,
actuators, and other smart devices, use LoRa technology to transmit data to LoRaWAN gateways. These gateways act as access points
that receive the signals from the devices and forward the data to the central network server. LoRa’s long-range capability enables
IoT devices to communicate over large distances, eliminating the need for frequent communication infrastructure deployment,
especially in rural or remote areas. IoT devices deployed with LoRaWAN can operate on battery power for extended periods, reducing
maintenance costs and making them suitable for applications where power sources are limited or difficult to access [4]. LoRa
and LoRaWAN find applications in various IoT use cases, including smart cities for waste management, environmental monitoring,
agriculture for precision farming and livestock tracking, asset tracking, industrial monitoring, and many more [5].
Testbeds, simulations, and modeling are essential in evaluating and optimizing the performance of LoRa and LoRaWAN
technologies before their deployment in real-world applications. Testbeds are physical setups that mimic real-world environments to
evaluate the performance of devices and networks. These testbeds typically consist of LoRa devices (sensors or end nodes), LoRaWAN
gateways, and a network server [6]. These are used to assess factors like range, coverage, interference, and scalability. They can
be set up in various environments, such as urban, suburban, and rural areas, to measure the effectiveness of the technology under
different conditions. They are valuable for testing new protocols, optimizing network parameters, and conducting experiments before
large-scale deployments [7].
Moreover, simulations and mathematical modeling provide a virtual and mathematical environment to model LoRa and
LoRaWAN. They are an efficient and cost-effective way to analyze network behavior, especially in scenarios where real-world
deployments might be impractical or expensive [8,9]. Simulations can help researchers and engineers understand how LoRa and
LoRaWAN perform in different network topologies, node densities, and interference scenarios. Various simulation tools are available
for LoRa and LoRaWAN, allowing users to configure network parameters, simulate thousands of devices, and analyze the network’s
performance metrics. Using simulations and modeling, researchers can explore different scenarios and compare performance metrics
under varying conditions, which helps in fine-tuning the technology for specific IoT use cases. Additionally, they are instrumental
in assessing network scalability and estimating the network’s capacity to handle increasing numbers of connected devices [10].
Previous studies focused mainly on reviewing and comparing different LPWAN technologies, evaluating the performance of
LoRa and LoRaWAN platforms in either general or specific applications, or a single test evaluation methodology. However, the
literature does not include any articles dedicated to comprehensively reviewing the current testing and performance evaluation
methodologies used in LoRa-based and LoRaWAN-based networks deployed for IoT applications. Therefore, this work makes the
following contributions:

• Overview and review of state-of-the-art LoRa and LoRaWAN test and evaluation methodologies deployed in various IoT
applications.
• Classification and unified view of state-of-the-art LoRa-based and LoRaWAN-based IoT systems and mechanisms in improving
MAC layer features based on test parameters, test architecture, and evaluation methodologies.
• Identification of issues and challenges of test and evaluation methodologies in LoRa and LoRaWAN for IoT applications.
• Discussion of future research directions of test and evaluation methodologies in LoRa and LoRaWAN for IoT applications.

The remainder of this paper is organized as follows: Section 2 discusses the fundamental concepts of LoRa and LoRaWAN
architectures, operational parameters, performance evaluation methodologies, and metrics. Section 3 provides a review of related
surveys and review papers focusing on LoRa and LoRaWAN implementation and evaluation for IoT applications. The search
workflow, paper distribution, and bibliometric results are discussed in Section 4. Section 5 presents the review of the state-of-the-
art LoRa-based and LoRaWAN-based networks in various IoT applications based on test parameters, architectures, and performance
evaluation methodologies. It also presents a summary and unified view of test and evaluation methodologies for assessing LoRa and
LoRaWAN network performance. The issues and challenges concerning test and evaluation methodologies are discussed in Section 6
while the possible future directives of this research domain are discussed in Section 7. Finally, Section 8 summarizes and concludes
the paper.

2. Background

This section provides an overview and background of the LoRa and LoRaWAN network general architectures, operational
parameters, applications, testing methodologies, and performance evaluation metrics.

2.1. LoRa and LoRaWAN architectures, operational parameters, and applications

LoRa and LoRaWAN are wireless communication technologies specifically designed for low-power, long-range communication
in the context of the IoT and other machine-to-machine (M2M) applications. They are ideal for connecting battery-powered IoT
devices over large distances, often in a cost-effective manner.
LoRa is a proprietary wireless modulation technique developed by Semtech Corporation.2 It operates in the sub-GHz license-free
spectrum, which allows for long-range communication while consuming minimal power. The modulation technique used in LoRa,

2 https://www.semtech.com/lora

2
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Fig. 1. General architecture of LoRaWAN network in IoT.

known as Chirp Spread Spectrum (CSS), enables the transmission of data over long distances while maintaining robustness against
interference and noise [11]. On the other hand, LoRaWAN is designed to enable secure, bi-directional communication between IoT
devices and a centralized network server. It defines the communication protocol and system architecture for LoRa-based networks,
making it interoperable across different vendors’ devices and network providers [12].
The LoRaWAN architecture can be divided into four main components as shown in Fig. 1:

• End Devices (Nodes): These are battery-powered IoT devices equipped with sensors or actuators. They collect data from the
environment and transmit it to the network server. End devices can operate in three classes (Class A, B, and C) with different
power and latency characteristics.
• Gateways: LoRaWAN gateways serve as the bridge between the end devices and the network server. They receive data from
the end devices and forward it to the network server using the LoRa modulation.
• Network Server: The network server is responsible for managing the entire LoRaWAN network. It handles device authentication,
data encryption, and data routing to the appropriate application server.
• Application Server: Application servers are responsible for securely handling, managing, and interpreting sensor application
data. They also generate all the application-layer downlink payloads to the connected end devices.

The communication in LoRaWAN follows a star-of-stars topology, where the end devices communicate with multiple gateways,
and gateways forward the data to the network server. The uplink (device-to-network) communication uses unlicensed ISM (Indus-
trial, Scientific, and Medical) bands, such as 868 MHz in Europe and 915 MHz in North America. The downlink (network-to-device)
communication is limited to minimize the power consumption of the end devices [11].
LoRaWAN supports three device classes with varying power and latency characteristics:

1. Class A: This is the most common class for end devices. It allows bidirectional communication, but it has the lowest power
consumption. After transmitting data, it opens two receive windows at specific time intervals to listen for potential downlink
data.
2. Class B: Class B devices extend Class A capabilities by periodically opening receive windows to receive downlink messages.
They synchronize with the network server for additional downlink slots at scheduled times.
3. Class C: Class C devices have the highest power consumption among the three classes because they keep their receive windows
continuously open, allowing for almost instantaneous downlink communication.

Another important aspect of LoRa and LoRaWAN is their operational parameters, which are critical when evaluating network
performance, as these parameters are used to create test scenarios and performance evaluation methods. They are as follows:

• Spreading Factor (SF): LoRa uses a technique called Chirp Spread Spectrum modulation, and the spreading factor determines
how much the signal spreads in the frequency domain [13]. Higher spreading factors provide a longer range but lower data
rates, while lower spreading factors offer higher data rates but a shorter range.
• Bandwidth (BW): The bandwidth represents the range of frequencies used to transmit the signal. LoRa supports different
bandwidth options, typically ranging from 125 kHz to 500 kHz [11]. A narrower bandwidth generally results in better receiver
sensitivity and longer range.
• Coding Rate (CR): The coding rate represents the amount of error correction used in the transmission. It is usually denoted as
CR4/N, where 𝑁 is the number of parity bits used for error correction [14]. A higher coding rate provides better resistance
to noise and interference at the expense of reduced data throughput.

3
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

• Frequency Bands: LoRa operates in various frequency bands, including 433 MHz, 868 MHz, and 915 MHz, among others. The
specific frequency band used may vary depending on the regulatory region.

LoRaWAN can be deployed by public network providers, private enterprises, or even individual enthusiasts for specific
applications. Public network providers deploy LoRaWAN infrastructure to cover large areas and support a broad range of IoT use
cases. Private networks are suitable when organizations need to have full control over the infrastructure and data [15]. Moreover,
LoRaWAN finds applications in various domains, including smart cities, agriculture, asset tracking, environmental monitoring,
industrial automation, and more. Its long-range capabilities, low power consumption, and cost-effectiveness make it an attractive
choice for many IoT use cases, especially those that involve remotely located devices or large-scale deployments [16]. As technology
continues to evolve, it is expected that further advancements and improvements will be made to LoRa and LoRaWAN, making them
even more integral to the future of IoT [17].

2.2. Testing methodologies and performance evaluation metrics

When it comes to testing LoRa and LoRaWAN implementations, various methodologies are employed to ensure the reliability
and performance of the network. It is important to note that LoRa and LoRaWAN test cases and evaluation methods may vary
based on the specific application and deployment scenario. A combination of testbeds, simulation, and mathematical modeling
is often employed to comprehensively evaluate the network’s performance under various conditions [9,18]. Regular updates and
improvements to the network may also require retesting to ensure ongoing reliability and functionality.
Additionally, LoRa and LoRaWAN have specific performance metrics that are essential for evaluating their effectiveness in IoT
deployments. LoRa’s performance is evaluated using the range, sensitivity, data rate, spectral efficiency, and battery life to name a
few. On the other hand, LoRaWAN is evaluated in terms of network capacity, latency, Quality of Service (QoS), security, adaptability,
and reliability to name a few [19]. In addition, it is important to note that actual performance may vary depending on factors like
network topology, interference, radio conditions, and the specific implementation of LoRa and LoRaWAN in a given IoT deployment.
When planning and designing an IoT system, it is crucial to consider these performance metrics to ensure the technology meets the
requirements of the IoT application [20].
Below is a list of typical network performance metrics used to evaluate LoRa and LoRaWAN [21]:

• Received Signal Strength Indicator (RSSI): It is used to measure the strength of the signal received by a LoRaWAN device or
gateway. RSSI provides information about the power level of the radio signal that arrives at the receiver after traveling through
the wireless medium.
• Signal-to-Noise Ratio (SNR): It is a measure of the strength of the received signal compared to the level of background noise
present in the communication channel. It is a crucial parameter that determines the quality and reliability of data transmission
in LoRaWAN.
• Packet Delivery Ratio (PDR): It indicates the percentage of successfully received packets at the gateway. A high PDR is necessary
to ensure reliable data transmission in various IoT applications.
• Data Rate and Throughput : Data rate refers to the speed at which information can be transmitted over the LoRaWAN network.
Throughput, on the other hand, is the actual amount of data that can be transmitted in a given time. Higher data rates and
throughput are desirable for applications requiring faster data exchange.
• Packet Error Rate (PER): It represents the ratio of the number of packets that were received with errors to the total number of
packets transmitted.
• Symbol Error Rate (SER): It measures the probability that a received symbol is incorrectly detected or decoded.
• Battery/Energy Efficiency: IoT devices are often battery-powered and expected to operate for an extended period. Battery life is a
crucial metric to evaluate the network’s efficiency in minimizing energy consumption and maximizing the device’s operational
lifetime.
• Time of Air (ToA): It refers to the time duration that a LoRaWAN transmission consumes when it is sent over the airwaves and
affects battery life and network capacity.
• Delay/Latency: It refers to the time taken for a packet to travel from the device to the gateway and back. Low latency is
essential for real-time or time-critical applications.
• Accuracy: It is a critical factor when determining the precision in determining the geographical position of an end device or
sensor within a LoRaWAN network.

3. Motivation and related work

This section presents the state-of-the-art review and survey papers on LoRa and LoRaWAN implementation and evaluation for
IoT applications. Additionally, a summary of papers based on the year of publication, contributions, and limitations of each work
is presented at the end of the section.
A comprehensive paper by Marais et al. tried to review created testbeds of LoRa and LoRaWAN and categorized them into
outdoor, indoor, and computer simulations [22]. This work also enumerated and defined the different testbed and other situational
evaluation metrics. In addition, the paper presented a comparison of the different testbed implementations in terms of the number
of gateways, nodes, SF, BW, frequency, distance, and payload, to name a few [22]. However, the paper did not cover other critical
aspects of testing, such as the cases and scenarios.

4
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Another survey focused on reviewing and analyzing LPWAN technologies for IoT applications. It identified other competing
technologies that can be investigated in terms of their limitation and their prospect augmentation capability to LPWAN. Furthermore,
the concept of LPWAN technologies is discussed and suggested as a potential solution for long-range IoT connectivity [23]. However,
the review paper is too general and does not focus on a preferred single technology that is used more frequently than others for
various IoT applications.
In another work, an in-depth comparison of different LPWAN technologies, including alternatives to data link and network
layers over LoRa other than LoRaWAN, are presented and discussed. This short survey also provided potential application scenarios
and major advantages of each technology [24]. Similarly, another paper reviewed LPWAN technologies and compared them by
evaluating the performance using different metrics, including coverage, bandwidth, and cost, to name a few [25]. However, neither
paper focused on a single technology to provide a more in-depth analysis when it comes to testing and evaluation.
LoRaWAN architecture, applications, and security concerns were reviewed in another study. In this paper, the attack vector
associated with the LoRaWAN vulnerabilities was explained and the authors proposed possible countermeasures to guard against
the different security attacks [26]. Another paper presented a systematic review of state-of-the-art works for LoRaWAN optimization
solutions for IoT networking operations by Silva et al. [27]. Additionally, the paper presented a taxonomy covering five aspects
related to LoRaWAN optimizations for efficient IoT networks. However, both survey papers were not able to cover test procedures
to ensure secure and reliable connectivity using LoRaWAN.
A survey comparison of LPWAN technologies namely, NB-IoT, LTE Cat-M1, Sigfox, and LoRa testbeds was presented. In the
comparison, a drone flying at 70 km/hr along a 10 km motorway was used to act as a mobile IoT end device to do the performance
measurements among the available technologies [28]. However, the paper only focused on the high-speed mobile conditions in the
air performance and not on the test case scenario itself.
Another review paper by Almuhaya et al. partly presented a comparison of the most important simulation tools for investigating
and analyzing LoRa and LoRaWAN network performance that has been developed recently. The comparison used several features
such as license, operating system, language type, popularity, and versions, to name a few [29]. However, this work only focused
on computer simulators in testing and evaluating the performance of LoRa and LoRaWAN.
In another article by Pagano et al. a survey on the adoption of LoRa in the agricultural field was provided, and it reviewed
state-of-the-art solutions for smart agriculture, analyzing the potential of this technology in different infield applications [30]. The
paper analyzed the main challenges of using LoRa Technology in smart agriculture and includes latency on the downlink channel,
energy management, heterogeneity and interoperability of the devices, data management, and scalability. On the other hand, another
article focused on surveying LoRa-based platforms using a different application, specifically in localization. The article presented
different localization techniques along with the metrics employed [31]. Although the papers focused on a specific IoT application,
they were not able to cover how the LoRa and LoRaWAN deployment on various smart agriculture projects was tested and evaluated
in different scenarios.
The summary of the state-of-the-art survey and review papers on LoRa and LoRaWAN implementation and evaluation in IoT
applications is shown in Table 1. The table classifies the survey and review papers in terms of the publication year, their focus,
contributions, and limitations. In summary, the related survey and review papers focused mainly on comparing current LPWAN
technologies and evaluating the performance of LoRa and LoRaWAN platforms in either general or specific applications, or a single
test evaluation methodology. To the best of our knowledge, the literature does not include any articles dedicated to comprehensively
reviewing the current testing and performance evaluation methodologies used in LoRa-based or LoRaWAN-based IoT applications.
This survey article is novel as it only focuses on a single LPWAN technology and covers all aspects of testing and evaluation methods.

4. Search workflow and paper distribution

This section discusses the method used in conducting the search workflow and paper distribution of the current LoRa and
LoRaWAN testing and performance evaluation in IoT applications.
The methodology comprises three stages: searching, filtering, and analysis. Each stage is discussed as follows:

1. Stage 1 - Searching: In this stage, the research gap is defined to identify the focus of the study. The research gap is also the
basis for determining the appropriate and correct keywords and search strings to be used during the search. Several online
databases were used to search for the relevant papers.
2. Stage 2 - Filtering: In this stage, the most relevant papers are selected based on the reputation of sources, reliability of indexing,
and quality of content, while irrelevant papers are excluded.
3. Stage 3 - Analysis: In this stage, the critical information from the selected relevant papers is extracted and analyzed to fill the
research gap. The information retrieved is also used to establish new technical insights, identify issues and challenges, and
define future directions.

A keyword bibliometric search analysis in the combined Scopus and Web of Science (WoS) databases on the topics of ‘‘LoRa’’,
‘‘LoRaWAN’’, ‘‘test’’, and ‘‘IoT’’ resulted in a total of 73 papers, which are composed of 63 research articles and 10 survey papers.
The keyword co-occurrence diagram of the bibliometric search shown in Fig. 2 was generated using VOS Viewer.3

3 https://www.vosviewer.com/

5
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 1
Summary of state-of-the-art survey and review papers on LoRa and LoRaWAN in IoT.
Paper Year Focus Strength Limitation
Marais et al. 2017 LoRa and LoRaWAN testbed and Reviewed and classified LoRa and Lack of test methodologies
[22] identification of the evaluation LoRaWAN testbed evaluation analysis
metrics methods namely outdoor, indoor,
simulation, and comparisons
Bembe et al. 2019 Conventional low-power, long-range LPWAN associated modeling Insufficient focus on a
[23] network technologies that consider techniques, performance metrics, and single technology
IoT applications and requirements enablers
Queralta et al. 2019 Focused on a reduced number of In-depth comparison of LPWAN Insufficient focus on a
[24] technologies that represent LPWAN technologies, including alternatives single technology
technology - LoRaWAN as an open to link and network layers over LoRa
network standard other than LoRaWAN
Noura et al. [26] 2020 LoRaWAN architecture, applications, Performed risk assessment of the Lack of analysis on test
and security concerns, and a list of different attacks and proposed methodologies
several possible countermeasures to several security countermeasures
address existing vulnerabilities and
prevent related attacks
Gu et al. [25] 2020 Wireless communication approaches Explored different LPWA application Insufficient focus on a
in IoT and introduced LPWAN domains and presented how to single technology
technologies by evaluating choose LPWA technology to make
performance in different metrics the connectivity more efficient
including coverage, bandwidth, and
cost
Silva et al. [27] 2021 LoRaWAN performance, co-existence Taxonomy classification to categorize Lack of analysis on test
of IoT devices and applications, aspects that affect the LoRaWAN methodologies
resource allocation mechanisms, MAC operation
layer protocols, network planning,
and mobility issues
Wang et al. [28] 2022 Performance of LPWAN technologies Used a drone flying as mobile IoT Lack of analysis on test
in high-speed mobile conditions in end device to do performance methodologies
the air measurements
Almuhaya et al. 2022 Simulation tools for investigating and Classified recent efforts to improve Lack of comparison and
[29] analyzing LoRa/LoRaWAN network LoRa/LoRaWAN performance in analysis on testbed testing
performance and comparative terms of energy consumption, pure
evaluation data extraction rate, network
scalability, network coverage, quality
of service, and security
Pagano et al. 2023 LoRa-based smart agriculture Analyzed LoRa systems in terms of Lack of analysis on test
[30] systems, possible adoption of scalability, interoperability, network methodologies
Machine Learning, control architecture, energy efficiency, and
automation techniques, and energy open issues
autonomy features
Marquez et al. 2023 Localization techniques and metrics Provided criteria for comparing the Lack of analysis on test
[31] employed, challenges of LoRa performance of LoRa-based methodologies
technology, and comparative analysis localization algorithms
of unresolved issues
Alipio et al. (this 2023 Testing and evaluation methodologies Classification and unified view of test –
work) in LoRa and LoRaWAN-based parameters and evaluation
networks in IoT methodologies of LoRa and
LoRaWAN deployments in IoT

Of the 73 publications, 63 were critically surveyed and analyzed to address the research gap. 38 were published in journals and
25 were published in conference proceedings. Figs. 3 to 5 show the visualization results that represent the distribution of papers
in terms of publication year, database source, and country, respectively. The database sources include IEEEXplore, ScienceDirect,
MDPI, SpringerLink, and ACM Digital Library. For the publication period, this work considered the recent works published in the
last five years, from 2018 to August of 2023.

5. Review of testing and evaluation methodologies of LoRa and LoRaWAN in IoT applications

This section presents a review of the state-of-the-art testing and performance evaluation methodologies in LoRa and LoRaWAN
used for IoT applications. The review includes the general classification of the studies based on the testing parameters, testing
architecture, and evaluation methodologies. In addition, a baseline testing framework of LoRa and LoRaWAN is presented based on
the review of current test and evaluation methods. 43 out of 63 papers were included in this review and classification.

6
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Fig. 2. Network keyword analysis for ‘‘LoRa’’, ‘‘LoRaWAN’’, ‘‘test’’, and ‘‘IoT’’.

Fig. 3. Paper distribution based on publication year.

Tables 2 to 4 show the summary of the papers based on the following general classification: solution, contribution, IoT
application, test parameters, and evaluation method. The objective refers to the solution of the study, the contribution refers to
the novelty of the approach or system, and the application indicates the specific IoT system or deployment. Lastly, test parameters
pertain to the LoRa and LoRaWAN performance features while evaluation is the methodology used to assess the performance of
the network. The classification of studies based on the test parameters and evaluation methods are further discussed in the next
sections.

5.1. Testing parameters

This section discusses the different test parameters used and evaluated by the state-of-the-art studies on LoRa-based and
LoRaWAN-based IoT applications. The test parameter serves as the test objective in identifying the corresponding test and evaluation
methodologies. Based on the review, a classification of test parameters was established. They are as follows:

7
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Fig. 4. Paper distribution based on database source.

Fig. 5. Paper distribution based on country.

1. Range: This parameter evaluates the maximum range at which LoRa devices can effectively communicate with each other and
the gateway. It measures the signal strength and quality at different distances to determine the coverage area of the network.
The tests under various environmental conditions include obstacles and interference to assess real-world performance.
2. Reliability: This parameter validates the network’s ability to handle packet loss, interference, and retransmissions. It also
assesses the behavior of the network under stressful conditions, such as high device density.
3. Scalability: This parameter tests the LoRaWAN network’s ability to scale with an increasing number of devices. It evaluates
the impact of a large number of devices on network performance.
4. Functionality: This parameter involves testing the basic functionalities of LoRa and LoRaWAN devices, such as data
transmission, reception, and acknowledgment. It verifies that the devices can join the network, transmit data, and receive
data as expected. Additionally, it checks for correct handling of data formats, including payload size, MAC commands, and
application data.
5. Delay: This parameter measures the data delay or latency of the LoRaWAN network to ensure it meets the application
requirements. It also evaluates the network’s ability to handle data traffic from multiple devices simultaneously.
6. Energy Efficiency: This parameter assesses the power consumption of LoRa devices to optimize battery life. It measures
power usage during different operational modes (e.g., idle, transmit, receive) and duty cycles.
7. Geolocation: This parameter tests the accuracy and reliability of geolocation services based on LoRaWAN, which enables
locating and tracking of devices.
8. Security: This parameter verifies the implementation of security features like encryption and authentication to protect data
and prevent unauthorized access. It also conducts penetration testing to identify vulnerabilities and potential attack vectors.

8
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 2
Summary of state-of-the-art LoRa-based and LoRaWAN-based IoT applications.
Paper Objective Contribution Application Parameter Evaluation
Zhao et al. Underground-to-aboveground Quantitatively verified Underground Reliability; Simulation
[32] network-level simulator of LoRaWAN-based WUSNs and Monitoring Scalability
LoRaWAN-based Wireless impact of the underground
Underground Sensor Networks environment
(WUSN)
Miles et al. Performance of the LoRaWAN Mathematical model to predict Smart Reliability; Simulation;
[33] communication network in an IoT successful packet delivery rate of Agriculture Scalability Mathemati-
pilot farm LoRaWAN cal
Model
Carvalho Evaluation of LoRaWAN Evaluated timing performance of IoT - Delay Testbed
et al. [34] infrastructure performance in a real LoRaWAN infrastructure General
real-world scenarios
Dragulinescu End-to-end multi-purpose IoT Testing platform to accommodate IoT - Range Testbed
et al. [35] platform with LoRaWAN network LoRaWAN devices with sensing General
coverage testing capabilities from different vendors
Gauld et al. Miro-Nomad GPS tracker logger for New LoRa miniaturized device Animal Geolocation Testbed
[36] high-resolution tracking solution with GNSS position accuracy, Bioteleme-
using LoRa data transmission range, and try;
post-deployment performance Tracking
Pérez et al. Comparative evaluation of LoRaWAN Experiments conducted in urban IoT - Range; Testbed
[37] and SigFox based on coverage and and rural environments General Energy
energy-efficiency test performance Efficiency
de Camargo Evaluated commercial and Evaluation of LoRaWAN network Smart Cities; Range; Testbed;
et al. [38] non-commercial LoRa-based tracking deployed at a university campus Tracking Geolocation Simulation
devices and municipality
Marais et al. Outdoor permanent testbed Evaluation of LoRaWAN IoT - Range Testbed
[39] consisting of LoRa devices technology over long and short General
distances and over long time
periods in urban environments
Lopez et al. Performance of LoRaWAN-based Investigated performance of LoRa Smart Delay Testbed
[40] LPWAN technology for Building technology in terms of Time on Buildings
Information Modeling (BIM) systems Air (ToA)
under emergency scenarios
Elbsir et al. Evaluated performance of LoRaWAN Developed LoRaWAN Class B IoT - Functional Simulation
[41] class B to actuator-based IoT module on the NS-3 simulator General
applications
Saban et al. Measurement and evaluation of LoRa Performed testing indoors and IoT - Range Testbed
[42] and LoRaWAN outdoors, and investigated sensor General
placement effect
Dai et al. LoRaWAN-based IoT network on a Intra-site logistics and task Logistics Range Testbed
[43] reference construction site scheduling system to validate Automation
LoRaWAN in
Construction
Apriantoro Analysis of characterization of Presented a signal quality analysis IoT - Range Testbed
et al. [44] LoRaWAN coverage in urban areas in LoRaWAN network General
Dragulinescu LoRaWAN-based Medical IoT solution Optimal conditions and Medical; Range Testbed
et al. [45] for hospital and homecare parameters for homecare and E-Health
hospital LoRa networks
Bardram Simulation and field test for IoT LoRaWAN IoT solution and Harbour Reliability; Testbed;
et al. [46] systems in harbors capacity simulation and field test Environment Scalability; Simulation
Range
Paternina Network coverage of LoRaWAN in Testing methodology for IoT - Range Testbed
et al. [47] dense urban environments operational coverage of LoRaWAN General
by connection link quality
Prasad et al. Wireless jamming attacks causing Sensitivity analysis of jamming Security Reliability; Mathemati-
[48] fatal denial-of-services (DoS) on the and DoS attacks using Security cal
LoRaWAN mathematical simulation model Model
Michaelis Testing LoRa signal coverage in Focused on two testing Military Range Testbed
et al. [49] dense urban infrastructure perspectives - indoor and outdoor Communica-
transmission tions

9
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 3
Summary of state-of-the-Art LoRa-based and LoRaWAN-based IoT applications (cont’d).
Paper Objective Contribution Application Parameter Evaluation
Lombardo et al. Comparison test using a Characterized LoRaWAN and NB-IoT Critical Reliability; Testbed
[50] multiprotocol wireless sensor node facilities in multiple environments Environment Range
prototype
Feltrin et al. Technical limits of LoRaWAN from Assessed LoRaWAN network capacity IoT - General Range; Testbed;
[51] both link- and system-level in single and multicell scenarios in Capacity Simulation
realistic traffic conditions
Marquez et al. Empirical model of LoRa Determine the amount of Smart Cities Reliability; Testbed
[52] communication immunity region electromagnetic interference and Range
radio range
Bouras et al. Performance and application Comparative evaluation of LoRa and Search and Range Testbed
[53] development for rescue monitoring Wi-Fi networks in Search and Rescue Rescue
systems (SAR) scenario Monitoring
Ingabire et al. LoRaWAN radio network coverage at Radio coverage simulation model of Urban Range Testbed;
[54] various propagation models LoRaWAN in urban scenarios using Environment Simulation
several popular propagation models
Cappelli et al. Underwater maximum achievable LoRaWAN networking for real-time Underwater Range Testbed
[55] transmission depth data collection from underwater
depths
Saavedra et al. Universal testbed for wireless IoT Abstracted the IoT wireless IoT - General Functional Testbed
[56] technology protocols and hardware technology characteristics and
platforms compared wireless IoT technologies
Ould et al. [57] Power model for lowest power LoRa Formulaic power estimate to consider IoT - General Energy Testbed
module likely transmission characteristics, Efficiency
battery size, and transmission
interval of LoRa module
Alghamdi et al. Monitoring and detecting water Realistic network model to simulate Water Leak Reliability; Simulation
[58] leakage in a housing complex water monitoring and leakage Detection and Scalability;
detection Monitoring Energy
Efficiency
Harinda et al. LoRaWAN network over multiple Performance analysis of live IoT - General Range Testbed
[59] gateways at multiple sites carrying LoRaWAN multi-gateway in indoor
live traffic and outdoor measurements
Pirayesh et al. Multi-antenna LoRa gateway with LoRa PHY design with enhanced IoT - General Scalability Testbed
[60] multi-user MIMO (MU-MIMO) LoRa uplink concurrent packet detection
communications in both uplink and and enables downlink concurrent
downlink packet transmission
Xie et al. [61] Developed RobotSen utilizing a LoRa Proposed combining the sensing Wireless Range Testbed
device on a moving robot for capability of LoRa signals with the Sensing
wireless sensing of device motions mobility robots to support a larger
sensing coverage
Santana et al. Analyzed the behavior and Derivation optimal planning and Smart Range Testbed
[62] performance of a LoRaWAN network deployment of LoRaWAN for Parking
employed for supporting a smart supporting smart services in urban
parking service scenarios
Chen et al. [63] Contactless system for sensing human New algorithms and design Wireless Range Testbed
targets using drone with LoRa methodologies to address a series of Sensing
interference and sensing ambiguity
issues in LoRa
Busacca et al. Integrated platform for sharing of Hybrid underwater-terrestrial IoT Internet of Reliability Testbed
[64] data collected in an underwater with marine acoustic modems Underwater
scenario for remote monitoring and connected using LoRaWAN Things
control purposes
Mikhaylov et al. Investigated the robustness of Defined possible energy attack Energy Security Testbed
[65] LoRaWAN against energy depletion vectors and validated the feasibility Depletion
attacks of an energy attack over one of these Attack
vectors
Xie et al. [66] ChirpSen that takes full advantage of System that adopts chirp signal and Wireless Range Testbed
the LoRa chirp property to increase utilizes chirp multiplication to Sensing
sensing range concentrate power at a particular
frequency

10
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 4
Summary of state-of-the-Art LoRa-based and LoRaWAN-based IoT applications (cont’d).
Paper Objective Contribution Application Parameter Evaluation
Subbaraman Busy Signal Multiple Access First in-band full-duplex LoRa Urban Scalability Testbed;
et al. [67] where the LoRa gateway gateway that can simultaneously Environment Simulation
transmits downlink busy signal receive and transmit LoRa packets
while receiving uplink over the same channel
transmission
Jiang et al. [68] New sensing technology for First-of-its-kind long-range sensing Wireless Range Testbed
cyclists in the public bicycle system based on the LoRa Sensing
sharing systems backscatter technology
Wang et al. [69] Node authentication and Strategy exploiting hardware IoT - General Security Testbed
improved security performance of imperfections of low-cost
LoRa communications components in LoRa radios
Li et al. [70] Neural-enhanced demodulation Deep Neural Networks to extract Campus-scale Range; Testbed
method that achieves ultra-low the fine-grained information Environment Energy
SNR LoRa communication with a embedded inside the chirps for Efficiency
single gateway decoding LoRa
Shahid et al. Demodulation technique that Demodulation technique that IoT - General Reliability Testbed
[71] decodes multiple concurrent LoRa enables decoding of multiple
transmissions by canceling colliding packets
symbols from interfering
transmissions
Tong et al. [72] Addressed the link dynamics by Low-power channel activity Urban Reliability Testbed
enabling data transmission for detection method to detect other Environment
very low SNR or even LoRa transmissions for selective
disconnected LoRa links interference
Liu et al. [73] Accurate localization for Utilized channel state information Indoor and Geolocation Testbed
LoRaWAN devices both indoors of multiple channels as Outdoor
and outdoors incoherent measurements and Localization
super-resolution algorithms on
multiple gateways
Xia et al. [74] Addressed limited link capacity of Novel technique that tunes the IoT - General Capacity Testbed
LoRa and the diverse bandwidth number of modulated symbols to
requirements of IoT systems adapt bit-rates according to
channel conditions
Leonardi et al. Multi-hop protocol to provide System that enables long-range Industrial IoT Reliability Simulation
[75] bounded delays to real-time flows communications while
for industrial applications maintaining shorter ToA at each
hop
Taleb et al. [76] Monitoring healthcare system Method to select the most Medical; Energy Simulation
using LoRa to adopt data convenient SF based on the E-Health Efficiency
transmission in reliable and patient’s medical state
energy-efficient way
Rodic et al. [77] Analyzed side-channel attacks Simulated privacy leakage of Smart Security Testbed;
from vehicles as metallic obstacles LoRaWAN smart parking Parking Simulation
obscuring smart parking sensors communication devices in terms
of passive side-channel attacks at
large distance

9. Capacity: This parameter assesses the network’s performance under heavy load conditions to determine its capacity and
potential limitations.

Fig. 6 shows the percentage distribution of the test parameters used to measure the performance of LoRa and LoRaWAN in various
IoT systems. It should be noted that some of the studies have evaluated a single parameter while other studies have evaluated a
combination of two or more parameters. Using multiple evaluation methods can result in a more comprehensive, robust, and reliable
network evaluation.
The majority of the studies comprising 52.17% evaluated the maximum range parameter of their LoRa and LoRaWAN. They
showed that by testing the maximum range, potential coverage gaps or areas with weak signal strength can be identified. That,
therefore, allows for the optimization of the placement and configuration of gateways and repeaters to improve overall network
performance. Another common test parameter is reliability, which comprises 23.91% of the reviewed studies. This reveals that
reliability is another critical parameter to evaluate to ensure that the signals can reach their intended destinations consistently and
maintain a strong and stable connection over distances, especially in challenging environments with obstacles and interference.
Furthermore, other studies performed comparisons of LoRa and LoRaWAN with other LPWAN technologies. Based on the
review, 21% of the papers performed a comparative test among other wireless technologies. These wireless technologies are SigFox,

11
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Fig. 6. Paper distribution based on test parameters.

Narrowband-IoT (NB-IoT), WiFi, Zigbee, Bluetooth Low Energy (BLE), and 6LoWPAN. This comparative test is very useful as each
of these technologies has its strengths and weaknesses, and understanding their differences helps in selecting the most suitable
technology for specific IoT applications.

5.2. Testing architectures and evaluation methodologies

This section presents the testing architectures and evaluation methodologies used by state-of-the-art studies employing LoRa and
LoRaWAN technologies in IoT systems.
Tables 5 to 6 show the summary of the studies based on the testing architectures employed in terms of LoRaWAN class, number
of end devices, number of gateways, number of network servers, number of application servers, the mobility of end devices, and
transmission range. From this summary table, it can be seen that most studies used class A type of LoRaWAN mainly because of
its very low power consumption. It can also be deduced that the majority of the studies have used either a single end-device or
multiple end-device communicating to a single gateway in a star topology manner. In terms of mobility, a significant number of
studies have evaluated the performance of LoRa and LoRaWAN in both fixed and mobile end-device scenarios. Lastly, 53 km was
the longest distance the LoRa and LoRaWAN were evaluated which was applied to animal movement tracking and localization.
In terms of the evaluation methodology, the studies can be classified into a testbed, simulation, mathematical modeling, or a
combination of methods. Fig. 7 shows the percentage distribution of the evaluation methodologies used to evaluate the performance
of LoRa and LoRaWAN deployed in different IoT applications. It is shown that 72% have solely used actual testbed implementation,
11% have solely used simulation using computer simulation platforms, and 2% have solely used mathematical modeling. The
remaining studies opt to perform a combination of two methods that comprise 15% of the surveyed studies.
Based on the review, the testbed is the most utilized method among the studies as it assesses the actual performance of LoRa
and LoRaWAN devices and gateways in a specific location or scenario. It also helps to understand how the LoRa and LoRaWAN
perform under different environmental conditions, interference, and distance. However, setting up and maintaining a testbed can
be resource-intensive and expensive [78]. It may also be challenging to replicate and control specific scenarios consistently. Hence,
other studies have used computer-based network simulations. Simulations can be easily scaled up to simulate large-scale LoRa and
LoRaWAN deployments, which might be impractical or expensive with a physical testbed. In addition, it allows precise control of
the conditions and variables to focus on specific aspects of the evaluation. However, simulations are only as accurate as the models
they are based on. It is essential to validate the simulation results against real-world data to ensure their reliability [79].
Other studies used a combination of both approaches which was also beneficial. Simulations can be used for initial testing and
parameter tuning, while a physical testbed can provide crucial real-world validation. Overall, the choice of evaluation methodology
depends on the specific research or evaluation objectives, funding, time constraints, and available resources. In the following
sections, these performance evaluation methodologies are further discussed and analyzed.

5.2.1. Testbed-based evaluation

This section presents the studies that used testbed as a method to evaluate the performance of LoRa-based and LoRaWAN-based
IoT systems. Tables 7 to 9 show the summary of state-of-the-art studies based on how the systems were evaluated using actual
testbed implementation and deployment. The studies are classified based on the general network scenario, gateway device model,
end node device model, application (app) server, test cases, and performance metrics.

12
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 5
Summary of state-of-the-art LoRa-based and LoRaWAN-based IoT applications based on testing architectures.
Paper Class End device Gateway Network server Application server Mobility Distance (km)
Zhao et al. [32] A 100 1 – – No 0.05
Miles et al. [33] A 10–2000 1 – – No 7.5
Carvalho et al. [34] – 1 3 1 1 No –
Dragulinescu et al. – 1 9 – – Yes –
[35]
Gauld et al. [36] – 4 2 1 1 Yes 40.7–53
Pérez et al. [37] A 1 1 – – No 3–8 (urban);
15–22 (rural)
de Camargo et al. A 4 1 – – Yes 3
[38]
Marais et al. [39] – 18 1 – – No 0.50–5.19
Lopez et al. [40] – 1 1 1 1 No 0.01
Elbsir et al. [41] B 1000–4000 1 – – No 8
Saban et al. [42] A 1 1 1 1 No 0.05–0.803
Dai et al. [43] A 3 3 1 1 No 5–21
Apriantoro et al. A 1 43 1 1 No 4
[44]
Dragulinescu et al. A 2 1 1 1 No 0.217; 0.735
[45]
Bardram et al. [46] A 1 (testbed); 1500 1 1 1 No 0.004–0.591
(simulation)
Paternina et al. [47] A 1 2 1 1 No 9.2
Prasad et al. [48] – 1 1 – – No 2
Michaelis et al. [49] – 5 1 – – Yes Up to 2
Lombardo et al. A 1 3 – – No –
[50]
Feltrin et al. [51] A 1 (testbed); 1 – – No Up to 10.8
Undefined
(simulation)
Marquez et al. [52] – 1 1 1 1 No 0.378–6.810
Bouras et al. [53] – 1 1 1 1 Yes 2–5
Ingabire et al. [54] – 1 3 – – Yes Up to 2.275
Cappelli et al. [55] A 1 1 1 1 No –
Saavedra et al. [56] – 1 1 1 1 No –
Ould et al. [57] – 1 1 – – No –
Alghamdi et al. [58] A 3 1 1 1 No –
Harinda et al. [59] A 8 2 1 1 Yes 1.9; 2.1
Pirayesh et al. [60] – 6 1 – – No –
Xie et al. [61,66] – 1 1 1 1 Yes 0.003–0.010
Santana et al. [62] A 24–45 10 1 1 No 0.025–0.4
Chen et al. [63] – 1 1 1 1 Yes 0.001–0.0035
Busacca et al. [64] A 1 1 1 1 No –
Mikhaylov et al. A 1 2 1 1 No –
[65]
Xie et al. [61,66] – 1 1 1 1 Yes 0.001–0.055
Subbaraman et al. – 10–1000 1 1 1 No Up to 2.5
[67]

In Tables 7 to 9, it can be deduced that most of the studies used the general LoRaWAN architecture composed of at least an end
device and a receiving gateway in the network implementation and deployment. The most common end devices are Arduino-based4

4 https://docs.arduino.cc/learn/communication/lorawan-101

13
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 6
Summary of state-of-the-art LoRa-based and LoRaWAN-based IoT applications based on testing architectures (cont’d).
Paper Class End device Gateway Networkserver Application server Mobility Distance (km)
Jiang et al. [68] – 1 1 1 1 Yes 0.01–0.1
Wang et al. [69] – 1 1 1 1 No Up to 0.4
Li et al. [70] – 1 1 1 1 No 1.2–1.7
Shahid et al. – 20 1 1 1 No 0.01–1.6
[71]
Tong et al. [72] – 1–10 1 – – No –
Liu et al. [73] – 1 4 1 1 No 0.002–0.0116
Xia et al. [74] – 39 1 – – No 0.05–1.4
Leonardi et al. – 18–54 1 – – No 0.125–0.535
[75]
Taleb et al. [76] – 1 1 1 1 No –
Rodic et al. [77] – 5 (testbed); 1 3 (testbed); 1 1 1 No 0.004–0.145
(simulation) (simulation)

Fig. 7. Paper distribution based on evaluation methodologies.

and RPi-based5 boards mainly because of their modularity, inexpensiveness, and accessibility. On the other hand, Semtech-based6
boards are the popular device choice for gateway platforms as they provide a reliable and cost-effective solution for establishing
wide-area IoT networks that can support a variety of applications. For the application server, The Things Network (TTN)7 is the
most preferred option as it provides a set of open tools and a global, open network to build IoT applications at low cost, featuring
maximum security and ready to scale.
In terms of test cases, it is essential to design comprehensive test cases that cover various aspects of the network’s performance
and functionality. For instance, the majority of the studies have evaluated LoRaWAN in terms of range. This test evaluates the
LoRaWAN’s coverage area and range by testing devices in different locations within the network’s deployment area. In addition, the
test measures the signal strength, packet loss, and data rate at different distances from the gateway. Hence, it shows how popular
and effective the use of an actual testbed implementation is when assessing the coverage range of LoRaWAN in IoT.

5.2.2. Simulation-based and mathematical modeling evaluation

This section presents the studies that used simulation or mathematical modeling as a method to evaluate the performance of their
LoRa-based and LoRaWAN-based IoT systems. Table 10 shows the summary of state-of-the-art studies based on how the systems
were evaluated using computer-based network simulation or mathematical modeling methods. The studies are classified based on
the general network scenario, simulation software platform, test cases, and performance metrics.
Based on Table 10, the common network simulation platform used to evaluate LoRaWAN is NS38 with the addition of the
LoRaWAN module. This simulator is indeed a popular and powerful open-source simulation tool used for various network protocols.

5 https://learn.adafruit.com/lora-and-lorawan-radio-for-raspberry-pi/overview
6 https://www.semtech.com/lora
7 https://www.thethingsnetwork.org/
8 https://apps.nsnam.org/app/lorawan/

14
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 7
Summary of state-of-the-art LoRa-based and LoRaWAN-based IoT applications based on testbed evaluation.
Paper Scenario Gateway End device App Test cases Metrics
Carvalho et al. Infrastructure- Semtech LoRa – – Varying message Time Interval; Delay
[34] related Microchip RN2483 lengths at different
delay time intervals
Dragulinescu IoT architecture Semtech SX1272 – – Varying gateway RSSI; SNR
et al. [35] with Device, 8-channel Pygate positions at
Network, and Cloud different heights
Layers and locations
Gauld et al. [36] High resolution bird GNSS-LoRa – LORIOT and Varying GPS cycles Transmit
and mammal MultiTech Conduit TTN at different locations Probability;
movement tracking MTCAP-LEU1-868- and positions Accuracy
001A
mobile
Pérez et al. [37] Comparison in Multitech Conduit Pycom – varying distances RSSI; PDR; Energy
urban and rural IP67 BS LoPy/LoPy4 from gateway
areas
de Camargo Device tracking in Semtech SX1301 – TTN Varying locations Delay; PDR; RSSI;
et al. [38] indoor building while on mobility SNR
Marais et al. indoor and outdoor Multitech – – Varying locations PDR; RSSI; SNR
[39] residential and MultiConnect and positions
campus Conduit 3.3.9
Lopez et al. [40] Indoor emergency Dragino Arduino UNO TTN Varying Time-on-Air ToA
sensing LG308/LG01-N (ToA)
communication
parameters
Saban et al. [42] Indoor and outdoor iC880A LoRa WiMOD Demo TTN Varying gateway PDR; RSSI; SNR
campus and Board - iM880B locations and
metropolitan city LoRa positions
Dai et al. [43] Intra-site logistics 10-channel Dragino TTGO LoRa32 NodeRED Varying end device RSSI; SNR
and task scheduling DLOS8 GPS-LoRa board position between
system in two locations;
construction sites Varying gateway
height position on
the construction
crane
Apriantoro et al. Urban area Global Sat LT-100 Smartphone – Varying time RSSI; SNR
[44] GPS-LoRa periods of coverage
Dragulinescu IoT-based Medical LoPy v1 Pycom Expansion Board – Varying locations Coverage
et al. [45] data acquisition with IR and distance
MLX90614
Bardram et al. Harbour area LORANK8 Arduino UNO – Varying distance to Collisions; RSSI;
[46] gateway SNR
Paternina et al. University area Single-channel ATMega2560 TTN Varying operational RSSI
[47] Dragino LoRa-GPS parameters
SX1276/SX1278
Michaelis et al. Subways and MultiConnect PyCom LoPy4 – Varying gateway PDR
[49] vehicles in military Conduit IP67 locations and
environment positions
Lombardo et al. Comparison and RAK831 on RPi Nucleo-L073RZ – Varying RSSI; Losses
[50] characterization in with STM32L073 environment and
critical medium
environments
Feltrin et al. Link-level Semtech SX1272 Microchip – Varying distance, PDR; RSSI;
[51] performance RN2483 SF, and multiple Throughput;
characterization devices
Marquez et al. Smart cities outdoor LoRa RHF0M301 on Seeeduino LoRa – Varying distance, RSSI; SNR; PER
[52] deployment RPi locations, and
considering effect of positions
interferences
Bouras et al. Rescue monitoring FiPy on Espressif FiPy on Espressif TTN Varying SF and BW RSSI
[53] localization ESP32 Nano ESP32
Ingabire et al. Campus urban area SX1301-enabled MultiTech mDot – Varying distance RSSI
[54] Kerlink mote over RPi and positions

15
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 8
Summary of state-of-the-art LoRa-based and LoRaWAN-based IoT applications based on testbed evaluation (cont’d).
Paper Scenario Gateway End device App Test cases Metrics
Cappelli et al. Radio channel Multi-channel Microchip – Varying positions/depth RSSI
[55] transmission from RAKWireless RAK831 ATtiny84 with
underwater to above over RPi HopeRF RFM95x
water characterization
Saavedra et al. Temporal performance Semtech SX1301 over Pycom FiPy – External interrupts; Latency; Error
[56] comparison of IoT RPi internal clock; Rate; Stability
wireless technologies Self-delay; Error range
Ould et al. [57] Market-available LoRa – LoRa-E5 Mini – Power consumption and Energy
boards comparison Grasshopper profile stages over
LOPY4 period of time and
multiple runs
Harinda et al. Indoor and outdoor Indoor - Multitech Multitech mDot: – Varying positions, RSSI; SNR;
[59] area with interference Conduit Outdoor - locations, time of the Frequency
measurement Kerlink day, SF, and
frequencies
Pirayesh et al. Indoor and outdoor MaLoRaGW HELTEC ESP32 - – Varying locations and RSSI; PER;
[60] campus environment LoRa SX1276 SF Throughput
Xie et al. Indoor human Semtech SX1276 USRP X310 – Varying speed, Accuracy
[61,66] movement sensing Arduino Uno direction, trajectory,
mobility
Santana et al. Car park sensing Byda LoRaWAN-based – TTN-TTS Varying distance, RSSI
[62] Parking Sensors locations, and SF
Chen et al. [63] Wide-area human Semtech SX1276 LimeSDR-mini – Varying distance, Accuracy
movement sensing (Software- locations, frequency
Define)
Busacca et al. Underwater monitoring Dragino LG02 Dragino LoRa TTN Varying frame size, Throughput;
[64] shield + Arduino paylod size Transmission
UNO Time
Mikhaylov et al. Under energy attack MultiConnect Conduit; STM32F217 – Varying attack Energy
[65] SX1308-P868GW conditions Consumption
Picocell
Xie et al. Indoor detection Semtech SX1276 Dragino LoRa – Varying object detected Sensing Range;
[61,66] sensing shield + Arduino and interference Detection Range
UNO
Subbaraman City-wide outdoor SX1301 COTS SX1272 – Varying offered load Throughput;
et al. [67] deployment with bursty and distance Packet Reception
traffic Ratio
Jiang et al. [68] Public bicycle-sharing USRP N210 Semtech SX1276 – Varying distance, Accuracy (Error)
system velocity, motion
parameters, and
interference
Wang et al. [69] Node authentication USRP N210 Commercial – Varying transmit True Positive
and active attacks LoRa node power, frequency (legitimate) and
resolution, and distance False Positive
(attacker)
Li et al. [70] Campus-scale USRP N210 SX1278 – Varying SNR, SF, BW, SER; Battery Life
deployment covering and position
various landcover types
Shahid et al. Indoor and outdoor USRP B200 Adafruit Feather – Varying Line-of-Sight, Throughput;
[71] deployment M0 + RFM 95 location SER; Packet
(indoor/outdoor), and Detection Rate
aggregate transmit rate

However, it is worth noting that the suitability of any simulator depends on the specific requirements and applications. Other options
for simulation-based evaluation are MATLAB9 and Omnet++.10
In terms of test cases using simulation or mathematical modeling, it is essential to consider a variety of factors to ensure
comprehensive testing and accurate performance analysis. Based on the survey, scalability is a common parameter to evaluate

9 https://www.mathworks.com/matlabcentral/fileexchange/81166-loramatlab
10 https://omnetpp.org/download-items/FLoRA.html

16
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 9
Summary of state-of-the-art LoRa-based and LoRaWAN-based IoT applications based on testbed evaluation (cont’d).
Paper Scenario Gateway End device App Test cases Metrics
Tong et al. [72] Indoor and USRP N210 SX1278 – Varying SF, frequency, Processing Time; Packet
campus-wide urban CR, packet length, Delivery Ratio; SER;
environment position, and location SNR; Energy Cost
Liu et al. [73] Indoor and outdoor BladeRF 2.0 SDR Raspberry Pi – Varying localization Accuracy (Error)
deployment 4 estimation error
Xia et al. [74] Indoor and outdoor USRP N210 SX1276 – Varying parameters, Bit Rate; SER; Goodput;
deployment traffic, and range Energy per Bit; Packet
Reception Ratio
Rodic et al. [77] Car park sensing RPi with iC880A and Libelium TTN Varying car occupancy SNR; RSSI; Power
RAK831

Table 10
Summary of state-of-the-art LoRa-based and LoRaWAN-based IoT applications based on simulation and mathematical modeling evaluation.
Paper Scenario Platform Test cases Metrics
Zhao et al. [32] Communication from LoRaSIM (Python) Varying underground Goodput; Energy
underground to above with SimPy Library depth, number of end per Packet
ground nodes, distance, and
underground environment
Miles et al. [33] IoT-based smart agriculture NS3 with LoRaWAN Varying number of nodes PDR
in pilot farm module and transmission interval
Elbsir et al. [41] LoRaWAN Class B NS3 with LoRaWAN Varying types of losses; PDR; Delay
optimization module Class B Impact of internal collision
and RX2 conflict
Bardram et al. [46] Capacity and collision Matlab Varying number of devices, Collision, RSSI, SNR
behavior of LoRaWAN in SF, size of message,
harbor areas number of messages, and
packet size
Prasad et al. [48] Wireless jamming attacks Modeling – PER
that cause fatale
Denial-of-Services
Feltrin et al. [51] Characterization of Event-driven Capacity of small area RSSI; Throughput;
system-level performance simulator (varying offered traffic); PDR
coverage of large area
(varying number of GWs);
capacity of large area
Ingabire et al. [54] Non-LOS Coverage in ATDI ICS Telecom Varying distance and error RSSI
urban environment - radio planning and parameters
university areas spectrum
management tool
Alghamdi et al. [58] Water monitoring and FloRa on OMNeT++ With and w/o water PDR; Energy;
leakage detection system leakage while varying Throughput;
network size and SF Collisions
Subbaraman et al. [67] City-wide outdoor Matlab Increasing number of end Throughput; Packet
deployment with bursty devices Reception Ratio
traffic
Leonardi et al. [75] Distributed measurement OMNet++ Varying SF and distance Packet Loss Ratio;
system Deadline Miss Ratio;
End-to-End Delay
Taleb et al. [76] Healthcare system for vital MIMIC Varying SF and distance Energy Consumption
sign monitoring
Rodic et al. [77] Car park sensing Wireless InSite Varying Time-of-Arrival SNR; RSSI; Power

in LoRaWAN-based IoT systems. It can be deduced that the test cases used were able to assess the network’s ability to scale not only
in terms of the number of devices but also in terms of geographic coverage and increasing data traffic. Hence, simulation-based
evaluation is a great choice when it comes to obtaining valuable insights into the performance and behavior of the LoRaWAN
network for IoT applications and making informed decisions for optimizing its design and operation.

5.3. Testing and evaluation on improving LoRa and LoRaWAN MAC layer features

This section extends the survey paper to classify the test and evaluation methodologies of state-of-the-art mechanisms in
improving the MAC layer features of LoRa and LoRaWAN. The MAC layer features are responsible for managing the communication

17
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Fig. 8. Paper distribution based on evaluation methodologies of the MAC layer features.

between end devices and gateways, and their performance directly impacts the overall functionality and efficiency of the network.
The importance of these features in evaluating the performance of LoRa and LoRaWAN lies in optimizing network efficiency,
minimizing energy consumption, and ensuring equitable access for all devices in any IoT and LPWAN applications. These features
are as follows:

• Scheduling: Efficient scheduling of communication tasks is essential in LoRa and LoRaWAN to optimize network utilization,
reduce energy consumption, and minimize latency. Proper scheduling ensures that devices transmit their data at the right
time, avoiding congestion and collisions.
• Synchronization: Synchronization is critical to ensure that devices within a LoRaWAN are coordinated in time and frequency,
which helps in avoiding interference and collisions. Inaccurate synchronization can lead to data loss and inefficient network
operation.
• Collision Avoidance: Collision avoidance mechanisms are essential to prevent data packet collisions in LoRa and LoRaWAN.
Collisions can result in the loss of valuable data and can lead to higher energy consumption as devices attempt retransmissions.
• Variable Payload: In IoT applications, data payloads can vary in size and content. LoRa and LoRaWAN should support variable
and dynamic payload lengths to accommodate different types of devices and applications. Adaptive payload handling enhances
network efficiency.
• Fairness: Fairness in LoRa and LoRaWAN ensures that all devices have a reasonable opportunity to transmit their data,
preventing some devices from monopolizing network resources. Fairness is crucial for ensuring equitable service to all devices.

17 out of 63 papers were included in this extension. Fig. 8 shows the paper distribution based on the evaluation methodologies
of the MAC layer features in LoRa and LoRaWAN. Either testbed or simulation methodologies are the most utilized evaluation for
MAC layer features with 41% of the papers while the remaining mechanisms have used the hybrid approach.
Tables 11 to 12 show the summary of the state-of-the-art mechanisms and protocols in enhancing MAC layer features of Lora
and LoRaWAN in terms of test and evaluation methodologies. Based on the summary, important points need to be considered:

• The evaluation of scheduling mechanisms involves assessing their ability to allocate time slots, frequencies, and data rates
effectively. A well-designed scheduling algorithm can improve network capacity and reliability.
• Evaluating synchronization mechanisms involves testing the accuracy and stability of time and frequency synchronization.
This may include assessing the impact of clock drift and the ability to maintain synchronization over time.
• Evaluating collision avoidance techniques involves assessing their effectiveness in reducing collision rates, optimizing network
throughput, and minimizing the energy required for retransmissions. It also involves analyzing how well these mechanisms
adapt to network load and interference.
• The performance evaluation should consider the ability of LoRa and LoRaWAN to handle variable and dynamic payloads
efficiently. This includes assessing the impact of payload size on transmission efficiency and energy consumption.
• Fairness evaluation involves examining how well LoRa and LoRaWAN allocate network resources among devices, particularly
in scenarios with varying traffic loads and different classes of service (e.g., Class A, B, and C in LoRaWAN). It is essential to
assess whether all devices receive fair access to the network.

18
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 11
Summary of state-of-the-art mechanisms in enhancing LoRa and LoRaWAN MAC layer features.
Paper Solution Feature Evaluation Scenario Metric
Zorbas et al. Time-slotted approach supporting Scheduling; Testbed; Increasing number Execution time;
[80] collision-free transmissions as Collision Avoidance; Simulation of end devices; Guard time; PDR;
well as re-transmissions in case of Synchronization varying distance Energy Consumption
packet losses and locations
Triantafyllou FCA-LoRa that enables a fair, Scheduling; Fairness Simulation Increasing number Throughput;
et al. [81] scalable, and reliable scheduling of end devices Collision; PER;
procedure Energy
Consumption;
Channel Utilization
Muthanna et al. Markov Decision Process to Variable Payload Simulation Varying BER and BER; Energy
[82] formulate the transmission frame payload Consumption; SNR
parameter selection problem for length
static and mobile LoRa nodes
Garrido-Hidalgo Multi-agent-based network Scheduling; Simulation Increasing number Channel Utilization;
et al. [83] management architecture for Synchronization of end devices Downlink Usage
flexible allocation of resources in
multi-SF LoRaWAN
Ferrari et al. Comparative analysis of LoRa-REP Collision Avoidance Simulation Varying maximum Failure Probability;
[84] scalability and customizes delay Transaction Delay
standards slotted delay strategies
to specific LoRA-REP scenarios
Garrido-Hidalgo End-to-end solution providing Scheduling: Testbed Varying Clock Skew and
et al. [85] experimental clock drift Synchronization synchronization Drift; PDR; Packet
measurements and validation of a period and SF Lost;
LoRa-based synchronization and Synchronization
scheduling scheme Error; Number of
Timeslots
Gao et al. [86] EF-LoRa that allocates resources Fairness Simulation Increasing number Energy Efficiency;
in multi-gateway LoRa networks of end devices and Network Lifetime;
to realize fairness in energy gateways Energy Fairness;
efficiency among end devices Convergence Time
Ahmar et al. CRAM MAC protocol that delivers Synchronization Testbed; Increasing number Channel Utilization;
[87] slow-overhead network Simulation of end devices; Packet Loss Ratio;
coordination and jamming Varying SF PDR; Energy
resilience using cryptographic Consumption
frequency hopping
Wang et al. [88] Paralign LoRa collision decoder Collision Avoidance Testbed Increasing number SER; Throughput
supporting decoding LoRa of concurrent end
collisions with confusing symbols devices
via parallel alignment
Mai et al. [89] Minimized latency multi-hop Collision Avoidance Testbed Varying number of Delay; Number of
protocol that is collision-free with end devices Timeslots
low latency
Sallum et al. Fine-tuning specific radio Collision Avoidance Simulation Increasing number Data Extraction
[90] parameters through a Mixed of end devices Rate; Number of
Integer Linear Programming Collisions; Energy
problem formulation Consumption
Rajab et al. [91] Accurate and efficient way to Scheduling; Simulation; Increasing number Probability of
calculate the probability of Collision Avoidance Mathematical of end devices Collisions
collision rate and packet loss Modeling
under various circumstances
Centenaro et al. Disruptive approach exploiting Variable Payload Simulation Increasing number Average Number of
[92] novel operational mode at the of end devices Transmissions
gateways which fully uses the
degrees of freedom of the
unlicensed bands’ regulations

Testing and evaluating the performance of these features is essential to ensure the reliability, efficiency, and overall effectiveness
of IoT networks using LoRa and LoRaWAN technologies. It helps address specific challenges related to long-range, low-power
communication, and it ensures that the network meets the requirements of various IoT applications and regulatory standards.

19
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Table 12
Summary of state-of-the-art mechanisms in enhancing LoRa and LoRaWAN MAC layer features (cont’d).
Paper Solution Feature Evaluation Scenario Metric
Benkahla et al. [93] New time allocation algorithm Scheduling; Variable Testbed Varying locations Time on Air
based on classification of different Payload
requests according to QoS
requirements
Hou et al. [94] New protection method that can Synchronization; Testbed Varying transmit Packet Reception
separate LoRa chirps from Collision Avoidance power of jammer, Rate; Throughput
jamming chirps by leveraging SF, BW, and CR
their difference in power domain
Tong et al. [95] Decompose concurrent Collision Avoidance Testbed Varying SNR, SF, SER; Throughput
transmissions by leveraging subtle and Concurrency
interpacket time offsets for low
SNR LoRa collisions
Wu et al. [96] Algorithm for recovering distorted Variable Payload Testbed Increasing number PDR; Throughput;
packet payload, which involves of gateways and Energy Consumption
exclusive-OR based bitwise varying SF
inversion, majority voting based
bitwise inversion, and weighted
bitwise decision operations

Fig. 9. Summary of test and performance evaluation methodologies in LoRa-based and LoRaWAN-based IoT applications.

5.4. Summary and unified view on test and evaluation methodologies for LoRa-based and LoRaWAN-based networks in IoT application

Based on the review and analysis of state-of-the-art studies on LoRa and LoRaWAN in various IoT deployments, a summary of
testing and performance evaluation methodologies is presented as shown in Figs. 9 and 10. This unified view is the result and the
summary of the analysis of the literature review conducted by this work.
The test and evaluation methodologies are summarized and organized into four phases:

1. Identifying Test Objective: This phase provides a clear and specific purpose for conducting the performance evaluation. It helps
in understanding what aspects of LoRa and LoRaWAN performance are being assessed and what goals they want to achieve
through the evaluation. In this phase, appropriate test parameters are identified.
2. Verify Test Environment : This phase verifies the test environment which is a critical aspect of performance evaluation, as it
is used to conduct performance tests. The accuracy and reliability of the test environment are essential to obtain meaningful

20
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Fig. 10. Summary of test and performance evaluation methodologies in enhancing MAC layer features in LoRa-based and LoRaWAN-based IoT applications.

and actionable performance evaluation results. In this phase, a suitable evaluation methodology is verified which should be
aligned with the test objective. This includes testbed, simulation, modeling, and a combination of two methods.
3. Create and Execute Test Scenarios: By creating realistic scenarios, it can simulate how the system would perform under typical
and peak load conditions and provides an understanding of how the LoRaWAN behaves in the context of actual device
interactions and traffic patterns. In this phase, test cases are created following the test parameters and evaluation method.
These cases are also variables that are supposed to be varied during the testing and evaluation process.
4. Analyze Test Results: Analyzing test results in performance evaluation is of paramount importance as it provides valuable
insights into the effectiveness and efficiency of LoRa and LoRaWAN. The metrics serve as quantitative measures that help
assess various aspects of LoRaWAN performance and can guide decision-making, improvements, and optimizations in IoT
deployments. In this phase, appropriate metrics are selected and analyzed concerning the test parameters, evaluation method,
and test scenarios.

Figs. 9 and 10 aim to provide an organized approach to consolidate the test and evaluation methodologies to assess the
performance characteristics of LoRa and LoRaWAN under various conditions when deployed in IoT-driven applications. By defining
test cases and methodologies, this summary can promote fairness and objectivity in performance evaluation. Finally, this unified
view encourages a cycle of continuous improvement and for the researchers to identify test and evaluation trends, patterns, and
recurring issues over time, leading to the implementation of measures that optimize the LoRa and LoRaWAN efficiency for connecting
IoT devices. Some of these critical issues and potential future directions are discussed in the next sections.

6. Issues and challenges

This section discusses the issues and challenges in designing and implementing test scenarios and cases to evaluate the
performance of a LoRaWAN-based network in IoT. Understanding the challenges in LoRaWAN testing and deployment allows
for building a more reliable network infrastructure. It ensures that the network operates as expected, with minimal downtime or
connectivity issues for connected devices. Moreover, this enables the optimization of the LoRaWAN’s performance. This may involve
adjusting parameters, selecting appropriate hardware, or fine-tuning configurations to achieve better data transmission rates and
coverage [97]. The issues and challenges concerning test scenarios and evaluation methods are as follows:

• Coverage and Range: LoRaWAN is designed to provide long-range communication, but its performance heavily depends on
the specific deployment environment. Testing the coverage and range requires careful consideration of factors like signal
interference, obstructions, and topography that can impact signal propagation.
• Interference and Noise: LoRaWAN operates in unlicensed ISM bands, which means it shares the spectrum with other devices
and systems. Interference and noise from other wireless devices can affect the overall performance and reliability of LoRaWAN
connections.
• Scalability: IoT deployments often involve a large number of devices connected to LoRaWAN. Testing the scalability of the
network to handle a significant number of devices transmitting data concurrently is crucial.
• Battery Life: Many IoT devices operate on battery power, and optimizing energy consumption is essential. Testing the battery life
of LoRaWAN-connected devices under different usage scenarios and duty cycles is critical for assessing real-world performance.

21
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

• Latency and Throughput : LoRaWAN is optimized for low-power, long-range communication, which may result in higher latency
and lower throughput compared to other wireless technologies [98]. Evaluating and managing these trade-offs is vital for IoT
applications with specific latency and data rate requirements.
• Security: IoT devices are vulnerable to various security threats. Testing the security features of LoRaWAN, including encryption,
authentication, and integrity, is essential to ensure data privacy and protection against attacks [99].
• Network Server Performance: The LoRaWAN network server plays an important role in handling device registrations, message
routing, and security management. Evaluating the performance of the network server and its ability to handle increasing device
loads is necessary [100].
• Environmental Variability: LoRaWAN deployments can be in diverse environments, ranging from urban to rural areas, with
varying levels of radio frequency interference and environmental conditions [101]. Testing across different scenarios helps
understand the impact of these variables on performance.
• Over-the-Air (OTA) Updates: IoT devices often require firmware updates to fix bugs or add new features. Ensuring the reliability
and security of OTA updates in LoRaWAN is a critical aspect of evaluation.
• Compliance and Regulation: LoRaWAN deployments must adhere to local regulations and compliance standards related to
spectrum usage, power levels, and data privacy [102]. Testing for compliance is necessary to avoid legal issues and ensure
the network operates within the defined parameters.
• Testing Next-Generation IoT (NG-IoT): NG-IoT is about building upon the foundations of traditional IoT to create more
advanced, efficient, and capable systems. LoRa and LoRaWAN technologies are quite useful and efficient in the NG-IoT
domain, particularly for applications that value long-range, low-power consumption, and cost-effectiveness. On the other
hand, it may not be suitable for applications with high bandwidth or low latency requirements. Employing a combination
of laboratory testing, field testing, simulation, and monitoring tools is highly preferable to better gauge the performance of
LoRa and LoRaWAN depending on the use case. It is crucial to have a comprehensive testing strategy that covers all aspects
of LoRaWAN-based NG-IoT systems to ensure their reliability, performance, and security.

These issues and challenges in LoRaWAN testing and deployment are fundamental for a successful and sustainable IoT
implementation. They allow for informed decision-making, improved network performance, enhanced security, and better overall
user satisfaction. To overcome these challenges, comprehensive testing methodologies that involve real-world testing, simulation,
and validation are necessary. Additionally, collaboration among network operators, device manufacturers, and IoT application
developers can lead to more robust and reliable LoRaWAN deployments in IoT.

7. Future research directions

This section provides future research directions for testing and evaluating LoRa and LoRaWAN in IoT. As with any emerging
technology, continuous research and development are essential in advancing its capabilities and ensuring its successful deployment of
LoRa and LoRaWan in IoT-driven applications. Tests and evaluations are essential to verify the theoretical findings and assumptions
made during research and development. Future research should involve large-scale real-world testing and validation of LoRaWAN
to identify potential issues and gather insights to enhance the technology further [103]. Based on the detailed analysis of related
literature, we suggest the following potential research directions:

• Scalability and Density Testing : As the number of IoT devices and applications grows, it becomes crucial to test the scalability
and density of LoRaWAN. Researchers may explore how the system performs under increasing loads of connected devices and
how to optimize network parameters and protocols to handle large-scale deployments efficiently [104].
• Interference and Coexistence Studies: LoRaWAN operates in unlicensed spectrum bands, making it susceptible to interference
from other wireless technologies. Future research may focus on evaluating the performance of LoRaWAN in coexistence with
other wireless technologies like Wi-Fi, Bluetooth, Zigbee, etc. Researchers could develop interference mitigation techniques
and adaptive channel allocation algorithms.
• Security and Privacy Assessments: Security is of utmost importance in IoT systems. Future research could delve into the
evaluation of security measures in LoRaWAN deployments, including encryption mechanisms, key management, and secure
authentication. Additionally, privacy concerns related to device tracking and data collection may also be addressed [105].
• Energy Efficiency and Battery Life Optimization: IoT devices often operate on limited power sources, such as batteries. Research
efforts may concentrate on optimizing the energy efficiency of LoRaWAN devices and protocols to extend the battery life
of connected devices. This could involve low-power wake-up strategies, duty cycling, and optimizing data transmission
methods [106].
• Mobility and Dynamic Environments: Traditionally, LoRaWAN has been deployed in static IoT scenarios. Future research may
investigate how the system performs in dynamic environments, such as IoT devices in vehicles or wearables, and explore
strategies to handle mobility efficiently [107].
• Integration with 5G and Edge Computing : As 5G networks and edge computing gain prominence, researchers may explore the
integration of LoRaWAN with these technologies. Investigating how LoRaWAN can coexist with 5G networks and leverage
edge computing capabilities for enhanced performance could be a relevant research direction.
• Real-World Deployment Studies: Conducting large-scale real-world deployments of LoRaWAN in different environments such as
urban, rural, indoor, and outdoor, can provide valuable insights into the system’s performance under diverse conditions [108].
Such studies can highlight challenges and opportunities for improvement.

22
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

• Use of Machine Learning and Data Analytics for LoRaWAN Optimization: Applying machine learning algorithms and data analytics
to the vast amounts of data generated by LoRaWAN can help optimize various aspects, such as predictive maintenance, anomaly
detection, and resource management.
• Quality of Service (QoS) and Service Level Agreements (SLAs): Research could focus on defining QoS metrics and SLAs for
LoRaWAN deployments, enabling better service guarantees and performance monitoring in IoT applications [109].
• Standardization and Regulatory Considerations: As LoRaWAN adoption grows globally, research on standardization and reg-
ulatory issues becomes essential [102]. Future efforts may aim to address interoperability challenges, regional spectrum
regulations, and policy frameworks.
• LoRaWAN-driven Next-Gen IoT : NG-IoT is expected to involve a variety of communication technologies, and the choice of
LoRaWAN and LoRa will depend on specific use cases, requirements, and deployment scenarios. However, their characteristics
make them well-suited for many NG-IoT applications, particularly those requiring long-range, low-power communication in
diverse and challenging environments. Aside from the research directions previously listed, testing and evaluating LoRaWAN-
driven NG-IoT may consider human-centric design and usability. Ensuring that IoT systems are user-friendly and aligned
with human needs and preferences is important. Future research can investigate methods for testing the usability and user
experience of IoT devices and interfaces.

Based on the review conducted, LoRaWAN has proven to be effective in both urban and rural settings through various tests
and evaluation methodologies. It is mainly suggested that further research can explore ways to optimize LoRaWAN network
deployments in different environments, ensuring reliable connectivity and coverage across various geographical locations. Overall,
future directives in LoRa and LoRaWAN testing and evaluation in IoT deployment should focus on advanced and innovative methods
to ensure that this promising technology continues to evolve, meets the requirements of diverse IoT applications, and remains a
viable and efficient connectivity solution for the foreseeable future.

8. Conclusion

This paper reviewed the state-of-the-art LoRa and LoRaWAN test scenarios and evaluation methods in various IoT applications
from 2018 to 2023. These studies emerged from reputable journal publications searched from various databases such as IEEE
Xplore, ScienceDirect, SpringerLink, MDPI, and ACM Digital Library and comprised 63 papers in total. The review included a
classification of state-of-the-art LoRa-based and LoRaWAN-based IoT systems based on test parameters, test architecture, and
evaluation methodologies. Based on the review, seven test parameters were identified namely range, reliability, delay, scalability,
functionality, energy, geolocation, security, and capacity. The studies were also classified based on their test architecture regarding
the number of end devices, gateway, network server, and application server. In addition, the studies were organized based on their
evaluation methods namely testbed, simulation, and mathematical modeling. Furthermore, the survey was extended to review the
test and evaluation methodologies utilized to assess the performance of state-of-the-art mechanisms that enhanced the MAC layer
features of LoRa and LoRaWAN, namely scheduling, synchronization, collision avoidance, variable payload, and fairness.
Based on the findings in this review, the majority of the studies have tested the maximum range of LoRa and LoRaWAN by
utilizing an actual testbed implementation in various environment scenarios. On the other hand, the most commonly used network
performance metrics measured among the studies are RSSI and SNR. Finally, a summary and consolidated view of test and evaluation
methodologies are the results, and the conclusion of the analysis of the literature review conducted in this work is presented.
The issues and challenges behind the techniques were also identified. It was suggested that among the identified challenges, the
focus should be put on designing comprehensive testing methodologies that involve real-world testing, simulation, and validation as
well as collaboration among network operators, device manufacturers, and IoT application developers that can lead to more robust
and reliable LoRaWAN deployments in IoT. Additionally, the potential future research directions and opportunities in testing and
evaluation methodologies in LoRa-based and LoRaWAN-based networks in IoT were also presented. Among the possible directions,
attention should be given to exploring ways to optimize LoRaWAN network deployments in different environments, ensuring reliable
connectivity and coverage across various geographical locations. Lastly, it should focus on advanced and innovative methods to
ensure that LPWAN technology such as LoRa and LoRaWAN continues to evolve, meets the requirements of diverse IoT applications,
and remains a viable and efficient connectivity solution for IoT devices and systems.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing
interests: Melchizedek Alipio reports financial support was provided by Republic of the Philippines Department of Science and
Technology.

Data availability

No data was used for the research described in the article.

23
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

Declaration of Generative AI and AI-assisted technologies in the writing process

During the preparation of this work, the authors used the AI-based language feedback tools Grammarly11 and QuillBot12 to
improve the grammar and writing style of the article. After using these tools, the authors reviewed and edited the content as
needed. They take full responsibility for the content of the publication.

Acknowledgments

The author would like to acknowledge the support of the Department of Science and Technology - Science Education Institute
(DOST-SEI) of the Republic of the Philippines through the Engineering Research and Development for Technology (ERDT) and the
Office of the Provost of the De La Salle University Manila.

References

[1] J.L.V. de Barros, M.E.P. Monteiro, G. de Santi Peron, G.L. Moritz, O.K. Rayel, R.D. Souza, LoRaWAN vs. 6TiSCH: Which one scales bet-
ter? Comput. Commun. 184 (2022) 1–11, http://dx.doi.org/10.1016/j.comcom.2021.12.004, URL https://www.sciencedirect.com/science/article/pii/
S0140366421004709.
[2] M. El-Aasser, P. Edward, M. Mandour, M. Ashour, T. Elshabrawy, A comprehensive hybrid bit-level and packet-level LoRa-LPWAN simula-
tion model, Internet Things 14 (2021) 100386, http://dx.doi.org/10.1016/j.iot.2021.100386, URL https://www.sciencedirect.com/science/article/pii/
S2542660521000305.
[3] C. Milarokostas, D. Tsolkas, N. Passas, L. Merakos, A comprehensive study on LPWANs with a focus on the potential of LoRa/LoRaWAN systems, IEEE
Commun. Surv. Tutor. 25 (1) (2023) 825–867, http://dx.doi.org/10.1109/COMST.2022.3229846.
[4] W. Ayoub, A.E. Samhat, F. Nouvel, M. Mroue, J.-C. Prévotet, Internet of mobile things: Overview of LoRaWAN, DASH7, and NB-IoT in LPWANs standards
and supported mobility, IEEE Commun. Surv. Tutor. 21 (2) (2019) 1561–1581, http://dx.doi.org/10.1109/COMST.2018.2877382.
[5] S. Bagwari, A. Gehlot, R. Singh, N. Priyadarshi, B. Khan, Low-cost sensor-based and LoRaWAN opportunities for landslide monitoring systems on IoT
platform: A review, IEEE Access 10 (2022) 7107–7127, http://dx.doi.org/10.1109/ACCESS.2021.3137841.
[6] D. Zorbas, ActLoRa: Supporting actuators in LoRaWAN, in: ICC 2022 - IEEE International Conference on Communications, 2022, pp. 4426–4431,
http://dx.doi.org/10.1109/ICC45855.2022.9839166.
[7] M. Rizzi, P. Ferrari, A. Flammini, E. Sisinni, Evaluation of the IoT LoRaWAN solution for distributed measurement applications, IEEE Trans. Instrum.
Meas. 66 (12) (2017) 3340–3349, http://dx.doi.org/10.1109/TIM.2017.2746378.
[8] T.-H. To, A. Duda, Simulation of LoRa in NS-3: Improving LoRa performance with CSMA, in: 2018 IEEE International Conference on Communications,
ICC, 2018, pp. 1–7, http://dx.doi.org/10.1109/ICC.2018.8422800.
[9] J.R. Cotrim, J.H. Kleinschmidt, An analytical model for multihop LoRaWAN networks, Internet Things 22 (2023) 100807, http://dx.doi.org/10.1016/j.
iot.2023.100807, URL https://www.sciencedirect.com/science/article/pii/S2542660523001300.
[10] A. Mahmood, E. Sisinni, L. Guntupalli, R. Rondón, S.A. Hassan, M. Gidlund, Scalability analysis of a LoRa network under imperfect orthogonality, IEEE
Trans. Ind. Inform. 15 (3) (2019) 1425–1436, http://dx.doi.org/10.1109/TII.2018.2864681.
[11] G. Pasolini, On the LoRa chirp spread spectrum modulation: Signal properties and their impact on transmitter and receiver architectures, IEEE Trans.
Wireless Commun. 21 (1) (2022) 357–369, http://dx.doi.org/10.1109/TWC.2021.3095667.
[12] P.J. Marcelis, N. Kouvelas, V.S. Rao, R.V. Prasad, DaRe: Data recovery through application layer coding for LoRaWAN, IEEE Trans. Mob. Comput. 21
(3) (2022) 895–910, http://dx.doi.org/10.1109/TMC.2020.3016654.
[13] S. Lee, J. Lee, H.-S. Park, J.K. Choi, A novel fair and scalable relay control scheme for internet of things in LoRa-based low-power wide-area networks,
IEEE Internet Things J. 8 (7) (2021) 5985–6001, http://dx.doi.org/10.1109/JIOT.2020.3034185.
[14] J. Park, K. Park, H. Bae, C.-K. Kim, EARN: Enhanced ADR with coding rate adaptation in LoRaWAN, IEEE Internet Things J. 7 (12) (2020) 11873–11883,
http://dx.doi.org/10.1109/JIOT.2020.3005881.
[15] M.I. Zakaria, W.A. Jabbar, N. Sulaiman, Development of a smart sensing unit for LoRaWAN-based IoT flood monitoring and warning system in catchment
areas, Internet Things Cyber-Phys. Syst. 3 (2023) 249–261, http://dx.doi.org/10.1016/j.iotcps.2023.04.005, URL https://www.sciencedirect.com/science/
article/pii/S2667345223000263.
[16] G. Premsankar, B. Ghaddar, M. Slabicki, M.D. Francesco, Optimal configuration of LoRa networks in smart cities, IEEE Trans. Ind. Inform. 16 (12) (2020)
7243–7254, http://dx.doi.org/10.1109/TII.2020.2967123.
[17] W.A. Jabbar, T. Subramaniam, A.E. Ong, M.I. Shu’Ib, W. Wu, M.A. de Oliveira, LoRaWAN-based IoT system implementation for long-range outdoor
air quality monitoring, Internet of Things 19 (2022) 100540, http://dx.doi.org/10.1016/j.iot.2022.100540, URL https://www.sciencedirect.com/science/
article/pii/S2542660522000427.
[18] H.-C. Lee, K.-H. Ke, Monitoring of large-area IoT sensors using a LoRa wireless mesh network system: Design and evaluation, IEEE Trans. Instrum. Meas.
67 (9) (2018) 2177–2187, http://dx.doi.org/10.1109/TIM.2018.2814082.
[19] B. Paul, A novel mathematical model to evaluate the impact of packet retransmissions in LoRaWAN, IEEE Sensors Lett. 4 (5) (2020) 1–4, http:
//dx.doi.org/10.1109/LSENS.2020.2986794.
[20] J. Grübel, T. Thrash, L. Aguilar, M. Gath-Morad, D. Hélal, R.W. Sumner, C. Hölscher, V.R. Schinazi, Dense indoor sensor networks: Towards
passively sensing human presence with LoRaWAN, Pervasive Mob. Comput. 84 (2022) 101640, http://dx.doi.org/10.1016/j.pmcj.2022.101640, URL
https://www.sciencedirect.com/science/article/pii/S1574119222000700.
[21] L. Prade, J. Moraes, E. de Albuquerque, D. Rosário, C.B. Both, Multi-radio and multi-hop LoRa communication architecture for large scale IoT deployment,
Comput. Electr. Eng. 102 (2022) 108242, http://dx.doi.org/10.1016/j.compeleceng.2022.108242, URL https://www.sciencedirect.com/science/article/pii/
S0045790622004773.
[22] J.M. Marais, R. Malekian, A.M. Abu-Mahfouz, Lora and LoRaWAN testbeds: A review, in: 2017 IEEE AFRICON, 2017, pp. 1496–1501, http://dx.doi.org/
10.1109/AFRCON.2017.8095703.
[23] M. Bembe, A. Abu-Mahfouz, M. Masonta, T. Ngqondi, A survey on low-power wide area networks for IoT applications, Telecommun. Syst. 71 (2) (2019)
249–274.

11 https://app.grammarly.com/
12 https://quillbot.com/

24
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

[24] J.P. Queralta, T. Gia, Z. Zou, H. Tenhunen, T. Westerlund, Comparative study of LPWAN technologies on unlicensed bands for M2M communication
in the IoT: beyond LoRa and LoRaWAN, Procedia Comput. Sci. 155 (2019) 343–350, http://dx.doi.org/10.1016/j.procs.2019.08.049, URL https://www.
sciencedirect.com/science/article/pii/S1877050919309639. The 16th International Conference on Mobile Systems and Pervasive Computing (MobiSPC
2019), The 14th International Conference on Future Networks and Communications (FNC-2019), The 9th International Conference on Sustainable Energy
Information Technology.
[25] F. Gu, J. Niu, L. Jiang, X. Liu, M. Atiquzzaman, Survey of the low power wide area network technologies, J. Netw. Comput. Appl. 149 (2020) 102459,
http://dx.doi.org/10.1016/j.jnca.2019.102459, URL https://www.sciencedirect.com/science/article/pii/S1084804519303194.
[26] H. Noura, T. Hatoum, O. Salman, J.-P. Yaacoub, A. Chehab, LoRaWAN security survey: Issues, threats and possible mitigation techniques, Internet Things
12 (2020) 100303, http://dx.doi.org/10.1016/j.iot.2020.100303, URL https://www.sciencedirect.com/science/article/pii/S2542660520301359.
[27] F.S.D. Silva, E.P. Neto, H. Oliveira, D. Rosário, E. Cerqueira, C. Both, S. Zeadally, A.V. Neto, A survey on long-range wide-area network technology
optimizations, IEEE Access 9 (2021) 106079–106106, http://dx.doi.org/10.1109/ACCESS.2021.3079095.
[28] S.-Y. Wang, J.-E. Chang, H. Fan, Y.-H. Sun, Comparing the performance of NB-IoT, LTE Cat-M1, Sigfox, and LoRa for IoT end devices moving at high
speeds in the air, J. Signal Process. Syst. 94 (1) (2022) 81–99.
[29] M.A.M. Almuhaya, W.A. Jabbar, N. Sulaiman, S. Abdulmalek, A survey on LoRaWAN technology: Recent trends, opportunities, simulation tools and future
directions, Electronics 11 (1) (2022) http://dx.doi.org/10.3390/electronics11010164, URL https://www.mdpi.com/2079-9292/11/1/164.
[30] A. Pagano, D. Croce, I. Tinnirello, G. Vitale, A survey on LoRa for smart agriculture: Current trends and future perspectives, IEEE Internet Things J. 10
(4) (2023) 3664–3679, http://dx.doi.org/10.1109/JIOT.2022.3230505.
[31] L.E. Marquez, M. Calle, Understanding LoRa-based localization: Foundations and challenges, IEEE Internet Things J. 10 (13) (2023) 11185–11198,
http://dx.doi.org/10.1109/JIOT.2023.3248860.
[32] G. Zhao, K. Lin, T. Hao, A feasibility study of LoRaWAN-based wireless underground sensor networks for underground monitoring, Comput. Netw. 232
(2023) 109851, http://dx.doi.org/10.1016/j.comnet.2023.109851, URL https://www.sciencedirect.com/science/article/pii/S1389128623002967.
[33] B. Miles, E.-B. Bourennane, S. Boucherkha, S. Chikhi, A study of LoRaWAN protocol performance for IoT applications in smart agriculture,
Comput. Commun. 164 (2020) 148–157, http://dx.doi.org/10.1016/j.comcom.2020.10.009, URL https://www.sciencedirect.com/science/article/pii/
S0140366420319575.
[34] D.F. Carvalho, P. Ferrari, A. Flammini, E. Sisinni, A test bench for evaluating communication delays in LoRaWAN applications, in: 2018 Workshop on
Metrology for Industry 4.0 and IoT, 2018, pp. 248–253, http://dx.doi.org/10.1109/METROI4.2018.8428323.
[35] A.-M. Drăgulinescu, I. Marcu, C. Zamfirescu, An end-to-end lorawan-based iot platform with built-in network coverage testing capability, in: 2022 25th
International Symposium on Wireless Personal Multimedia Communications, WPMC, 2022, pp. 474–479, http://dx.doi.org/10.1109/WPMC55625.2022.
10014713.
[36] J. Gauld, P.W. Atkinson, J.P. Silva, A. Senn, A.M.A. Franco, Characterisation of a new lightweight LoRaWAN GPS bio-logger and deployment on griffon
vultures gyps fulvus, Animal Biotelem. 11 (1) (2023) 17.
[37] M. Pérez, F.E. Sierra-Sánchez, F. Chaparro, D.M. Chaves, C.-I. Paez-Rueda, G.P. Galindo, A. Fajardo, Coverage and energy-efficiency experimental test
performance for a comparative evaluation of unlicensed lpwan: Lorawan and sigfox, IEEE Access 10 (2022) 97183–97196, http://dx.doi.org/10.1109/
ACCESS.2022.3206030.
[38] E.T. de Camargo, F.A. Spanhol, Á.R. Castro e Souza, Deployment of a LoRaWAN network and evaluation of tracking devices in the context of smart
cities, J. Internet Serv. Appl. 12 (1) (2021) 8.
[39] J.M. Marais, R. Malekian, A.M. Abu-Mahfouz, Evaluating the LoRaWAN protocol using a permanent outdoor testbed, IEEE Sens. J. 19 (12) (2019)
4726–4733, http://dx.doi.org/10.1109/JSEN.2019.2900735.
[40] V. Lopez, H. Kaushal, Z. Prodanoff, A. Jiang, Evaluation of communication delays in LoRaWAN networks for indoor emergency scenarios, J. Commun.
Technol. Electron. 66 (2) (2021) S149–S158.
[41] H.E. Elbsir, M. Kassab, S. Bhiri, M.H. Bedoui, Evaluation of LoRaWAN class B performances and its optimization for better support of actuators,
Comput. Commun. 198 (2023) 128–139, http://dx.doi.org/10.1016/j.comcom.2022.11.016, URL https://www.sciencedirect.com/science/article/pii/
S014036642200439X.
[42] M. Saban, O. Aghzout, L.D. Medus, A. Rosado, Experimental analysis of IoT networks based on LoRa/LoRaWAN under indoor and outdoor environments:
Performance and limitations, IFAC-PapersOnLine 54 (4) (2021) 159–164, http://dx.doi.org/10.1016/j.ifacol.2021.10.027, URL https://www.sciencedirect.
com/science/article/pii/S2405896321014294. 4th IFAC Conference on Embedded Systems, Computational Intelligence and Telematics in Control CESCIT
2021.
[43] R. Dai, O. Diraneyya, S. Brell-Ç.okcan, Improving data communication on construction sites via LoRaWAN, Constr. Robot. 5 (2) (2021) 87–100.
[44] R. Apriantoro, A. Suharjono, K. Kurnianingsih, I.K.A. Enriko, Investigation of coverage and signal quality of LoRaWAN network in urban area, in:
2020 International Conference on Computer Engineering, Network, and Intelligent Multimedia, CENIM, 2020, pp. 326–331, http://dx.doi.org/10.1109/
CENIM51130.2020.9297982.
[45] A.M.C. Drăgulinescu, A.F. Manea, O. Fratu, A. Drăgulinescu, LoRa-Based medical IoT system architecture and testbed, Wirel. Pers. Commun. 126 (1)
(2022) 25–47.
[46] A.V.T. Bardram, M. Delbo Larsen, K.M. Malarski, M.N. Petersen, S. Ruepp, LoRaWan capacity simulation and field test in a harbour environment, in:
2018 Third International Conference on Fog and Mobile Edge Computing, FMEC, 2018, pp. 193–198, http://dx.doi.org/10.1109/FMEC.2018.8364064.
[47] C. Paternina, R. Arnedo, J. Dominguez-Jimenez, J. Campillo, LoRAWAN network coverage testing design using open-source low-cost hardware, in: 2020
IEEE ANDESCON, 2020, pp. 1–6, http://dx.doi.org/10.1109/ANDESCON50619.2020.9272128.
[48] N. Prasad, P. Lynggaard, LoRaWan sensitivity analysis and prevention strategies against wireless DoS attacks, Wirel. Pers. Commun. 126 (4) (2022)
3663–3675.
[49] J. Michaelis, A. Morelli, L. Hernandez, D. James, J. Freeman, N. Suri, LoRaWAN testing for military communications in urban environments, in: 2021
IEEE 7th World Forum on Internet of Things, WF-IoT, 2021, pp. 885–890, http://dx.doi.org/10.1109/WF-IoT51360.2021.9595596.
[50] A. Lombardo, S. Parrino, G. Peruzzi, A. Pozzebon, LoRaWAN versus NB-IoT: Transmission performance analysis within critical environments, IEEE Internet
Things J. 9 (2) (2022) 1068–1081, http://dx.doi.org/10.1109/JIOT.2021.3079567.
[51] L. Feltrin, C. Buratti, E. Vinciarelli, R. De Bonis, R. Verdone, LoRaWAN: Evaluation of link- and system-level performance, IEEE Internet Things J. 5 (3)
(2018) 2249–2258, http://dx.doi.org/10.1109/JIOT.2018.2828867.
[52] L.E. Marquez, A. Osorio, M. Calle, J.C. Velez, A. Serrano, J.E. Candelo-Becerra, On the use of LoRaWAN in smart cities: A study with blocking interference,
IEEE Internet Things J. 7 (4) (2020) 2806–2815, http://dx.doi.org/10.1109/JIOT.2019.2962976.
[53] C. Bouras, A. Gkamas, V. Kokkinos, N. Papachristos, Performance evaluation of monitoring IoT systems using LoRaWan, Telecommun. Syst. 79 (2) (2022)
295–308.
[54] W. Ingabire, H. Larijani, R.M. Gibson, Performance evaluation of propagation models for LoRaWAN in an urban environment, in: 2020 International
Conference on Electrical, Communication, and Computer Engineering, ICECCE, 2020, pp. 1–6, http://dx.doi.org/10.1109/ICECCE49384.2020.9179234.
[55] I. Cappelli, A. Fort, M. Mugnaini, S. Parrino, A. Pozzebon, Underwater to above water LoRaWAN networking: Theoretical analysis and field
tests, Measurement 196 (2022) 111140, http://dx.doi.org/10.1016/j.measurement.2022.111140, URL https://www.sciencedirect.com/science/article/pii/
S0263224122004006.

25
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

[56] E. Saavedra, L. Mascaraque, G. Calderon, G. del Campo, A. Santamaria, A universal testbed for IoT wireless technologies: Abstracting latency, error rate and
stability from the IoT protocol and hardware platform, Sensors 22 (11) (2022) http://dx.doi.org/10.3390/s22114159, URL https://www.mdpi.com/1424-
8220/22/11/4159.
[57] S. Ould, N.S. Bennett, Energy performance analysis and modelling of LoRa prototyping boards, Sensors 21 (23) (2021) http://dx.doi.org/10.3390/
s21237992, URL https://www.mdpi.com/1424-8220/21/23/7992.
[58] A.M. Alghamdi, E.F. Khairullah, M.M. Al mojamed, LoRaWAN performance analysis for a water monitoring and leakage detection system in a housing
complex, Sensors 22 (19) (2022) http://dx.doi.org/10.3390/s22197188, URL https://www.mdpi.com/1424-8220/22/19/7188.
[59] E. Harinda, A.J. Wixted, A.-U.-H. Qureshi, H. Larijani, R.M. Gibson, Performance of a live multi-gateway LoRaWAN and interference measurement across
indoor and outdoor localities, Computers 11 (2) (2022) http://dx.doi.org/10.3390/computers11020025, URL https://www.mdpi.com/2073-431X/11/2/25.
[60] H. Pirayesh, S. Zhang, P.K. Sangdeh, H. Zeng, MaLoRaGW: Multi-user MIMO transmission for lora, in: Proceedings of the 20th ACM Conference on
Embedded Networked Sensor Systems, SenSys ’22, Association for Computing Machinery, New York, NY, USA, 2023, pp. 179–192, http://dx.doi.org/10.
1145/3560905.3568533.
[61] B. Xie, D. Ganesan, J. Xiong, Embracing LoRa sensing with device mobility, in: Proceedings of the 20th ACM Conference on Embedded Networked Sensor
Systems, SenSys ’22, Association for Computing Machinery, New York, NY, USA, 2023, pp. 349–361, http://dx.doi.org/10.1145/3560905.3568524.
[62] J.R. Santana, P. Sotres, J. Pérez, L. Sánchez, J. Lanza, L. Muñoz, LoRaWAN-based smart parking service: Deployment and performance evaluation, in:
Proceedings of the 19th ACM International Symposium on Performance Evaluation of Wireless Ad Hoc, Sensor, & Ubiquitous Networks, in: PE-WASUN
’22, Association for Computing Machinery, New York, NY, USA, 2022, pp. 107–114, http://dx.doi.org/10.1145/3551663.3558685.
[63] L. Chen, J. Xiong, X. Chen, S.I. Lee, K. Chen, D. Han, D. Fang, Z. Tang, Z. Wang, WideSee: Towards wide-area contactless wireless sensing, in: Proceedings
of the 17th Conference on Embedded Networked Sensor Systems, SenSys ’19, Association for Computing Machinery, New York, NY, USA, 2019, pp.
258–270, http://dx.doi.org/10.1145/3356250.3360031.
[64] F. Busacca, L. Galluccio, S. Mertens, D. Orto, S. Palazzo, S. Quattropani, An experimental testbed of an internet of underwater things, in: Proceedings of
the 14th International Workshop on Wireless Network Testbeds, Experimental Evaluation & Characterization, WiNTECH ’20, Association for Computing
Machinery, New York, NY, USA, 2020, pp. 95–102, http://dx.doi.org/10.1145/3411276.3412186.
[65] K. Mikhaylov, R. Fujdiak, A. Pouttu, V. Miroslav, L. Malina, P. Mlynek, Energy attack in LoRaWAN: Experimental validation, in: Proceedings of the
14th International Conference on Availability, Reliability and Security, ARES ’19, Association for Computing Machinery, New York, NY, USA, 2019,
http://dx.doi.org/10.1145/3339252.3340525.
[66] B. Xie, M. Cui, D. Ganesan, X. Chen, J. Xiong, Boosting the long range sensing potential of LoRa, in: Proceedings of the 21st Annual International
Conference on Mobile Systems, Applications and Services, MobiSys ’23, Association for Computing Machinery, New York, NY, USA, 2023, pp. 177–190,
http://dx.doi.org/10.1145/3581791.3596847.
[67] R. Subbaraman, Y. Guntupalli, S. Jain, R. Kumar, K. Chintalapudi, D. Bharadia, BSMA: Scalable LoRa networks using full duplex gateways, in: Proceedings
of the 28th Annual International Conference on Mobile Computing and Networking, MobiCom ’22, Association for Computing Machinery, New York, NY,
USA, 2022, pp. 676–689, http://dx.doi.org/10.1145/3495243.3560544.
[68] H. Jiang, J. Zhang, X. Guo, Y. He, Sense me on the ride: Accurate mobile sensing over a LoRa backscatter channel, in: Proceedings of the 19th
ACM Conference on Embedded Networked Sensor Systems, SenSys ’21, Association for Computing Machinery, New York, NY, USA, 2021, pp. 125–137,
http://dx.doi.org/10.1145/3485730.3485933.
[69] X. Wang, L. Kong, Z. Wu, L. Cheng, C. Xu, G. Chen, Slora: Towards secure LoRa communications with fine-grained physical layer features, in: Proceedings of
the 18th Conference on Embedded Networked Sensor Systems, SenSys ’20, Association for Computing Machinery, New York, NY, USA, 2020, pp. 258–270,
http://dx.doi.org/10.1145/3384419.3430770.
[70] C. Li, H. Guo, S. Tong, X. Zeng, Z. Cao, M. Zhang, Q. Yan, L. Xiao, J. Wang, Y. Liu, NELoRa: Towards ultra-low SNR LoRa communication with neural-
enhanced demodulation, in: Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, SenSys ’21, Association for Computing
Machinery, New York, NY, USA, 2021, pp. 56–68, http://dx.doi.org/10.1145/3485730.3485928.
[71] M.O. Shahid, M. Philipose, K. Chintalapudi, S. Banerjee, B. Krishnaswamy, Concurrent interference cancellation: Decoding multi-packet collisions in LoRa,
in: Proceedings of the 2021 ACM SIGCOMM 2021 Conference, SIGCOMM ’21, Association for Computing Machinery, New York, NY, USA, 2021, pp.
503–515, http://dx.doi.org/10.1145/3452296.3472931.
[72] S. Tong, Z. Shen, Y. Liu, J. Wang, Combating link dynamics for reliable lora connection in urban settings, in: Proceedings of the 27th Annual International
Conference on Mobile Computing and Networking, MobiCom ’21, Association for Computing Machinery, New York, NY, USA, 2021, pp. 642–655,
http://dx.doi.org/10.1145/3447993.3483250.
[73] J. Liu, J. Gao, S. Jha, W. Hu, Seirios: Leveraging multiple channels for LoRaWAN indoor and outdoor localization, in: Proceedings of the 27th Annual
International Conference on Mobile Computing and Networking, MobiCom ’21, Association for Computing Machinery, New York, NY, USA, 2021, pp.
656–669, http://dx.doi.org/10.1145/3447993.3483256.
[74] X. Xia, Q. Chen, N. Hou, Y. Zheng, Hylink: Towards high throughput LPWANs with LoRa compatible communication, in: Proceedings of the 20th
ACM Conference on Embedded Networked Sensor Systems, SenSys ’22, Association for Computing Machinery, New York, NY, USA, 2023, pp. 578–591,
http://dx.doi.org/10.1145/3560905.3568516.
[75] L. Leonardi, L. Lo Bello, G. Patti, MRT-LoRa: A multi-hop real-time communication protocol for industrial IoT applications over LoRa networks, Comput.
Commun. 199 (C) (2023) 72–86, http://dx.doi.org/10.1016/j.comcom.2022.12.013.
[76] H. Taleb, G. Andrieux, A. Nasser, N. Charara, Energy efficient selection of spreading factor in LoRaWAN-based WBAN medical systems, Internet of Things
24 (2023) 100896, http://dx.doi.org/10.1016/j.iot.2023.100896, URL https://www.sciencedirect.com/science/article/pii/S2542660523002196.
[77] L. Dujić Rodić, T. Perković, M. Škiljo, P. Šolić, Privacy leakage of LoRaWAN smart parking occupancy sensors, Future Gener. Comput. Syst. 138 (2023)
142–159, http://dx.doi.org/10.1016/j.future.2022.08.007, URL https://www.sciencedirect.com/science/article/pii/S0167739X22002680.
[78] J. Fox, A. Donnellan, L. Doumen, The deployment of an IoT network infrastructure, as a localised regional service, in: 2019 IEEE 5th World Forum on
Internet of Things, WF-IoT, 2019, pp. 319–324, http://dx.doi.org/10.1109/WF-IoT.2019.8767188.
[79] R. Ramesh, M. Arunachalam, H.K. Atluri, C. Kumar S, S. Anand, P. Arumugam, B. Amrutur, 15 - LoRaWAN for smart cities: Experimental study in a
campus deployment, in: B.S. Chaudhari, M. Zennaro (Eds.), LPWAN Technologies for IoT and M2M Applications, Academic Press, 2020, pp. 327–345,
http://dx.doi.org/10.1016/B978-0-12-818880-4.00016-8, URL https://www.sciencedirect.com/science/article/pii/B9780128188804000168.
[80] D. Zorbas, K. Abdelfadeel, P. Kotzanikolaou, D. Pesch, TS-LoRa: Time-slotted LoRaWAN for the industrial internet of things, Comput. Commun. 153
(2020) 1–10, http://dx.doi.org/10.1016/j.comcom.2020.01.056, URL https://www.sciencedirect.com/science/article/pii/S0140366419314677.
[81] A. Triantafyllou, P. Sarigiannidis, T. Lagkas, I.D. Moscholios, A. Sarigiannidis, Leveraging fairness in LoRaWAN: A novel scheduling scheme for collision
avoidance, Comput. Netw. 186 (2021) 107735, http://dx.doi.org/10.1016/j.comnet.2020.107735, URL https://www.sciencedirect.com/science/article/pii/
S1389128620313232.
[82] M.S.A. Muthanna, A. Muthanna, A. Rafiq, M. Hammoudeh, R. Alkanhel, S. Lynch, A.A. Abd El-Latif, Deep reinforcement learning based transmission
policy enforcement and multi-hop routing in QoS aware LoRa IoT networks, Comput. Commun. 183 (2022) 33–50, http://dx.doi.org/10.1016/j.comcom.
2021.11.010, URL https://www.sciencedirect.com/science/article/pii/S0140366421004394.
[83] C. Garrido-Hidalgo, L. Roda-Sanchez, F.J. Ramírez, A. Fernández-Caballero, T. Olivares, Efficient online resource allocation in large-scale LoRaWAN
networks: A multi-agent approach, Comput. Netw. 221 (2023) 109525, http://dx.doi.org/10.1016/j.comnet.2022.109525, URL https://www.sciencedirect.
com/science/article/pii/S138912862200559X.

26
M. Alipio and M. Bures Internet of Things 25 (2024) 101053

[84] P. Ferrari, P. Bellagente, A. Depari, D.F. Carvalho, A. Flammini, M. Pasetti, S. Rinaldi, E. Sisinni, Improving LoRa-REP scalability in emergency scenarios
by means of slotted delay, in: 2022 IEEE International Symposium on Measurements & Networking, M&N, 2022, pp. 1–6, http://dx.doi.org/10.1109/
MN55117.2022.9887767.
[85] C. Garrido-Hidalgo, J. Haxhibeqiri, B. Moons, J. Hoebeke, T. Olivares, F.J. Ramirez, A. Fernández-Caballero, LoRaWAN scheduling: From concept to
implementation, IEEE Internet Things J. 8 (16) (2021) 12919–12933, http://dx.doi.org/10.1109/JIOT.2021.3064430.
[86] W. Gao, W. Du, Z. Zhao, G. Min, M. Singhal, Towards energy-fairness in LoRa networks, in: 2019 IEEE 39th International Conference on Distributed
Computing Systems, ICDCS, 2019, pp. 788–798, http://dx.doi.org/10.1109/ICDCS.2019.00083.
[87] A.-U.-H. Ahmar, E. Aras, T.D. Nguyen, S. Michiels, W. Joosen, D. Hughes, Design of a robust MAC protocol for LoRa, ACM Trans. Internet Things 4 (1)
(2023) http://dx.doi.org/10.1145/3557048.
[88] Y. Wang, F. Zhang, X. Zheng, L. Liu, H. Ma, Decoding LoRa collisions via parallel alignment, ACM Trans. Sen. Netw. 19 (3) (2023) http://dx.doi.org/
10.1145/3571586.
[89] D.L. Mai, M.K. Kim, Multi-hop LoRa network protocol with minimized latency, Energies 13 (6) (2020) http://dx.doi.org/10.3390/en13061368, URL
https://www.mdpi.com/1996-1073/13/6/1368.
[90] E. Sallum, N. Pereira, M. Alves, M. Santos, Improving quality-of-service in LoRa low-power wide-area networks through optimized radio resource
management, J. Sensor Actuator Netw. 9 (1) (2020) http://dx.doi.org/10.3390/jsan9010010, URL https://www.mdpi.com/2224-2708/9/1/10.
[91] H. Rajab, T. Cinkler, T. Bouguera, IoT scheduling for higher throughput and lower transmission power, Wirel. Netw. 27 (3) (2021) 1701–1714.
[92] M. Centenaro, L. Vangelista, Time-Power multiplexing for LoRa-Based IoT networks: An effective way to boost LoRaWAN network capacity, Int. J. Wireless
Inf. Netw. 26 (4) (2019) 308–318.
[93] N. Benkahla, H. Tounsi, Y.-Q. Song, M. Frikha, Enhanced dynamic duty cycle in LoRaWAN network, in: N. Montavont, G.Z. Papadopoulos (Eds.), Ad-Hoc,
Mobile, and Wireless Networks, Springer International Publishing, Cham, 2018, pp. 147–162.
[94] N. Hou, X. Xia, Y. Zheng, Jamming of LoRa PHY and countermeasure, in: IEEE INFOCOM 2021 - IEEE Conference on Computer Communications, 2021,
pp. 1–10, http://dx.doi.org/10.1109/INFOCOM42981.2021.9488774.
[95] S. Tong, J. Wang, Y. Liu, Combating packet collisions using non-stationary signal scaling in LPWANs, in: Proceedings of the 18th International
Conference on Mobile Systems, Applications, and Services, MobiSys ’20, Association for Computing Machinery, New York, NY, USA, 2020, pp. 234–246,
http://dx.doi.org/10.1145/3386901.3388913.
[96] W. Wu, H. Wang, Z. Cheng, ReLoRaWAN: Reliable data delivery in LoRaWAN networks with multiple gateways, Ad Hoc Netw. 147 (2023) 103203,
http://dx.doi.org/10.1016/j.adhoc.2023.103203, URL https://www.sciencedirect.com/science/article/pii/S1570870523001233.
[97] G. Zhao, K. Lin, D. Chapman, N. Metje, T. Hao, Optimizing energy efficiency of LoRaWAN-based wireless underground sensor networks: A multi-agent
reinforcement learning approach, Internet Things 22 (2023) 100776, http://dx.doi.org/10.1016/j.iot.2023.100776, URL https://www.sciencedirect.com/
science/article/pii/S2542660523000999.
[98] E. Sisinni, D. Fernandes Carvalho, A. Depari, P. Bellagente, A. Flammini, M. Pasetti, S. Rinaldi, P. Ferrari, Assessing a methodology for evaluating
the latency of IPv6 with SCHC compression in LoRaWAN deployments, Sensors 23 (5) (2023) http://dx.doi.org/10.3390/s23052407, URL https:
//www.mdpi.com/1424-8220/23/5/2407.
[99] S. Milani, I. Chatzigiannakis, Design, analysis, and experimental evaluation of a new secure rejoin mechanism for LoRaWAN using elliptic-curve
cryptography, J. Sensor Actuator Netw. 10 (2) (2021) http://dx.doi.org/10.3390/jsan10020036, URL https://www.mdpi.com/2224-2708/10/2/36.
[100] C. Garrido-Hidalgo, L. Roda-Sanchez, F.J. Ramírez, A. Fernández-Caballero, T. Olivares, Efficient online resource allocation in large-scale LoRaWAN
networks: A multi-agent approach, Comput. Netw. 221 (2023) 109525, http://dx.doi.org/10.1016/j.comnet.2022.109525, URL https://www.sciencedirect.
com/science/article/pii/S138912862200559X.
[101] T. Fedullo, A. Morato, G. Peserico, L. Trevisan, F. Tramarin, S. Vitturi, L. Rovati, An IoT measurement system based on LoRaWAN for additive
manufacturing, Sensors 22 (15) (2022) http://dx.doi.org/10.3390/s22155466, URL https://www.mdpi.com/1424-8220/22/15/5466.
[102] R. Sanchez-Iborra, J. Sanchez-Gomez, J. Ballesta-Viñas, M.-D. Cano, A.F. Skarmeta, Performance evaluation of LoRa considering scenario conditions,
Sensors 18 (3) (2018) http://dx.doi.org/10.3390/s18030772, URL https://www.mdpi.com/1424-8220/18/3/772.
[103] M. Jouhari, N. Saeed, M.-S. Alouini, E.M. Amhoud, A survey on scalable LoRaWAN for massive IoT: Recent advances, potentials, and challenges, IEEE
Commun. Surv. Tutor. (2023) 1, http://dx.doi.org/10.1109/COMST.2023.3274934.
[104] C. Shao, O. Muta, Q. Du, K.R. Dandekar, X. Wang, Multiple access in large-scale LoRaWAN: Challenges, solutions, and future perspectives, IEEE Consum.
Electron. Mag. (2022) 1–9, http://dx.doi.org/10.1109/MCE.2022.3182518.
[105] K.-L. Tsai, F.-Y. Leu, L.-L. Hung, C.-Y. Ko, Secure session key generation method for LoRaWAN servers, IEEE Access 8 (2020) 54631–54640, http:
//dx.doi.org/10.1109/ACCESS.2020.2978100.
[106] M. Mehic, M. Duliman, N. Selimovic, M. Voznak, LoRaWAN end nodes: Security and energy efficiency analysis, Alex. Eng. J. 61 (11) (2022) 8997–9009,
http://dx.doi.org/10.1016/j.aej.2022.02.035, URL https://www.sciencedirect.com/science/article/pii/S1110016822001235.
[107] H. Uyanık, T. Ovatman, An investigation of the transmission success in lorawan enabled IoT-HAPS communication, Internet Things 20 (2022) 100611,
http://dx.doi.org/10.1016/j.iot.2022.100611, URL https://www.sciencedirect.com/science/article/pii/S2542660522000932.
[108] P.J. Basford, F.M.J. Bulot, M. Apetroaie-Cristea, S.J. Cox, S.J. Ossont, LoRaWAN for smart city IoT deployments: A long term evaluation, Sensors 20 (3)
(2020) http://dx.doi.org/10.3390/s20030648, URL https://www.mdpi.com/1424-8220/20/3/648.
[109] H. Mroue, B. Parrein, S. Hamrioui, P. Bakowski, A. Nasser, E.M. Cruz, W. Vince, LoRa+: An extension of LoRaWAN protocol to reduce infrastructure
costs by improving the quality of service, Internet Things 9 (2020) 100176, http://dx.doi.org/10.1016/j.iot.2020.100176, URL https://www.sciencedirect.
com/science/article/pii/S2542660520300160.

27