Mobile User Environment Detection using Deep

Learning based Multi-Output Classification

Illyyne Saffar, Marie-Line Alberi-Morel, Mohanned Amara, Kamal Deep
Singh, César Viho

To cite this version:

Illyyne Saffar, Marie-Line Alberi-Morel, Mohanned Amara, Kamal Deep Singh, César Viho. Mo-
bile User Environment Detection using Deep Learning based Multi-Output Classification. WMNC
2019 - 12th IFIP Wireless and Mobile Networking Conference, Sep 2019, Paris, France. pp.16-23,
�10.23919/WMNC.2019.8881474�. �hal-02437854�

HAL Id: hal-02437854

https://hal.science/hal-02437854
Submitted on 19 Jun 2021

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est

archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents
entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non,
lished or not. The documents may come from émanant des établissements d’enseignement et de
teaching and research institutions in France or recherche français ou étrangers, des laboratoires
abroad, or from public or private research centers. publics ou privés.
 Mobile User Environment Detection using
Deep Learning based Multi-Output Classification
Illyyne Saffar∗‡ , Marie Line Alberi Morel∗ , Mohanned Amara∗ , Kamal Deep Singh† , Cesar Viho‡
∗ Nokia
Bell Labs, Nozay, France ‡ Univ Rennes, Inria, CNRS, IRISA, France
† Univ Lyon, UJM Saint-Etienne, CNRS, Laboratoire Hubert Curien UMR 5516, Saint-Etienne, France
Email: [email protected], marie [email protected], [email protected],
[email protected], [email protected]

Abstract—Future mobile networks can hugely benefit In literature, the environment detection issue has been stud-
from cognition of mobile user behavior. Indeed, knowing ied mainly considering an Indoor Outdoor Detection (IOD)
what/when/where/how the user consumes their mobile services binary classification to detect the environment [3], [4]. How-
can notably improve the self-adaptation and self-optimization ca-
pabilities of these networks and, in turn, ensure user satisfaction. ever, identifying the user’s environment is a more complicated
The cognition of mobile user behavior will thus help 5G networks task than just IOD. Actually, the environment deeply affects
to face the variable consuming habits of users which in turn the way a user interacts with his or her mobile phone. He or
impact the network conditions, by predicting them in advance. she usually tends to have different attitudes at home than at
In this paper, we focus on the “where” part, i.e., the detection work, at a café than in transport, during the working week
of the environment where a given user consumes different
mobile applications. A statistical study on the real activity of than during the week-end [5], etc. The way, individuals use
users reveals that there are multiple various environment types their phones, varies notably as a function of these factors.
corresponding to the mobile phone usage. A Deep Learning based Classifying the environment as only two states (Indoor or
model is introduced to intelligently detect the user environment, Outdoor) can be seen as a first level of analysis, but it does not
using supervised and semi-supervised multi-output classification. reveal the complexity of the situation. So, a more granular clas-
Relevant multi-class schemes are proposed to efficiently regroup
the multiple environment categories in more than two classes. sification is desired in order to detect/predict where a mobile
We empirically evaluate the effectiveness of the proposed model user uses/prefers certain mobile applications. However, a more
using new real-time radio data, gathered massively from multiple detailed classification impacts the distribution of measurement
typical and diversified environments of mobile users. instances into the resulting classes. This is because users prefer
Index Terms—User Environment Detection, Multi-output clas- to use their mobile phone in specific situations [6]. As a
sification, Deep Learning, Semi-supervised Learning, 3GPP sig-
nals, real user activity.
consequence, some classes (the most popular) have much more
measurement points than others. This unbalancing between
the categories of various environments, observed in case of
I. I NTRODUCTION two classes in [3], is also augmented when classifying with
more than two classes. Therefore, achieving a multi-output
Autonomic 5G networks are the next evolution of mobile classification will first require to tackle this phenomenon.
networks for accommodating the ever-growing user-demands In this paper, we investigate the user environment detection
for better services and applications with high Quality of using a supervised as well as a semi-supervised multi-output
Experience with lowest cost of management and operation [1]. classification technique. Multi-output refers to the detection
Such objectives are achievable thanks to the analysis of mobile of multiple types of environment. We study relevant multi-
user data in order to extract actionable knowledge related to class schemes that smartly regroup the multiple environment
user profile or the use context of mobile services. categories in more than two classes. However to what level
According to Kipling’s method [2], a mobile user profile and detail the model requires to classify a user’s environment?
is defined in terms of how/ when/ what/ where/ who/ why Which classification scheme leads to better balanced data? Can
(5W1H) the user consumes mobile applications. The cognition we classify the user environment with detailed classes and with
and knowledge gained about the consuming habits of individ- good performance? In the following, a comparative analysis of
uals and communities are then exploited by mobile autonomic classification between schemes of three, four and five classes
5G networks to grow smarter. Furthermore, this can make will show that using four classes with relevant labels enables
5G networks more efficient when faced with the increasing a good trade-off between balancing and more granularity.
complexity of network management combined with numerous For the environment detection task, the deep-learning based
new applications and their heterogeneous needs. In this paper, approach has shown the best performance over classical ma-
we focus on detecting the environment where a mobile user chine learning [12]. Actually, this approach is preferred to
connected to a cellular network experiences his or her mobile achieve the multi-output classification since it is appropriated
applications. for resolving problems where modeling relationships between
ISBN 978-3-903176-18-8 © 2019 IFIP large number of parameters are not tractable. This is the case
when classifying the environment into more than two classes. outdoor as environment classes. For IOD, they propose a
Indeed the model has to extract the complexity of the variety semi-supervised approach using a co-training based solution.
of situations met by the mobile users. This detection of en- They used 2 classifiers in parallel with a weighted score of
vironment will help the operator to perform online adaptation classification probability to improve the final performance of
and optimisation of network and radio parameters according to IOD. To train these classifiers they not only considered the
users’ environment types, i.e, depending on what percentage same measurements set coming from the phone sensors as
of users are in indoor environment, how many in transport, [9], but also added sound intensity, battery temperature and
etc. This will also help the operator to provide context-aware the proximity sensor. The set of these papers deals with the
services. The evaluation of the proposed solution will be done IOD from phone devices side with UE-specific data which
using real and large 4G LTE radio data collected through a in turn is in most cases not known by the mobile networks.
crowdsourcing approach, i.e. collected by the phone device Moreover [9] uses a computed threshold to make a classifi-
and sent to the mobile network via 3GPP procedures [3]. Thus, cation decision between two states. Extending such methods
Machine Learning algorithms trained on datasets collected in to make a classification decision between a plurality of states
crowdsourcing mode allow to learn very diverse real-world (more than 2) is too complicated and is not performant.
environments. Deep learning lately is gaining much popularity The work in [10], used a Bayesian detector that combines
due to its supremacy in terms of accuracy when trained with the signals measured from the cellular (RSRP provided by the
huge amounts of data. Deep Learning’s performance continues cellular modem) and GNSS receiver (the confidence radius
to improve when more and more data is used for training. This of the location, provided by the active localization sensor).
results in it outperforming the traditional models/algorithm of This information provided by both measurements is combined
Machine Learning [7], [8]. However, using real data requires with a joint posteriori probability based on the distributions
to tackle the problem of data cleaning and also needs to of RSRP and GPS measurement to perform indoor/outdoor
overcome the noise inherent to real data, which in turn is not detection. In [11], authors localize mobile users by accurately
synthetic. In addition, we investigate optimization solutions estimating their longitude and latitude. As a first step of
during the training phase. We focus mainly on the initialization localization process, they also detect the user environment
of the model parameter values before training and on the type: indoor or outdoor. For this IOD step, they used RSRP
generalization of the model for various user profiles using an and RSRQ signals and tested many learning algorithms: SVM,
additional Gaussian Layer. logistic regression and random forest. In both [3] and [12],
The remaining paper is organized as follows. In Section authors studied UED as a binary classification (Indoor vs.
2, related existing works are discussed. Section 3, provides Outdoor) and using a Deep Learning approach. In [3], authors
an analysis of the user activity according to the environment used 4 signals for the UED task: a power signal RSRP, a
type. Section 4, describes the data collection and investigates quality signal CQI that replaced the RSRQ because the latter
multiple scheme of multi-output classification. The solution is not often reported to the network and finally the Timing
adopted to optimize the classification is detailed in Section 5. Advance. In [12], authors used the same signals as [3], but with
In Section 6, results of the environment detection system are addition of a mobility indicator to solve some difficult cases of
analysed and discussed. Finally conclusion and perspectives detection like when the user is in train (outdoor environment),
are presented in Section 7. and suffers from a drastic deterioration of the RSRP signal,
then the Mobility Indicator helps to better detect such case.
II. R ELATED WORK
Both papers [3] and [12] show good performance of the UED
In the literature, to the best of our knowledge, the user en-
vironment detection (UED) issue has not been largely studied.
So far, what has been carefully studied in literature, is UED as
a binary classification or detection problem, with only indoor
and outdoor as the classes.
In [9], authors detected the user environment according
to three environment types: indoor (inside a building), semi-
outdoor (close to or semi-open building) and outdoor (outside
a building). For such detection, they used a combination of
signals collected from some phone sensors related to radio
signals, cell signal strength, light intensity as well as the
magnetic field to infer whether the environment state is indoor,
outdoor or semi-outdoor. Detecting user environment with
only such granularity does not provide sufficient knowledge
for the mobile network to relate the detected use context with
the user’s application preferences. The user consumption will
be almost the same in an indoor environment as in semi-
outdoor. In [4], authors studied UED with only indoor and Fig. 1: Data collection points in France: multiple places
binary classification, i.e, around 95% of F1-score (a measure
of a machine learning accuracy).
However, considering UED as a classification with multiple
outputs (multiple environment classes) can be of interest for
user profiling. In this paper, we rather focus on UED for user
profiling. Thus, we use a deep learning approach trained on
a large and real dataset. We consider several input signals
which are collected within the network side. We look at the
performance in terms of F1-scores of supervised or semi-
supervised UED deep learning methods.

III. DATA COLLECTION Fig. 3: CDF of RSRP

Data constitutes the fuel and the core of any machine
learning or deep learning process. For our study, the collected
reflecting all weather types. With such collecting mode the
data comes from a large crowdsourcing campaign, collected
dataset built closely captures the complexity and the variety
by phone devices and sent to the mobile network. Using
of a mobile user moving in real world.
measurements done by a phone device is the most reasonable
choice for user behaviour analyses and user profiling as it For the user environment detection, a specific dataset is
allows to capture the real behavior of mobile users. Indeed, due built consisting of radio signals (RSRP, CQI), time related
to their small size and popularity, the portable phone devices features (TA, Time) and mobility indicator (MI) and finally
are always with users during their various activities in all kinds the environment label when it exists. The dataset is composed
of environments. of 6-feature samples as following:
As shown in Figure 1 our dataset has been collected in • Time: the recording time of signal or burst data arrival.
different locations in France: red dots on the map indicate the • RSRP: the average received power of the reference signal.
main ones, and the roads linking these locations where users RSRP varies between -140 dBm and -44 dBm.
are moving with different speeds. • CQI: Channel Quality indicator which is the selection tool
The data has been collected during 16 months, 24h/7, with for the best fitting transmission modulation and coding
an average of 1 measurement per 15 seconds, when the mobile scheme to be used in communication.
phone session is active, and 1 measurement per 2 minutes • TA: Timing Advance is used to control UL signal trans-
otherwise. Thus, around 2M lines of data per user have mission timing.
been collected. The collection has been performed in many • MI: the number of the Cell ID changes during a sliding
different indoor/outdoor environments. Indoor corresponds to window of length equal to 100s as fixed in [12].
the following locations: at home, in restaurant, in cafe, at • Environment: the label of environment crossed by user
work or in other types of building, etc. whereas, outdoor is during the measurement campaign.
associated to forest, streets, parks, mountain and beach, to a We consider the following categories of environment: [Work,
pedestrian, a running user, or a user in car moving with high Home, Building, Bus, Car, Mall, Pedestrian, Train]. They have
speed, etc. The gathering was done in many cities and places been chosen for labelling the data. This is because they are
in France. This long collection period allows us to have data highly representative of the locations where data have been
collected. The environment labeling was processed using GPS

Fig. 2: User activity per environment Fig. 4: CDF of number of Cell ID changes during 100s
coordinates for users who allowed us to collect them. Figure 2 illustrates the multiple diversity of indoor and
outdoor situations met by the users of mobile phone. These
IV. M ULTI - OUTPUT CLASSIFICATION SCHEMES situations are [Work, Home, Building, Bus, Car, Mall, Pedes-
trian, Train]. Consequently, it is preferable to consider the
A. Relation between user activity and environment type
user environment detection task as a multi-output classification
A better identification of the environment classification problem. We observed also in Figure 2 that the data instances
problem can be achieved when analysing the user’s activity are distributed unequally between the different categories. It
according to his or her environment. This investigation is shows that the data proportion in groups of label “Train”,
done with respect to the eight types of environment that “Bus”, “Mall”, “Building” or “Pedestrian” is very low com-
characterise the locations crossed during the measurement pared to the other groups. This unbalanced nature of data
campaign: [Work, Home, Building, Bus, Car, Mall, Pedestrian, remains a challenge for Machine Learning algorithms
Train]. These environment categories were selected in order
to reflect the complexity of a user’s daily life and capture the B. Classification schemes
variety of his movements in real world. In this section, the objective is to define relevant classifica-
The ‘phone usage’ state is defined as the state when the tion schemes with multiple classes by regrouping smartly the
user uses his or her phone or equivalently the screen is on various environment categories given in section III.
and unlocked and there is data exchanged. In order to track As a matter of fact, the issue of unbalanced data is not
this state, following additional data features are also collected related to the data collection constraints, but it is rather,
during the campaign. These features are the instance index inherently, due to the nature of real human activities. As
or timestamp, the screen state and the label of environment a consequence, in order to ensure an efficient scheme of
type. Analysing this data gives us a clear picture of mobile environment classification for real user activity, we have to
phone usage in different environments. Figure 2 depicts the find a trade-off to limit this inherent data bias for different
user activity by plotting the phone usage ratio for different classes. We target to design a classification scheme that detects
environment categories. User activity is measured as the ratio the detailed environment types of mobile users, with a fine
between the number of instances the user has been using his granularity and a low decision error margin. We fixed this
phone effectively and the total number of instances. This figure margin at 5%. An investigation has to be done to guess the
illustrates the percentage of total time the user is connected to relevant minimal value. But, it is out of scope of this paper.
4G network and exchanges data with it. We observe that most We focus on studying the relevant trade-offs in our case of
of the activity is spent more indoor (70 %), mainly at home and multi-output classification. We consider different possibilities
at work, as compared to being outdoor. Figure 2 highlights the ranging from a simple binary classification problem to a more
user activity trends that we observed after statistical analysis complex classification task for detecting detailed information
on mobile user behavior in literature. about a user’s environment. To ensure this, we decide to
Contrary to traditional TV users, who just watch scheduled regroup the environment categories {Work, Home, Building,
programs, mobile service users are free to choose the content Mall, Bus, Car, Pedestrian, Train}. Furthermore, these cate-
they want, at any point in time and space. During a day, a user gories have thick inter-relationships and we can merge them
can be in different situations, such as walking outdoor, in a car, into bigger consistent and recognizable groups. The created
at work, in a mall, in a café or at home. As a matter of fact, merged groups shall also include data with similar statistical
mobile users’ preferences for certain applications or contents properties in order to optimize the classification results. This
is linked with the usage situations [5]. In the literature, some is obtained by observing the cumulative distribution curves
statistical studies show that mobile phones are mostly used of the collected data {RSRP ,CQI,T A,M I} as well as the
in a building for internet service (80%) and for a call (70 variance σ 2 of the phone activity for each proposed multi-class
(Xi −X̄)2
P
2
%) [6]. This can be explained by the fact that the different schemes. The variance is written as: σ = N where
use contexts pose their own limitations, which in turn impact Xi is the number of instance in the class i and N the number of
the potential application usages. In [13], [14], [15], the most classes. This represents the percentage of total instances linked
commonly mentioned physical environments of application to this environment. The variance is an appropriate metric to
usage are indoor (waiting halls or lounges, work, home and measure the degree of unbalance by quantifying the variance
cafes), but also include vehicles, such as public transportation of the percentage of total instances per environment category
and private cars. The home environment is preferred by users, from its average.
while the bus or car environment is not for using the phone. Figures 3 and 4 depict respectively the cumulative distribu-
This is explained by the calm and the pleasant atmosphere tion curves of RSRP and M I. We observed in Figure 3 that
which is more suitable for focusing on viewing. Whereas, in three groups of similar curves can be extracted: a set with only
motion, the complicated tasks generally result in an unpleasant Home, a set regrouping { Work, Building, Mall} and another
entertainment experience. Furthermore, people spend most of set assembling {Bus, Car, Pedestrian, Train}. This results into
their time indoor than in mobility and outdoor. Consequently, a further split of indoor and outdoor classes. Furthermore, the
outdoor labels are less represented than indoor labels in phone environment “Home” is detached from the set of other indoor
usage data. situations. Analysing the Figure 4, we also note a similar
separation between indoor and outdoor curves. Moreover, CDF while, “5C 1” contains three classes “mall”, “home” and
curves of MI highlight a clear separation between “pedestrian” “building” (namely “works” and various “buildings”).
and the remaining set of outdoor labels and between “Mall” Figure 6 shows the variance of data size when the data is
and the others indoor situations. Clearly these two environment divided according to the above schemes. This is to quantify the
types are associated to users moving very slowly (walking) as balance or unbalance of data between different classes. When
compared to others that are either static or high speed. the variance is high the dispersion, in terms of data size in
different classes, is important and, thus, the scheme is very
unbalanced. As shown in Figure 6, the schemes “3C 0” and
“3C 1” deliver the smallest variance. They are followed by
“4CO 1”, “4CI 1” and “5C 1”. Indeed, the scheme 3C results
in the most balanced data among different classes. This scheme
regroups the instances coming from outdoor labels and splits
the indoor label instances. All the schemes are illustrated In
Figure 10. The figures represents the phone activity according
Fig. 5: Multiple class schemes example: “5C 0” and “5C 1” the environment type for the 8 proposed schemes.
Later, we will evaluate the relevance of a more detailed
environment detection as compared to outdoor or indoor.
Based on these observations, we propose eight grouping
schemes derived from the classes “indoor” and “outdoor” V. C LASSIFICATION OPTIMIZATION
investigated in [12]. The label “indoor” refers to “home”, A. Hyper-parameter Tuning
“work”, “mall” and “buildings”. The label “outdoor” regroups
The hardest step in every deep learning training procedure
the environments “pedestrianv”, “car”, “bus” and “train”.
is to find the best hyper-parameters set for the model. Hyper-
Grouping schemes of 3, 4 and 5 classes are then investigated
parameters are the set of variables to be optimized before
to better model the diversity of environments. The studied
applying any learning algorithm to a dataset. The set of
schemes are as follows (for illustration, some examples are
hyper-parameters is composed of number of hidden layers,
provided in Figure 5):
batch size, epoch size, the weight initialization, the activation
• “3C 0” and “3C 1”: these two schemes include three function, the loss function, the learning rate, the momentum
classes in total. Both include a class “outdoor” that and the dropout fraction. The challenge with these hyper-
groups all the outdoor categories. The scheme “3C 0” parameters is that there is no magic combination that always
contains two additional indoor classes “work” and “build- works. The best hyper-parameters combination depends on
ing” (grouping “home”, “mall” and various “buildings”). each task and also on each dataset. In this paper, we focus
Whereas, the scheme “3C 1” contains two additional in- on 3 main strategies: i) Manual Search, ii) Grid Search,
door classes which are “home” and “building” (grouping iii) Bayesian model-based optimization. Manual Search is
“home”, “mall” and various “building”). the traditional way to set hyper-parameters, i.e, setting them
• “4CO 0” and “4CO 1”: these two schemes include four manually by trial and error many times and quantifying the
classes in total. Both include a unique class “outdoor”. errors for each trial. This method is the fastest way to set the
The scheme “4CO 0” contains three more indoor classes hyper-parameters, but it never guarantees that we will have
“work”, “mall” and “building” (assembling “home” and the right combination and good classification performance.
various “building”). Whereas, the scheme “4CO 1” con- In [3], [12] authors used grid search to optimize the hyper-
tains three other indoor classes which are “home”, “mall” parameters of their deep learning model. Actually grid search
and “building” (assembling “work” and various “build-
ing”).
• “4CI 0” and “4CI 1”: these two schemes include four
classes. Both include a class “outdoor” split further in
two classes “pedestrian” and “in-transport” (which in
turn regroups “bus”, “train” and “car”). The scheme
“4CI 0” contains two additional indoor classes “mall”
and “building” (regrouping “home”, “work” and various
“buildings”). Meanwhile, the scheme “4CI 1” contains
two other indoor classes “home” and “building” (merging
“work”, “mall” and various “buildings”).
• “5C 0” and “5C 1”: these two schemes include five
classes. Both include a class “mall” and two classes
“pedestrian” and “in-transport”. The scheme “5C 0” con-
tains three additional classes “mall”, “work” and “build- Fig. 6: Variance of phone activity for the multiple class
ing” (namely “home” and various “buildings”). Mean- schemes
consists of comparing all possible combinations of hyper-
parameters and then returning the best set in terms of the
returned objective function value. The challenge with hyper-
parameters optimization is to minimize the number of times
the objective function is evaluated. Actually, the evaluation
of the objective function is generally very expensive. With a
complex model, a large dataset, and a high dimensionality of
hyper-parameters space, the problem becomes intractable and
grid search becomes inefficient because of its greediness.
Recently a new approach using a Bayesian optimization
for tuning hyper-parameters has been considered [16]. Based
on a probabilistic model of the objective function called
the surrogate function, Bayesian optimization reduces the
frequency of calls to the the objective function, to the lowest
possible. The surrogate model is represented by the probability Fig. 7: Supervised Training Model (Feed Forward Neuronal
of the score knowing a fixed hyper-parameters’ set (P (score | Network)
hyperparameters)). Actually, the use of a surrogate model
enables the algorithm to select the most promising hyper-
parameters for the objective function evaluation, so that, the The set of hyper-parameters (e.g. the number of hidden
search does not spend a significant amount of time on looping layers, batch size, epoch size, the weights) have been tuned
on bad combination of hyper-parameters. That is to say, such using Bayesian optimization. The implementation is done
optimization leads to a faster convergence of the deep learning under python and using keras with tensorflow as a back-
based model. end. For training and test, we have collected around 2M lines
of data per user. In this paper, we used 270K lines of data
B. Adding Gaussian noise collected only in France and corresponding to LTE networks.
In [17], the authors have shown that by allowing a lit- The dataset is made of 50% of labeled data and 50% of
tle random margin of inaccuracy in Feed Forward Neural unlabelled data. The training is done using the labeled part of
Networks, a model can perform better on both training and our dataset. For training, we used (70%) of the labeled data
accuracy. Neural Networks are able to model functions that and we used the (30%) remaining for the model performance
change their outputs spectacularly in response to just a small evaluation. As the multi-output classification problem leads to
input variation. Some noise is introduced to the neural network a issue of unbalanced classes, F1-score metric in addition to
model via a Gaussian Noise layer during the training phase. the accuracy is used for performance evaluation. F1-score is
Noise permits to have some robustness in the output and one of the most used metric in case of unbalanced data classes.
smoothed decision borders. Thus, noise helps reduce the The metric by definition is the weighted average of Precision
chance of over-fitting, even when training dataset is small, and Recall according to the following relation:
and aids in the generalization of the model. Thus, we add a
Gaussian Noise layer inside our model. It consists in adding P recision.Recall
F 1 − score = 2.
noise to output of the layer before the activation function. P recision + Recall
The added random value, i , follows a normal distribution
i ∼ N (0, Σ), i.e., with a mean equal to zero and the co- where Precision is the number of correct positive results
variance matrix of features Σ. divided by the number of all positive results returned by the
Precisely, the figure indicates the percentage of time a user classifier, and Recall is the number of correct positive results
is active in his or her phone. divided by the number of all relevant samples.

VI. E XPERIMENTAL S ETUP AND RESULTS

A. Configuration and performance metric
For our supervised multi-output classification problem, we
used a Feed Forward Neural Network architecture with a total
of 7 hidden layers that is depicted in Figure 7. To evaluate
the impact of injecting more data during the training phase
[18], we used a semi-supervised training model detailed in
[12] that is depicted in Figure 8. The model is trained with
a part of labelled data and another part of unlabeled data.
Authors in [12] demonstrate its good performance. We plan to
evaluate this brought when using in a semi-supervised multi-
output classification problem. Fig. 8: semi-supervised Training Model (Self-Training [12])
 2C 3C0 4CO0 4CI0 5C0
Accuracy F1-Score Accuracy F1-Score Accuracy F1-Score Accuracy F1-Score Accuracy F1-score
Supervised 95.8% 95.76% 91.89% 91.84% 91.58% 91.53% 93.18% 93.11% 90.70% 90.69%
Semi-Supervised 96.51% 96.45% 94.20% 94.19% 94.24% 94.24% 94.92% 94.92% 93.42% 93.38%
3C1 4CO1 4CI1 5C1
Accuracy F1-Score Accuracy F1-Score Accuracy F1-Score Accuracy F1-score
Supervised 92.06% 92.04% 91.42% 91.38% 94.31% 94.28% 90.79% 90.72%
Semi-Supervised 94.14% 94.13% 94.18% 94.17% 94.53% 94.55% 93.66% 93.63%

TABLE I: Deep Learning-based supervised and semi-supervised multi-ouput classification performance: F1-score vs. classifi-
cation schemes - Training with Noise Layer with and without using self-learning

B. Optimization performance results

To quantify the impact of the optimization method, we fix
the classification scheme to be 4CI 1. Now, if we fix the
number of the hidden layers to 7, the training step of the
neural network takes in average 130.86second. The challenge
is to find the best combination of the hyper-parameters that can
learn the best UED classifier. For optimization, we consider 7
hyper-parameters: the epoch size, the batch size, the activation
function, the initialization, the learning rate, the dropout rate,
the optimizer, and the number of neurons per layer. Different
values of these hyper-parameters and their combinations are
very large. This means that the convergence time to train
the best model is also very high. Let us assume that we Fig. 9: Impact of optimization methods on the accuracy
loop among 5 possibilities or values of the epoch size, 5 computing
possibilities of the batch size, 3 possibilities of the activation
function, 3 possibilities of the initialization, 3 possibilities
of the learning rate, 5 possibilities of the dropout rate, 5
possibilities of the optimizer, and 4 possibilities of the number
of neurons per layer. This leads us to test 472500 neural
and “4CI 0” obtained in supervised case are equal to 94.28%
networks and it would take around 472500 ∗ 130.86 seconds
and 93.11%, respectively. We notice a slight improvement of
for them to train. To limit the run time, we investigated 3
F1-score using the two-class scheme as compared to the F1-
methods: manual search, grid search and Bayesian search as
score obtained under same conditions in [12]. Indeed, this
shown in Fig. 9. We set a threshold of 50 iterations for both
enhancement is due to the fact that now our algorithms process
the Bayesian optimization and the grid search (an iteration
two times more real radio data collected in real conditions of
= one combination of hyper-parameter for training a model).
crowd-sourcing. Consequently, it impacts the performance in
As expected the manual search is the fastest one since it is
a favorable way. Using the semi-supervised method for multi-
conducted by a human (10 runs at most). As shown in Fig. 9
output detection is furthermore positive. It still enhances the
and supported by analysis before, the Grid search is too slow
scores thanks to the addition of unlabelled data in the training
and the computation time is high. In the worst case, the best
phase. F1-scores for the three schemes are equal to 96.45%,
model is delivered after looping on all combinations of hyper-
94.55% and 94.92%, respectively. In both methods, we also
parameters. Whereas, Bayesian method finds the right range
observe a maximum loss of around 6% when using a five-class
and parameters space from the first iterations.
scheme as compared to the binary classification. The loss is
C. Multi-ouput classification performance reduced to around 2% when using a 4-class scheme.

Table I presents the performance results of a supervised Thus, we show that a detailed learning of the environment
and semi-supervised multi-output classification for the 8 multi- can be achieved with a very minimal loss of performance.
output classification schemes as well as the two-class scheme. This is obtained using a smart cutting of the groups “indoor”
They are evaluated in terms of F1-score and then compared and “outdoor”. We observe that the coarse learning of the
with a classical IOD binary classification. We observe that all environment benefits more from the diversity brought by the
schemes deliver F1-scores higher than 90% that corresponds introduction of labels “pedestrian” and “in-transport” than by
to acceptable performance in terms of classification. However, the split of the group “indoor”. Furthermore, we also note
among all the schemes the two-class scheme and the two that the level of unbalance in data has an influence on the F1-
schemes “4CI 1” and “4CI 0” give the best performance. score. The most unbalanced scheme delivers small F1-scores.
As shown in Table I, the scheme “2C” is equal to 95.76% The two best schemes (outside the two-class scheme) offer
when supervised training is used. The F1-scores of “4CI 1” relatively balanced classes.
 Fig. 10: Phone Activity for various classification schemes of environment

VII. C ONCLUSION [2] JAVAID Nadee, SHER Arshad, NASIR Hina Nasir, GUIZANI Nadra.
Intelligence in IoT-Based 5G Networks: Opportunities and Challenges.
In order to deliver context-aware services, it is important In : IEEE Communications Magazine, October 2018. p. 94-100.
to first detect the use context. In this paper, we focused on [3] SAFFAR, Illyyne, ALBERI MOREL, Marie Line, SINGH, Kamal Deep,
detecting user environment which is an important part of the et al. Machine Learning with partially labeled Data for Indoor Outdoor
Detection. In : 2019 16th IEEE Annual Consumer Communications and
use context. Instead of just detecting the environment as indoor Networking Conference (CCNC). IEEE, 2019. p. 1-8.
or outdoor, we focused on detecting more detailed classes of [4] RADU, Valentin, KATSIKOULI, Panagiota, SARKAR, Rik, et al. A
the user environment. semi-supervised learning approach for robust indoor-outdoor detection
with smartphones. In : Proceedings of the 12th ACM Conference on
We showed that the mobile network can detect the user Embedded Network Sensor Systems. ACM, 2014. p. 280-294.
environment using a multi-class classifier trained on real data. [5] S. Ickin et al., Factors influencing quality of experience of commonly
This classifier uses standardised mobile network signals as used mobile applications, IEEE Communications Magazine, vol. 50, no.
4, pp. 48-56, 2012.
input. We studied how to divide the initial indoor/outdoor
[6] http://www.hetnetforum.com/, HetNet Forum, 2018
classes further into more detailed environment types. However, [7] ZHANG, Chaoyun, PATRAS, Paul, et HADDADI, Hamed. Deep Learn-
real data based training poses the problem that the data ing in Mobile and Wireless Networking: A Survey. arXiv preprint
corresponding to different classes can be highly unbalanced. arXiv:1803.04311, 2018.
[8] LECUN, Yann, BENGIO, Yoshua, et HINTON, Geoffrey. Deep learn-
This unbalance is not due to the way with which we collected ing.nature, 2015, vol. 521, no 7553, p. 436.
data collection, but is strongly related to the inherent user [9] ZHOU, Pengfei, ZHENG, Yuanqing, LI, Zhenjiang, et al. Iodetector: A
behavior and his preferences to services consumption. Know- generic service for indoor outdoor detection. In : Proceedings of the 10th
acm conference on embedded network sensor systems. ACM, 2012. p.
ing that this data unbalance can be a problem for machine 113-126.
learning, we tried to find a compromise between the data [10] MEKKI, Sami, KARAGKIOULES, Theodoros, et VALENTIN, Stefan.
instances per class and the granularity degree by varying the HTTP adaptive streaming with indoors-outdoors detection in mobile
number of environment classes. Thus, we studied different networks. arXiv preprint arXiv:1705.08809, 2017.
[11] RAY, Avik, DEB, Supratim, et MONOGIOUDIS, Pantelis. Localization
combinations of splitting the environment into different classes of LTE measurement records with missing information. In : Computer
and compared their performance. The performance peaked for Communications, IEEE INFOCOM 2016-The 35th Annual IEEE Inter-
2 class classifier as it is the easiest and for 4 class classifier national Conference on. IEEE, 2016. p. 1-9.
[12] SAFFAR, Illyyne, ALBERI-MOREL, Marie-Line, SINGH, Kamal
which offers more detailed environment classification than just Deep, et al. Semi-supervised Deep Learning-based Methods for Indoor
indoor and outdoor. Outdoor Detection. In : IEEE International Conference on Communica-
In future, we plan to build a classifier that can detect more tions (ICC). 2019.
[13] V. A. Siris, K. Balampekos and M. K. Marina, Mobile Quality of
components corresponding to the use context and not only the Experience: Recent Advances and Challenges, Workshop on Information
environment types. We aim also to quantify the impact of the Quality and Quality of Service for Pervasive Computing, 2014
environment and the other components on the context-aware [14] T. Soikkeli, J. Karikoski, H. Hämmäinen, Diversity and End User
services. Context in Smartphone Usage Sessions, Next Generation Mobile Ap-
plications, Services and Technologies (NGMAST), 5th International
Conference, 2011, pp. 7-12.
R EFERENCES
[15] E. Kaasinen, User Acceptance of Mobile Services - Value, Ease of Use,
[1] PIRINEN, Pekka. A brief overview of 5G research activities. In : 1st Trust and Ease of Adoption, 2005
International Conference on 5G for Ubiquitous Connectivity. IEEE, [16] BERGSTRA, James, YAMINS, Dan, et COX, David D. Hyperopt: A
2014. p. 17-22. python library for optimizing the hyperparameters of machine learning
 algorithms. In : Proceedings of the 12th Python in science conference.
2013. p. 13-20.
[17] SIETSMA, Jocelyn et DOW, Robert JF. Creating artificial neural net-
works that generalize. Neural networks, 1991, vol. 4, no 1, p. 67-79.
[18] Andrew Ng Slides, https://media.nips.cc/Conferences/2016/Slides/6203-
Slides.pd