This article has been accepted for publication in IEEE Transactions on Industrial Informatics.

This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2022.3199374

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. XX, NO. XX, XXXX 1

iTimes: Investigating Semi-supervised Time

Series Classification via Irregular Time Sampling
Xuxin Liu, Fengbin Zhang, Han Liu, and Haoyi Fan

Dataset
Abstract— Semi-supervised learning (SSL) provides a

Labeled
Full

Data
powerful paradigm to mitigate the reliance on large labeled
data by leveraging unlabeled data during model training.
However, for time series data, few SSL models focus on the
Self-supervised Transformation Type Prediction Supervised Classification
underlying temporal structure of time series, which results
in a sub-optimal representation learning quality on unla- Label 1:
beled time series. In this paper, we propose a framework of
semi-supervised time series classification by investigating Label 2:

irregular time sampling (iTimes), which learns the underly-

Trans. Type 1 Trans. Type 2 Trans. Type N Label C:
ing temporal structure of unlabeled time series in a self-
supervised manner to benefit semi-supervised time series
classification. Specifically, we propose four different irreg-
ular time sampling functions to transform the original time Fig. 1. Schematic diagram of the proposed semi-supervised time series
series into different transformations. Then, iTimes employs classification framework, which conducts the supervised classification
a supervised module to classify labeled time series directly on labeled data and conducts the self-supervised transformation (trans.)
and employs a self-supervised module on unlabeled time prediction on unlabeled data.
series by predicting the transformation type of irregular
time sampling. Finally, the underlying temporal structure
pattern of unlabeled time series can be captured in the
self-supervised module. The feature spaces between la- aforementioned data annotation challenges [6]–[9]. Semi-
beled data and unlabeled data can be aligned by jointly supervised learning aims to combine a tremendous amount of
training the supervised and self-supervised modules which unlabeled data with limited labeled data during model training
boost the ability of model learning and the representation
quality. Extensive experimental results on multiple real- to boost the model’s performance. To alleviate the dependence
world datasets demonstrate the effectiveness of iTimes of labeled time series, Dynamic Time Warping (DTW) [10],
compared with the state-of-the-art baselines. and Maximum Diagonal Line of Cross-Recursive Quantifi-
Index Terms— Irregular time sampling, Self-supervised
cation Analysis (MDL-CRQA) [11] distance-based methods
learning, Semi-supervised learning, Time series classifica- are used as classification criteria for one-nearest-neighbor (1-
tion NN). Different from these measure-based methods, techniques
such as spectral analysis, phase consistency regularization and
adversarial training are applied in [7] for semi-supervised
I. I NTRODUCTION learning. Recently, self-supervised learning has achieved great

T IME series classification is a fundamental task in many

application domains such as healthcare monitoring [1],
[2], activity recognition [3], power system security [4], [5]
success in natural language processing and computer vision
domains, which learns a useful representation of unlabeled
data via defining and solving various pretext tasks. Inspired by
and fault diagnosis [6]. In recent years, convolutional neural this, the subsequence forecast is selected as pretext task in [8]
networks (ConvNets) have been extensively used and achieved for self-supervised learning on unlabeled time series. Despite
much success in this field. Specifically, by training the Con- the encouraging results of these methods, few of them take into
vNets using massive manually labeled training data, they man- account the underlying temporal structure in the time series,
aged to learn the representations facilitates the classification which leaves the representation of learning on unlabeled time
task [6], [7]. However, data annotation is usually expensive series under-explored. More recently, SemiTime [9], takes a
and time-consuming, and even impossible in some domains past-future relation prediction of time segments as the pretext
because of expert annotation knowledge. task, which explores the underlying temporal structure within
Due to that, semi-supervised, self-supervised learning has time series, has made progress. However, the past-future
gained much attention, which provides a solution to the relation prediction tries to capture a coarse-grained temporal
structure, which results in a sub-optimal representation quality
Xuxin Liu, Fengbin Zhang and Han Liu are with the School on time series. Therefore, how to design an efficient self-
of Computer Sicence and Technology, Harbin University of Science
and Technology, Harbin, 150080, China (e-mail: [email protected], supervised pretext task that captures the temporal structure
[email protected], [email protected]). of unlabeled time series is still an open problem for semi-
Haoyi Fan, School of Computer and Artificial Intelligence, Zhengzhou supervised time series classification.
University, Zhengzhou, 450001, China (e-mail: [email protected]).
(Fengbin Zhang and Haoyi Fan are the corresponding authors. ) To cope with the aforementioned challenges, in this paper,

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Harbin University of Science and Technology. Downloaded on October 29,2022 at 04:18:54 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2022.3199374
2 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. XX, NO. XX, XXXX

we propose a framework of semi-supervised time series clas- to the different input perturbations, e.g., apply different data
sification by investigating irregular time sampling (iTimes), augmentations to the same input. For example, Π-Model [14]
which learns the underlying temporal structure of unlabeled uses self-ensembling to form a consistent prediction of un-
time series in a self-supervised manner to benefit semi- labeled data under different augmentation and regularization.
supervised time series classification. Specifically, as shown in MixMatch [15] predicts low-entropy labels for data-augmented
Fig. 1, for labeled data, iTimes classify the time series directly unlabeled examples and mixes labeled and unlabeled data
under the supervision of the annotated label. For unlabeled using MixUp for training on unlabeled data. FixMatch [16]
data, iTimes follows the self-supervised paradigm, and is tries to generate high-confident pseudo-labels of the weakly
proposed to learn the temporal structure by training ConvNets augmented unlabeled input and then use the pseudo-label to
to recognizing the temporal transformations applied to the time supervise the model for learning on the strongly-augmented
series used as input. More specifically, we first define a set version of the same input. For time-series data, dynamic time
of time sampling functions, and then apply these sampling warping distance [10] based methods are used as distance
functions to each time series in the dataset. The resulting metrics in K-NN classifier to measure the similarity between
transformations are then feed to a ConvNets to recognize the labeled and unlabeled time series, and a phase-consistency
transformations of each time series. It is the set time sampling regularization [7] is applied to forbearing time series. More
functions that define the classification pretext tasks that iTimes recently, SemiTime [9] takes a past-future relation prediction
has to learn. Finally, the underlying temporal structure pattern of time segments as the pretext task to train the model
of unlabeled time series can be captured by completing the on unlabeled time series. However, the past-future relation
pretext task in the self-supervised module. The feature spaces prediction tries to capture a coarse-grained temporal structure,
between labeled data and unlabeled data can be aligned by which results in a sub-optimal representation quality on time
jointly training the supervised and self-supervised modules series. Therefore, how to design an efficient self-supervised
which boost the ability of model learning and the repre- pretext task that captures the temporal structure of unlabeled
sentation quality. Experimental results on multiple real-world time series is still an open problem.
datasets of different sources show that iTimes consistently
outperforms the state-of-the-art baselines, which demonstrates B. Self-supervised Learning
the effectiveness of the proposed method.
Self-supervised learning aims to learn the underlying struc-
In summary, the main contributions of this paper are as
tural features within the data under the supervision of the self-
follows: (1) We propose a framework of semi-supervised time
generated labels from data, which has achieved great success
series classification by investigating irregular time sampling
in natural language processing and computer vision domains
(iTimes), which learns the underlying temporal structure of
by defining and solving various pretext tasks such as restoring
unlabeled time series in a self-supervised manner to benefit
aspect ratio distortion [17], rotation prediction [18] and visual
semi-supervised time series classification. (2) We introduce
contrastive learning [19] for image, and frame order validation
four different irregular time sampling methods to transform
[20], playback rate perception [21] and pace prediction [22]
the original time series into different transformations, based
for video. For audio/speech data, there are also different self-
on which the fine-grained temporal structure can be captured
supervised techniques such as multi-task learning from raw
by predicting the types of different transformations. (3) We
audio by predicting a number of handcrafted features such as
conduct extensive experiments on multiple real-world time-
MFCCs, prosody, and waveform [23], and contrastive learning
series datasets, and the results show that our method achieves
on audio/speech data [24]. In [25], a inter-intra relational
the state-of-the-art on semi-supervised time series classifica-
reasoning based method is proposed to learn the representation
tion task.
of time series by identifying short, medium, and long term
relationships between segments and predicting positive and
II. R ELATED W ORK negative inter-sample relationships jointly. More recently, in
A. Semi-supervised Learning [26], a self-supervised learning model based on the wavelet
In the past few years, semi-supervised learning has gained transform (WT) for scalogram-signal correspondence learning
massive research in the domain of machine learning, which is proposed. In [27], self-supervised representation learning
provides an effective way to leverage unlabeled data to im- using heart rate (HR) signals as self-supervised signals for the
prove model performance [9]. Existing semi-supervised learn- activity data is proposed. In [28], six different transformations
ing approaches can be divided into two categories: entropy applied to ECG signals are proposed and the recognition of
minimization based methods [12], [13] and consistency reg- signal transformation types is used as a pretext task.
ularization based methods [14]–[16]. Entropy minimization-
based methods assume that the decision boundary should not III. M ETHOD
cross the high-density region of marginal distribution, and In this section, we show the proposed iTimes in detail. As
therefore try to train a classifier that can make low-entropy shown in Fig. 2, iTimes consists of three modules includ-
predictions on unlabeled data. For example, Pseudo-Label ing irregular time sampling module, supervised classification
[13] generates pseudo-labels of unlabeled data to supervise module, and self-supervised transformation prediction module.
model training on unlabeled data. Consistency regularization- The inputs of iTimes are a set of labeled input-target pairs
based methods assume that the model should be invariant DL = {(ti , yi )|ti = (t(i,1) , ...t(i,T ) ), yi ∈ {1, 2, ..., C}}M
i=1 ,

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Harbin University of Science and Technology. Downloaded on October 29,2022 at 04:18:54 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2022.3199374
AUTHOR et al.: ITIMES: INVESTIGATING SEMI-SUPERVISED TIME SERIES CLASSIFICATION VIA IRREGULAR TIME SAMPLING 3

Fig. 2. The architecture of the proposed iTimes.

𝑔𝑔1 𝑔𝑔2 𝑔𝑔𝑙𝑙

where s/ (ti , α), s\ (ti , α) are sampling functions with lin-
𝒕𝒕𝑖𝑖
Original
Time Series
𝑡𝑡(𝑖𝑖,1) 𝑡𝑡(𝑖𝑖,2) ⋯ 𝑡𝑡(𝑖𝑖,𝑝𝑝1) � � ⋯ 𝑡𝑡(𝑖𝑖,𝑝𝑝2) ⋯ � � ⋯ 𝑡𝑡(𝑖𝑖,𝑝𝑝𝑙𝑙 ) early increasing and decreasing  time spans  gi , respec-
Irregular time sampling tively,
 and s ∨ (t i , α)
 = s\ (tp , α), s/ (t f , α) , s∧ (ti , α) =
𝒕𝒕′𝑖𝑖
s/ (tp , α), s\ (tf , α) are combinations of s/ and s\ , where
Transformation 𝑡𝑡(𝑖𝑖,𝑝𝑝1) 𝑡𝑡(𝑖𝑖,𝑝𝑝2) ⋯ 𝑡𝑡(𝑖𝑖,𝑝𝑝𝑙𝑙 ) tp and tf represent the past and future segments of the time
series respectively, and [•, •] is the concatenation operation.
Fig. 3. Schematic diagram of irregular sampling, where pj is the
sampling point, gj is the time span between pj−1 and pj . To obtain linearly increasing or linearly decreasing time
spans g = {g1 , g2 , · · · , gl }, one possible solution would
be to iterate through all possible  time span combinations
and a set of unlabeled T -length time series DU = {ti |ti = T −1
{g 1 , g 2 , · · · , g n } where n = under restriction
(t(i,1) , ...t(i,T ) )}N l−1
i=1 , where M ≪ N . Firstly, the original Pl +
time series is transformed into different transformations via i=1 gi = T , gi ∈ Z and find a time span such that Eq.
irregular time sampling. Then, iTimes conducts the supervised (1) obtains a maximum or minimum value which is used to
classification directly on the transformations of labeled data measure the linearity of g.
DL , and conducts the self-supervised transformation predic- Pl
tion on the transformations of unlabeled data DU . In iTimes, j=1 (gj − ḡ)(j − j̄)
ρ(g) = qP qP (1)
a weight sharing backbone encoder fθ is firstly employed l 2 l 2
j=1 (gj − ḡ) j=1 (j − j̄)
to extract the feature of transformation inputs, and following
which, a classification head hµ and a prediction head hφ are However, as the combination exploded, enumerating all pos-
used for supervised classification and prediction respectively. sible time spans is quite time-consuming, where the time
complexity is Θ((1 + ϵ)T −1 ) (ϵ is infinitesimal).
A. Irregular Time Sampling To address the abovementioned combination explosion
problem, we define the generation of time span as anPoptimiza-
In this section, we illustrate the irregular time sampling l
tion problem by using Eq.1 as objective function, i=1 gi =
function in detail, which refers to the sampling where the time +
T , gi ∈ Z as constraint. In this work, for s/ (ti , α) time
span between the sampling points is not fixed. In this work, we
sampling function, as shown in Eq. (2), we maximize Eq. (1).
use the function s• (ti , α) to represent irregular time sampling,
which takes the time series ti and the sampling rate α as max ρ(g)
inputs to generate a l-length transformed time series t′i , where l
l = ⌈T ·α⌉. As shown in Fig. 3, Irregular time sampling firstly
X (2)
s.t. gi = T, gi ∈ Z +
split the time series ti = (t(i,1) , ...t(i,T ) ) into l consecutive i=1
time segments {ti,1 , ti,2 , · · · ti,l }, with different time spans
Pl For s\ (ti , α) sampling function, as shown in Eq. (3), we
g i = {g1 , g2 , · · · gl } where j=1 gj = T . Then, the last
minimize Eq. (1).
timestamp t(i,pj ) of each segment ti,j is sampled to construct
the transformed time series t′i = (t(i,p1 ) , t(i,p2 ) , · · · , t(i,pl ) ), min ρ(g)
pj = pj−1 + gj , p0 = 0. l
Specifically, we define a set of irregular time sampling
X (3)
s.t. gi = T, gi ∈ Z +
functions S = {s/ (ti , α), s\ (ti , α), s∨ (ti , α), s∧ (ti , α)}, i=1

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Harbin University of Science and Technology. Downloaded on October 29,2022 at 04:18:54 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2022.3199374
4 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. XX, NO. XX, XXXX

Obviously, the number of solutions can be divide into 2 the sampled time series t′i,j to the backbone encoder fθ to get
cases, multiple solutions and unique solutions. For example, feature representation Zi,j = fθ (t′i,j ). Finally, the prediction
when l = 2, T = 5, all possible solutions will result in pi,j = hµ (Zi,j ) is made based on the obtained feature Zi,j .
a maximum or minimum value of ρ(g). If l = 4, T = 5 The supervised training loss Eq. (4) is defined as a cross-
maximizing ρ(g) or minimizing ρ(g) both have only one entropy loss:
solution, where ρ(g) obtains the maximum 0.774597 when the
|DL | |S|
solution g = {1, 1, 1, 2} and the minimum -0.774597 when 1 XX
g = {2, 1, 1, 1}. For the multi-solution case, we randomly Lsup = − yi · log(pi,j ) (4)
|DL | · |S| i=1 j=1
choose one of the solutions. After obtaining the time span g =
{g1 , g2 , · · · , gl } by maximizing or minimizing Eq. (1), the
sampling points {p1 , p2 , · · · , pl } are subsequently calculated
by pj = pj−1 + gj , p0 = 0, and finally the transformed time C. Training on Unlabeled Data
series t′i = (t(i,p1 ) , t(i,p2 ) , · · · , t(i,pl ) ) is obtained. To explore unlabeled data in temporal structure learning,
we use the self-generated time sampling function as super-
Algorithm 1 Irregular Time Sampling. vised learning signals and conduct transformation prediction
Require: tasks on unlabeled sampled data. Given unlabeled time series
ti : A T -length time series, α: Sampling rate datasets DU , set S of time sampling functions with α sampling
g: time spans rate. Each unlabeled time series ti will be sampled by the time
Ensure: sampling function sj ∈ S to obtain the sampled time series
t′i : Transformation of ti from irregular time sampling t′i,j . Input the sampled time series into the backbone encoder
1: function sampling(g) ▷ Generate sampling points fθ with shared weights to extract the feature embedding
2: p0 = 0 ▷ Starting sampling point Zi,j = fθ (t′i,j ), and then the obtained features are fed to the
3: for each gi in g do self-supervised transformation prediction head hϕ to get the
4: pi = pi−1 + gi ▷ Calculation of sampling points final prediction pi,j = hϕ (Zi,j ). Self-supervised training loss
5: end for Eq. (5) is defined as follow:
6: return (p1 , p2 , · · · , pl )
|DU | |S|
7: end function 1 XX
Lusp = − j · log(pi,j ) (5)
|DU | · |S| i=1 j=1
8: function s/ (ti , α)
9: l = ⌈T · α⌉
where j is the sampling function label. The model training
10: p = sampling(g) ▷ g obtained by Eq.2
algorithm is shown in Algorithm 2.
11: return (t(i,p1 ) , t(i,p2 ) , · · · t(i,pl ) )
12: end function
IV. E XPERIMENTS
13: function s\ (ti , α)
14: l = ⌈T · α⌉ In this section, we describe the experimental settings includ-
15: p = sampling(g) ▷ g obtained by Eq.3 ing datasets, baseline methods, and implementation. Then, we
16: return (t(i,p1 ) , t(i,p2 ) , · · · t(i,pl ) ) analysis the experimental results by comparing with the state-
17: end function of-the-art semi-supervised methods.

18: function s∨ (ti , α)

19: ti,p = (t(i,1) , · · · , ti,⌈T /2⌉ ) ▷ Get past segment A. Datasets
20: ti,f = (t(i,⌈T /2⌉+1) , · · · , t(i,T ) ) ▷ Get future segment
21: return s\ (ti,p , α), s/ (ti,f , α) To verify the effectiveness of the proposed method, in the
22: end function experiment, we use different categories datasets including four
public datasets SmallKitchenAppliances, CricketX, Phoneme
23: function s∧ (ti , α) and FordB from UCR Time Series Archive1 , as well as a
24: ti,p = (t(i,1) , · · · , ti,⌈T /2⌉ ) ▷ Get past segment real-world bearing dataset MFPT2 and a EEG dataset Epilep-
ticSeizure3 . All selected datasets have different number of
25: ti,f = (t(i,⌈T /2⌉+1) , · · · , t(i,T ) ) ▷ Get future segment
instances, signal lengths, and classes. In the experiment, the
26: return s/ (ti,p , α), s\ (ti,f , α)
27: end function proportion of training set, validation set and test set are set to
60%, 20%, 20% respectively. The details of the datasets are
shown in the Table I.
B. Training on Labeled Data
1 https://www.cs.ucr.edu/ eamonn/time_series_data_
Given labeled time series set DL , the time sampling function ˜
2018/
set S with α sampling rate, the backbone encoder fθ , and the 2 https://www.mfpt.org/fault-data-sets/
supervised classifier hµ . We first sample the input time series 3 https://archive.ics.uci.edu/ml/datasets/Epileptic+
ti to get the sampled time series t′i,j = sj (ti , α) and then fed Seizure+Recognition

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Harbin University of Science and Technology. Downloaded on October 29,2022 at 04:18:54 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2022.3199374
AUTHOR et al.: ITIMES: INVESTIGATING SEMI-SUPERVISED TIME SERIES CLASSIFICATION VIA IRREGULAR TIME SAMPLING 5

Algorithm 2 Model Training. pseudo-label and consistency regularization learning features.

Require: (7) SemiTime [9], is a semi-supervised learning model, which
Labeled data (ti , yi ) ∈ DL , and unlabeled input ti ∈ DU . learns the temporal relationship in time series by predicting the
Backbone encoder fθ ; Classification head hµ ; Transfor- past-future relationship between past and future segments.
mation prediction head hφ ; Learning rate η.
Sampling rate α, Sampling function set S.
1: for each epoch do
C. Implementation
2: for each labeled minibatch BL do The programming environment of iTimes is based on
3: t′i,j = sj (ti∈BL , α) ▷ Irregularly sampling labeled Python 3.7 and PyTorch. All the experiments are conducted
data on Unbuntu 18.04 with Intel Core i7-10700K, 64GB RAM,
′
4: Zi,j = fθ (ti,j ) ▷ Embedding of labeled inputs. and NVIDIA GeForce RTX 3080. ITimes uses the same
5: pi,j = hµ (Zi,j ) ▷ Label classification. backbone encoder fθ as the benchmark models, a simple 4-
P|B | P|S|
6: Lsup = − |BL1|·|S| i=1L j=1 i · log(pi,j )
y ▷ layer and 1-dimensional neural network using ReLu activation
Cross-entropy loss. function and batch normalization. The supervised learning
7: θ = θ − η∇θ Lsup , µ = µ − η∇µ Lsup ▷ Update classifier hµ and the self-supervised transformation prediction
models. head hϕ use one linear layer with 256 neurons and two
8: end for fully connected layers, respectively. The Adam optimizer with
9: for each unlabeled minibatch BU do learning rate 0.01 is used as the training optimizer. All models
′
10: ti,j = sj (ti∈BU , α) ▷ Irregularly sampling unlabeled were trained at 1000 iterations, 128 batch sizes, and an
data early stopping callback strategy with 200 patient iterations
11: Zi,j = fθ (t′i,j ) ▷ Embedding of unlabeled inputs. was used to detect validation metrics and to end training
12: pi,j = hφ (Zi,j ) ▷ sampling function classification. when no boost was observed. We use time warping, and
P|B | P|S| magnitude warping data augmentation [29] for all models. The
13: Lusp = − |BL1|·|S| i=1L j=1 j · log(pi,j ) ▷
Cross-entropy loss. effectiveness of the combination of irregular time sampling
14: θ = θ − η∇θ Lusp , φ = φ − η∇φ Lusp ▷ Update functions on the experimental results varies with the different
models. datasets. We selected the best combination for each datasets
15: end for through grid search and used classification accuracy as the
16: end for evaluation metric. We start the experiments by computing the
17: return Backbone encoder fθ and classification head hµ sampling points{p1 , p2 , · · · , pl } for all irregular time sampling
functions. The detailed architectural diagrams of iTimes are
shown in Table II.
TABLE I
S TATISTICS OF DATASETS . TABLE II
Dataset # Sample # Length # Class Category I MPLEMENTATION DETAIL OF I T IMES . H ERE , WE DENOTE 1D
SmallKitchenAppliances 750 720 3 Device CONVOLUTIONAL LAYER AS C ONV 1D( IN _ CHANNELS , OUT _ CHANNELS ,
CricketX 780 300 12 Motion KERNEL _ SIZE , STRIDE , PADDING ).
Phoneme 2110 1024 39 Sensor
MFPT 2574 1024 15 Machine Layer Description Output Tensor Dim.
FordB 3779 500 2 Sensor #0 Input time series (or time piece) 1 × T (or 1 × L)
EpilepticSeizure 11500 178 5 EEG Backbone Encoder
#1 Conv1D(1, 8, 4, 2, 1)+BatchNorm+ReLU 8 × T /2 (or 8 × L/2)
#2 Conv1D(8, 16, 4, 2, 1)+BatchNorm+ReLU 16 × T /4 (or 16 × L/4)
#3 Conv1D(16, 32, 4, 2, 1)+BatchNorm+ReLU 32 × T /8 (or 32 × L/8)
Conv1D(32, 64, 4, 2, 1)+BatchNorm+ReLU
B. Baseline Methods #4 64
+AvgPool1D+Flatten+Normalize
Classification Head
We compare the proposed iTimes against some baselines #1 Linear+BatchNorm+LeakyReLU 256
and several state-of-the-art semi-supervised methods: (1) Su- #2 Linear+Sigmoid C
Prediction Head
pervised, A supervised learning baseline method using a con- #1 Linear+BatchNorm+LeakyReLU 256
volutional neural network as encoder fθ and a linear layer as #2 Linear+Softmax 4
classifier. (2) Pseudo-Label [13], is a semi-supervised learning
model that expands the the supervised learning datasets by
generating pseudo-labels on unlabeled data. (3) Π-Model [14],
use self-ensembling to form a consensus prediction of the D. Experimental Results and Analysis
unknown labels using the outputs of the network-in-training 1) Ablation Study: In this section, we assess the effective-
on different epoch under different regularization and input ness and stability of iTimes by investigating the impact of
augmentation conditions. (4) MTL [8], is a semi-supervised different combinations of time sampling functions, sampling
learning model that leverages features learned from the self- rates, backbone encoders and embedding vector sizes on time
supervised task on unlabeled data. (5) MixMatch [15], is series classification.
a semi-supervised learning model that integrates consistency We first examine the effect of the combination of two time
regularization, entropy minimization, mixup in one. (6) Fix- sampling functions on the classification accuracy of the time
Match [16], is a semi-supervised learning model that leverages series with 10% labeled data under 80% sampling rate and 64

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Harbin University of Science and Technology. Downloaded on October 29,2022 at 04:18:54 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2022.3199374
6 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. XX, NO. XX, XXXX

75 24

st. Pred. ACC

st. Pred. ACC

0.00 0.00 50 0.00 81 74 92 48
74
s/

s/

s/
41

(Train)

(Train)
23 77 73 89

(Test)

(Test)
ACC

ACC
72.42 0.00 47.03 0.00 48 22.71 0.00 73 72 86 34
73
s\

s\

s\
22 69 71 83 27
72.70 73.35 0.00 72 49.56 44.11 0.00 46 23.37 20.48 0.00 65 70 80 20
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
s

s
71 21 Sampling Rate ( ) Sampling Rate ( )
72.69 70.36 74.67 0.00 50.67 46.86 47.76 0.00 44 23.30 23.77 23.13 0.00
s

70

s
s/ s\ s s s/ s\ s s s/ s\ s s 20
(a) SmallKitchenAppliances (b) CricketX
(a) SmallKitchenAppli- (b) CricketX (c) Phoneme
27

st. Pred. ACC

st. Pred. ACC
ances 84 24 91 70

(Train)
(Train)
87

(Test)
65

(Test)
78

ACC
ACC
72 21 83 60
75 94 80 66 18 79 55
0.00 0.00 0.00 60 15 75 50
78
s/

s/

s/
93 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
63.48 0.00 70 92.37 0.00 75.76 0.00 76 Sampling Rate ( ) Sampling Rate ( )
s\

s\

s\
92
72.80 62.80 0.00 65 92.44 92.91 0.00 77.36 74.31 0.00 74
(c) Phoneme (d) MFPT
s

91

s
62.33 61.11 67.87 0.00 92.35 91.90 92.60 0.00 76.46 74.46 76.42 0.00 72
s

60
s

90
s
s/ s\ s s s/ s\ s s s/ s\ s s 70 93

st. Pred. ACC

76

st. Pred. ACC

99 91 95
72

(Train)
(Train)
90

(Test)
(Test)
98

ACC
ACC
(d) MFPT (e) FordB (f) EpilepticSeizure 97 89 85 68
96 87 80 64
95 85 75 60
Fig. 4. Impact of different compositions of two time sampling functions 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Sampling Rate ( ) Sampling Rate ( )
with sampling rate (α) 80% on different datasets (10% labeled data).
(e) FordB (f) EpilepticSeizure

74
Class. ACC

Class. ACC

53 Fig. 6. Impact of different sampling rate(α) on different datasets (10%

73
(Test)

(Test)

72 50 labeled data).
71 47
70 44
s\, s , s s/, s , s s/, s\, s s/, s\, s s/, s\, s , s s\, s , s s/, s , s s/, s\, s s/, s\, s s/, s\, s , s 76
51
27
75
24

Class. ACC
47
Class. ACC

Class. ACC
74
73 21
(a) SmallKitchenAppliances (b) CricketX 72
43
39 18
71
70 35 15
8 64 128 256 512 8 64 128 256 512 8 64 128 256 512
28 68
Class. ACC

Class. ACC

26 66
(Test)

(Test)

(a) SmallKitchenAppli- (b) CricketX (c) Phoneme

24 64
22 62 ances
20 60
s\, s , s s/, s , s s/, s\, s s/, s\, s s/, s\, s , s s\, s , s s/, s , s s/, s\, s s/, s\, s s/, s\, s , s 76 94 78
72 93 77
Class. ACC

Class. ACC

Class. ACC
68 92 76
(c) Phoneme (d) MFPT 64 91 75
60 90 74
8 64 128 256 512 8 64 128 256 512 8 64 128 256 512
94 77
Class. ACC

Class. ACC

93 76
(Test)

(Test)

(d) MFPT (e) FordB (f) EpilepticSeizure

92 75
91 74
90 73 Fig. 7. Impact of different embedding size on different datasets (10%
s\, s , s s/, s , s s/, s\, s s/, s\, s s/, s\, s , s s\, s , s s/, s , s s/, s\, s s/, s\, s s/, s\, s , s labeled data).

(e) FordB (f) EpilepticSeizure

Fig. 5. Impact of different combinations of time sampling functions with and s/ that focuses on more information in the front may
sampling rate (α) 80% on different datasets (10% labeled data). not capture enough of the original information compared to a
uniform distribution of information in the time series.
In addition to examining the combination of two time
embedding size. As shown in Fig. 4, MFPT dataset and Epilep- sampling functions, we also examined the effect of the combi-
ticSeizure dataset obtains the best classification accuracy under nation of three time sampling functions and all time sampling
the combination of (s/ , s∨ ), CricketX, FordB, and SmallK- functions on time series classification. As shown in the Fig.
itchenAppliances dataset obtains the best classification under 5, compared with the combination of two time sampling func-
the combination of (s/ , s∧ ), (s\ , s∨ ), (s∨ , s∧ ) respectively, tions, the experimental results of three and all time sampling
and Phoneme dataset obtains the best classification accuracy function combinations do not have much variation, which
under the combination of (s\ , s∧ ) time sampling function. It is demonstrates the stability of iTimes.
evident that stable classification accuracies are achieved on the We next evaluate the effectiveness of the proposed irregular
MFPT and SmallKitchenAppliances datasets for all combina- time sampling module by investigating the effect of different
tions of two time sampling functions, more stable classification sampling rates α on time series classification with 10% labeled
accuracies are achieved on the two datasets containing the data, 64 embedding size under the optimal combination of time
largest number of samples for FordB and EpilepticSeizure, and sampling functions. As shown in Fig. 6, where the blue bar
some fluctuations in classification accuracies are observed on indicates transformation prediction accuracy on training data
the CricketX and Phoneme datasets. One possible reason for (St. Pred. ACC), and the line indicates classification accuracy
the fluctuations in classification accuracy in the two datasets on test data (ACC). As the sampling rate α increases, the
is that the information distribution of the time series is not classification accuracy keeps increasing, and after α increasing
suitable for the time sampling function. For example, the time to a certain value, the classification accuracy of all datasets
series may carry less information in the front than in the tail, except the MFPT dataset starts to decrease. We speculate that

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Harbin University of Science and Technology. Downloaded on October 29,2022 at 04:18:54 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2022.3199374
AUTHOR et al.: ITIMES: INVESTIGATING SEMI-SUPERVISED TIME SERIES CLASSIFICATION VIA IRREGULAR TIME SAMPLING 7

TABLE III the MFPT and EpilepticSeizure, respectively, and 13.71% and
I MPACT OF DIFFERENT ENCODERS ON I T IMES . 11.29% over Fixmatch, respectively.
- SimConv4 ResNet18 ResNet34 3) Time series Classification: In this section, we evaluate
CricketX 49.78±3.42 52.81±2.91 55.21±3.05 iTimes by comparing it with other semi-supervised based state-
of-the-arts models on time series classification task. Following
the previous research [8], [14], we randomly select 10%, 20%,
40%, and 100% of the training set as labeled data and use the
as the sampling rate increases, it will make each sampled time
entire training set as unlabeled data for model training. As
series carry more original information, which is beneficial for
shown in the Table IV, our proposed method consistently out-
the model to learn the temporal structure. As the sampling
performs all benchmark models on all datasets. For example,
rate continues to increase, the distinguishability of time series
given 10% labeled data, the accuracy is 4.90% higher than
sampled by different time sampling functions becomes the less
that of SemiTime and 6.68% higher than that of MixMatch
noticeable to the extent that self-supervised learning cannot
on CricketX, 7.86% higher than that of FixMatch and 8.78%
learn the temporal structure.
higher than that of MTL on EpilepticSeizure, 7.89% higher
We then evaluate the impact of different encoders, a sim-
than that of Π-Model on EpilepticSeizure. Given 20% labeled
ple four-layer CNN and two residual network, on iTimes
data, our method improves the accuracy over SemiTime by
via withn 10% labeled CricketX dataset. The experimental
4.26% on Phoneme, over Π-Model by 14.10% on Small-
results are shown Table III, ResNet18, ResNet34 improved
KitchenAppliances, and 2.39% on FordB, respectively. The
the classification accuracy by 6.09% and 10.90% respectively
results show that predicting unlabeled data by using self-
compared with SimConv4. Although ResNet obtained better
ensembling or generating pseudo-label alone does not capture
experimental results, ResNet took up more GPU memory than
the potential temporal structure of time series effectively,
SimConv4.
which is of great significance to the semi-supervised repre-
We finally evaluate the stability of iTimes by examining sentation learning of time series, coarse-grained predicting
different backbone encoder embedding vector sizes |Zi,j | with whether past, future time series segments belong to the same
10% labeled data under 80% sampling rate. As shown in time series, cannot fully explore the temporal relationship
Fig. 7, where the line indicates classification accuracy on test within time series. It can also be seen that the classification
data (Class. ACC). As the embedding size |Zi,j | increases, accuracy decreases slightly as the amount of labeled data
the classification accuracy first increases and then stabilizes decreases. But all exceed the supervised and other baselines,
within a small fluctuation. The time series classification ac- demonstrating the stability of the model. Moreover, we also
curacy is poor at low-dimensional embedding because the use 100% training set as labeled and unlabeled data to evaluate
embedding vector size is so tiny that the representation learned the performance of our proposed method for supervised and
by iTimes is insufficient to express the semantic informa- self-supervised training, and the experimental results demon-
tion of the original samples. The classification accuracy of strate that our method is consistently better than supervised
iTimes achieves a steady state when the embedding vector learning and self-supervised learning based MTL. This shows
size exceeds 64, reflecting the stability of iTimes. We also that the forecasting pretext task of MTL can not capture the
discovered that the EpilepticSeizure dataset with more than internal temporal structure of unlabeled time series effectively,
10,000 samples achieved the maximum classification accuracy while we designed transformation prediction is able to learn
at 64 dimensions of embedding vector size, while the MFPT the internal structure of time series.
data with around 2,000 samples achieved the maximum at 4) Visualization: To qualitatively analyze the learned tempo-
256 dimensions of embedding vector size, indicating that the ral structure, we extract the representation embedding vector of
embedding vector size is not correlated with the sample size. the time series using a trained backbone encoder and visualize
We conjecture that although the sample quantity of MFPT is it in 2D space using the t-SNE [30] in order to verify the
tiny, each sample carries more diverse information compared semantic consistency of the learned embeddings. Fig. 8 shows
to EpilepticSeizure, resulting in the 64-dimensional embedding the extracted representation embedding vectors by iTimes as
vector size being insufficient to express the semantic informa- well as the baseline models on EpilepticSeizure dataset. As
tion of the sample within MFPT. can be seen from the Fig. 8, class Eyes open bounded by
2) Applications: In this section, we examine the perfor- circle obtains a better clustering performance compared to the
mance of iTimes in practical applications on the MFPT and other baselines. Thus by fine-grained examining the temporal
EpilepticSeizure datasets. MFPT is a bearing fault dataset relationships within unlabeled time series, iTimes is able to
that comprises one data from a bearing test rig and three learn more useful embeddings and enables better clustering
real-world fault data, divided into 14 abnormal classes under ability of time series, maintaining more semantic consistency.
different loads and one normal class. And EpilepticSeizure is Interestingly, we also found that EEG recordings from healthy
a commonly used EEG data for seizure detection, it includes brain regions and tumor brain regions were not clustered very
11,500 instances with 5 classes(Eyes open, Eyes closed, Tu- well, despite the better performance of our proposed model,
mor, Healthy, Seizure). The experimental results are shown in but this finding provides additional insights for data inspection
the Table IV. The classification accuracy of iTimes exceeds and model optimization.
that of all baseline methods. Given 10% labeled data, iTimes
improves 36.43% and 13.28% over Supervised method on

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Harbin University of Science and Technology. Downloaded on October 29,2022 at 04:18:54 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2022.3199374
8 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. XX, NO. XX, XXXX

TABLE IV
T EST CLASSIFICATION ACCURACY (%, AVERAGES OF 5 SEEDS AND 5 RUNS ) FOR SUPERVISED BASELINE AND SEMI - SUPERVISED LEARNING ON
DIFFERENT DATASETS . A LL METHODS USE THE SAME 4- LAYER CONVOLUTIONAL BACKBONE .
Label Ratio 10% 20% 40% 100% 10% 20% 40% 100% 10% 20% 40% 100%
Dataset SmallKitchenAppliances CricketX Phoneme
Supervised 63.11±3.50 63.11±2.30 68.23±3.51 73.03±2.65 33.62±0.95 38.97±2.08 52.64±2.53 62.98±2.01 19.54±0.69 22.45±1.14 27.82±1.62 35.90±1.4
Pseudo-Label [13] 52.46±4.09 62.29±2.96 73.04±3.75 - 38.87±2.26 44.44±2.91 53.39±2.18 - 18.92±0.72 21.40±0.99 26.34±0.68 -
Π-Model [14] 57.04±1.86 60.93±2.82 65.44±1.85 - 38.61±2.29 48.18±2.07 54.73±1.04 - 20.71±0.65 22.21±1.71 25.74±1.78 -
MTL [8] 65.03±2.71 66.69±3.03 71.23±2.19 75.30±3.24 40.94±1.97 50.12±1.22 55.10±1.12 63.58±1.72 19.98±1.67 20.00±0.66 25.58±1.17 32.34±0.78
MixMatch [15] 55.57±2.58 55.55±1.59 59.76±1.35 - 43.10±2.34 56.80±3.16 64.46±1.66 - 23.09±0.63 26.89±1.02 31.53±0.40 -
FixMatch [16] 65.14±3.36 65.60±4.23 74.10±2.48 - 38.33±2.07 50.89±2.04 60.66±1.27 - 18.89±1.80 21.58±1.57 27.72±1.0 -
SemiTime [9] 69.35±2.08 70.55±2.06 71.57±2.10 74.58±2.67 44.88±3.13 51.61±0.66 58.71±2.78 65.66±1.58 20.64±1.34 25.28±1.42 30.11±1.19 36.96±1.63
iTimes 74.13±4.91 75.03±4.94 77.38±2.41 81.23±1.28 49.78±3.42 58.64±2.59 68.37±2.75 77.71±1.40 24.23±1.53 29.54±2.12 36.54±0.42 45.12±1.2
Dataset MFPT EpilepticSeizure FordB
Label Ratio 10% 20% 40% 100% 10% 20% 40% 100% 10% 20% 40% 100%
Supervised 50.88±0.32 57.14±0.54 69.76±0.48 81.63±0.15 68.40±0.43 70.77±0.70 73.49±0.60 77.77±1.13 89.69±0.65 90.48±1.08 91.80±0.47 92.60±0.90
Pseudo-Label [13] 63.90±2.62 65.39±1.70 69.60±2.27 - 68.57±0.50 72.92±0.48 74.60±0.65 - 89.02±0.77 90.58±0.34 91.71±0.37 -
Π-Model [14] 55.41±0.65 59.68±0.43 70.15±1.01 - 69.60±0.34 71.58±0.64 74.54±0.55 - 88.94±0.74 90.64±0.37 91.02±0.14 -
MTL [8] 40.84±1.97 50.12±1.22 55.10±1.12 63.58±1.72 68.71±0.94 73.17±0.81 74.77±0.75 78.53±0.62 90.43±0.88 91.21±1.10 91.84±0.77 92.55±0.96
MixMatch [15] 52.56±1.01 68.62±0.80 81.87±1.60 - 67.99±0.09 69.67±0.30 72.22±0.63 - 90.75±0.44 91.34±0.39 91.72±0.58 -
FixMatch [16] 61.05±1.65 71.72±1.76 81.04±0.8 - 69.63±0.67 71.37±0.24 72.30±0.76 - 90.31±0.48 90.54±0.19 91.76±0.20 -
SemiTime [9] 64.16±0.85 69.84±0.94 76.49±0.54 84.33±0.50 74.86±0.42 75.54±0.63 77.01±0.79 79.26±1.02 90.97±0.88 91.46±0.68 91.80±0.48 92.40±0.74
iTimes 69.42±1.23 78.84±1.87 85.78±1.45 92.03±0.45 77.49 ± 0.49 78.93 ± 0.54 82.23 ± 0.56 82.87 ± 0.56 92.08±1.06 93.03±0.58 93.35±0.57 94.29±0.47

Seizure Tumor area Healthy area Eyes closed Eyes open

[2] R. Rodrigues and P. Couto, “Semi-supervised learning for ecg clas-
sification,” in 2021 Computing in Cardiology (CinC), vol. 48, DOI
10.23919/CinC53138.2021.9662693, pp. 1–4, 2021.
[3] Q. Zhu, Z. Chen, and Y. C. Soh, “A novel semisupervised deep learning
method for human activity recognition,” IEEE Transactions on Industrial
Informatics, vol. 15, DOI 10.1109/TII.2018.2889315, no. 7, pp. 3821–
3830, 2019.
Supervised Pseudo-Label П-Model MTL [4] X. Zhao, M. Jia, and Z. Liu, “Semisupervised graph convolution deep
belief network for fault diagnosis of electormechanical system with
limited labeled data,” IEEE Transactions on Industrial Informatics,
vol. 17, DOI 10.1109/TII.2020.3034189, no. 8, pp. 5450–5460, 2021.
[5] Y. Yang, J. Zhong, W. Li, T. A. Gulliver, and S. Li, “Semisupervised
multilabel deep learning based nonintrusive load monitoring in smart
grids,” IEEE Transactions on Industrial Informatics, vol. 16, DOI
10.1109/TII.2019.2955470, no. 11, pp. 6892–6902, 2020.
MixMatch FixMatch SemiTime Ours
[6] T. Ko and H. Kim, “Fault classification in high-dimensional com-
plex processes using semi-supervised deep convolutional generative
models,” IEEE Transactions on Industrial Informatics, vol. 16, DOI
Fig. 8. t-SNE visualization of the temporal structure on EpilepticSeizure. 10.1109/TII.2019.2941486, no. 4, pp. 2868–2877, 2020.
[7] J. Yi and J. Park, “Semi-supervised bearing fault diagnosis with
adversarially-trained phase-consistent network,” in Proceedings of the
27th ACM SIGKDD Conference on Knowledge Discovery & Data
V. C ONCLUSION Mining, pp. 3875–3885, 2021.
[8] S. Jawed, J. Grabocka, and L. Schmidt-Thieme, “Self-supervised learn-
In this paper, we propose a semi-supervised time series ing for semi-supervised time series classification,” in Pacific-Asia
classification framework to explore the underlying temporal Conference on Knowledge Discovery and Data Mining, pp. 499–511.
Springer, 2020.
structure within unlabeled time series in a self-supervised man- [9] H. Fan, F. Zhang, R. Wang, X. Huang, and Z. Li, “Semi-supervised
ner. We design a simple but efficient irregular time sampling time series classification by temporal relation prediction,” in ICASSP
strategy and extract the underlying temporal structure features 2021-2021 IEEE International Conference on Acoustics, Speech and
Signal Processing (ICASSP), pp. 3545–3549. IEEE, 2021.
inside the time series by predicting the transformation type
[10] Y. Chen, B. Hu, E. Keogh, and G. E. Batista, “Dtw-d: time series semi-
of irregular time sampling. Our main finding is that the ex- supervised learning from a single example,” in Proceedings of the 19th
amination of fine-grained temporal relationships of unlabeled ACM SIGKDD international conference on Knowledge discovery and
time series is more conducive to the utilize of unlabeled data mining, pp. 383–391, 2013.
[11] L. de Carvalho Pagliosa and R. F. de Mello, “Semi-supervised time series
time series for semi-supervised learning. While conducting classification on positive and unlabeled problems using cross-recurrence
this study, we expected to find a perfect combination of quantification analysis,” Pattern Recognition, vol. 80, pp. 53–63, 2018.
time sampling function to examine the temporal structure. For [12] Y. Grandvalet and Y. Bengio, “Semi-supervised learning by entropy
minimization,” in Advances in Neural Information Processing Systems,
the sake of simplicity of implementation, we only selected pp. 529–536, 2005.
s/ , s\ , s∨ , s∧ from the infinite time sampling functions, which [13] D.-H. Lee et al., “Pseudo-label: The simple and efficient semi-supervised
will undoubtedly limit the feature learning capability of the learning method for deep neural networks,” in Workshop on challenges
in representation learning, ICML, vol. 3, no. 2, 2013.
ConvNets. We intend to explore more sampling transformation [14] S. Laine and T. Aila, “Temporal ensembling for semi-supervised learn-
methods in the future to find the most efficient transformation. ing,” in 5th International Conference on Learning Representations,
ICLR, 2017.
[15] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and
R EFERENCES C. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,”
arXiv preprint arXiv:1905.02249, 2019.
[1] Y. Peng, F. Jin, W. Kong, F. Nie, B.-L. Lu, and A. Cichocki, [16] K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk,
“Ogssl: A semi-supervised classification model coupled with opti- A. Kurakin, H. Zhang, and C. Raffel, “Fixmatch: Simplifying semi-
mal graph learning for eeg emotion recognition,” IEEE Transactions supervised learning with consistency and confidence,” arXiv preprint
on Neural Systems and Rehabilitation Engineering, vol. 30, DOI arXiv:2001.07685, 2020.
10.1109/TNSRE.2022.3175464, pp. 1288–1297, 2022. [17] R. Sakurai, S. Yamane, and J.-H. Lee, “Restoring aspect ratio distortion

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Harbin University of Science and Technology. Downloaded on October 29,2022 at 04:18:54 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2022.3199374
AUTHOR et al.: ITIMES: INVESTIGATING SEMI-SUPERVISED TIME SERIES CLASSIFICATION VIA IRREGULAR TIME SAMPLING 9

of natural images with convolutional neural network,” IEEE Transactions Xuxin Liu received the B.E. degree in Packag-
on Industrial Informatics, vol. 15, DOI 10.1109/TII.2018.2803041, no. 1, ing Engineeering from the Harbin University of
pp. 563–571, 2019. Commerce, Harbin, China, in 2020. He is cur-
[18] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation rently pursuing the M.S. degree in Computer Sci-
learning by predicting image rotations,” in 6th International Conference ence and Technology with the Harbin University
on Learning Representations, ICLR, 2018. of Science and Technology, Harbin, China. His
[19] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework current research interests include time series
for contrastive learning of visual representations,” in Proceedings of the analysis, network security, and machine learn-
37th international conference on machine learning (ICML), 2020. ing.
[20] D. Wei, J. J. Lim, A. Zisserman, and W. T. Freeman, “Learning and using
the arrow of time,” in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, CVPR, pp. 8052–8060, 2018.
[21] Y. Yao, C. Liu, D. Luo, Y. Zhou, and Q. Ye, “Video playback rate
perception for self-supervised spatio-temporal representation learning,”
in Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), Jun. 2020. Fengbin Zhang received the Ph.D. degree in
[22] J. Wang, J. Jiao, and Y.-H. Liu, “Self-supervised video representation Computer Application from Harbin Engineering
learning by pace prediction,” in European conference on computer University, China in 2005. He is currently a su-
vision, ECCV, 2020. pervisor and Professor at the Harbin University
[23] S. Pascual, M. Ravanelli, J. Serrà, A. Bonafonte, and Y. Ben- of Science and Technology. His current research
gio, “Learning problem-agnostic speech representations from multi- focuses on network and information security,
ple self-supervised tasks,” in Proc. of the Conf. of the Int. Speech firewall technology and intrusion detection tech-
Communication Association (INTERSPEECH), pp. 161–165, 2019. nology.
[24] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with
contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
[25] H. Fan, F. Zhang, and Y. Gao, “Self-supervised time series repre-
sentation learning by inter-intra relational reasoning,” arXiv preprint Han Liu Han Liu received the M.S. degree
arXiv:2011.13548, 2020. in Computer Science and Technology from the
[26] A. Saeed, F. D. Salim, T. Ozcelebi, and J. Lukkien, “Federated Harbin University of Science and Technology,
self-supervised learning of multisensor representations for embed- Harbin, China, in 2021. He is currently studying
ded intelligence,” IEEE Internet of Things Journal, vol. 8, DOI for a Ph.D. degree at Harbin University of Sci-
10.1109/JIOT.2020.3009358, no. 2, pp. 1030–1040, 2021. ence and Technology, Harbin, China. His current
[27] D. Spathis, I. Perez-Pozuelo, S. Brage, N. J. Wareham, and C. Mascolo, research interests include artificial intelligence,
“Self-supervised transfer learning of physiological representations from network security, and machine learning.
free-living wearable data,” in Proceedings of the Conference on Health,
Inference, and Learning, pp. 69–78, 2021.
[28] P. Sarkar and A. Etemad, “Self-supervised ecg representation learning
for emotion recognition,” IEEE Transactions on Affective Computing,
DOI 10.1109/TAFFC.2020.3014842, pp. 1–1, 2020.
[29] T. T. Um, F. M. Pfister, D. Pichler, S. Endo, M. Lang, S. Hirche,
U. Fietzek, and D. Kulić, “Data augmentation of wearable sensor data for
parkinsonâs disease monitoring using convolutional neural networks,” in Haoyi Fan is an associate research fellow with
Proceedings of the 19th ACM International Conference on Multimodal the School of Computer and Artificial Intelli-
Interaction, pp. 216–220, 2017. gence, Zhengzhou University, Zhengzhou. He
[30] P. G. Poličar, M. Stražar, and B. Zupan, “opentsne: a modular python received the Ph.D. degree in the School of Com-
library for t-sne dimensionality reduction and embedding,” BioRxiv, p. puter Science and Technology, Harbin University
731877, 2019. of Science and Technology in 2021. His current
research interests include time series analysis,
data mining, and deep learning.

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Harbin University of Science and Technology. Downloaded on October 29,2022 at 04:18:54 UTC from IEEE Xplore. Restrictions apply.