3.

Deep Learning Methods

Badan Meteorologi, Klimatologi, dan Geofisika
Juli – Agustus 2022
 Deep Learning Methods
Deep Learning Methods
Time Series Problems
MLP CNN LSTM
Univariate Model X X X
Multiple Input Series X X X
Multivariate
Multiple Parallel Series X X X
Multi-step Vector Output Model X X X
Multiple Input Multi-step Output X X X
Multivariate
Multi-step Multiple Parallel Input and Multi-
X X X
step Output

Workshop on Deep Learning Time Series by [email protected] July 2022 271

Prepare Time Series Data for
MLPs, CNNs and LSTMs
Badan Meteorologi, Klimatologi, dan Geofisika
Juli – Agustus 2022
 Time Series Data In Supervised Learning Model
◦ Time series data must be transformed before it can be used to fit a supervised learning model
◦ The data can be used immediately to fit a supervised machine learning algorithm and a Multilayer Perceptron neural
network
◦ One further transformation is required in order to ready the data for fitting a Convolutional Neural Network (CNN) or Long
Short-Term Memory (LSTM) Neural Network
◦ The two-dimensional structure of the supervised learning data must be transformed to a three-dimensional structure

◦ How to transform a time series dataset into a two-dimensional supervised learning format
◦ How to transform a two-dimensional time series dataset into a three-dimensional structure suitable for CNNs and
LSTMs
◦ How to step through a worked example of splitting a very long time series into subsequences ready for training a
CNN or LSTM model

Workshop on Deep Learning Time Series by [email protected] July 2022 273

 Time Series to Supervised
◦ Time series data requires preparation before it can be used to train a supervised learning model, such as an
LSTM neural network

◦ Example of a univariate time series represented as a vector of observations

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Workshop on Deep Learning Time Series by [email protected] July 2022 274

 Time Series to Supervised
◦ A supervised learning algorithm requires that data is provided as a collection of samples
◦ Each sample has an input component (X) and an output component (y)
X, y
sample input, sample output
sample input, sample output
sample input, sample output
...

◦ The model will learn how to map inputs to outputs from the provided examples

Y = f (X )

Workshop on Deep Learning Time Series by [email protected] July 2022 275

 Time Series to Supervised
◦ A time series must be transformed into samples with input and output components
◦ The transform both informs what the model will learn
◦ How intend to use the model in the future when making predictions
◦ What is required to make a prediction (X) and what prediction is made (y)

◦ For a univariate time series problem where we are interested in one-step predictions
◦ The observations at prior time steps, so-called lag observations, are used as input
◦ The output is the observation at the current time step

Workshop on Deep Learning Time Series by [email protected] July 2022 276

 Time Series to Supervised
◦ The 10-step univariate series can be expressed as a supervised learning problem
◦ Three time steps for input
◦ One step as output
X, y
[1, 2, 3], [4]
[2, 3, 4], [5]
[3, 4, 5], [6]
...

Workshop on Deep Learning Time Series by [email protected] July 2022 277

 Time Series to Supervised
◦ The split_sequence() function split a given univariate sequence into multiple samples
◦ Each sample has a specified number of time steps
◦ The output is a single time step
from numpy import array

# split a univariate sequence into samples

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 278

 Time Series to Supervised
◦ After transformed the data into a form suitable for training a supervised learning model
◦ It will be represented as rows and columns
◦ Each column will represent a feature to the model and may correspond to a separate lag observation
◦ Each row will represent a sample and will correspond to a new example with input and output components

Feature: A column in a dataset, such as a lag

observation for a time series dataset

Sample: A row in a dataset, such as an input

and output sequence for a time series
dataset

Workshop on Deep Learning Time Series by [email protected] July 2022 279

 Time Series to Supervised
◦ A univariate time series in terms of rows and columns
# define univariate time series
series = array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
print(series.shape)
# transform to a supervised learning problem
X, y = split_sequence(series, 3)
print(X.shape, y.shape)
# show each sample The dataset will be represented in Python using a NumPy array
for i in range(len(X)):
print(X[i], y[i])

(10,)
The length of each dimension is referred to as the shape of the array
(7, 3) (7,)
[1 2 3] 4
[2 3 4] 5
[3 4 5] 6
[4 5 6] 7
[5 6 7] 8
[6 7 8] 9
[7 8 9] 10
The array will have two dimensions

Workshop on Deep Learning Time Series by [email protected] July 2022 280

 Time Series to Supervised
(10,)
(7, 3) (7,)
[1 2 3] 4
[2 3 4] 5
[3 4 5] 6
[4 5 6] 7
[5 6 7] 8
[6 7 8] 9
[7 8 9] 10

X y

◦ Data in this form can be used directly to train a simple neural network, such as a Multilayer Perceptron
◦ The difficulty when trying to prepare this data for CNNs and LSTMs
◦ Require data to have a three-dimensional structure

Workshop on Deep Learning Time Series by [email protected] July 2022 281

 3D Data Preparation Basics
◦ Preparing time series data for CNNs and LSTMs requires one additional step beyond transforming the data into a
supervised learning problem
◦ The input layer for CNN and LSTM models is specified by the input shape argument on the first hidden layer of the network
◦ The first layer defined in the model be the input layer
◦ The LSTM() layer must specify the shape of the input data

◦ The input of CNN and LSTM layer must be three-dimensional

Samples Time Steps Features

• One sequence is one sample • One time step is one point of • One feature is one
• A batch is comprised of one observation in the sample observation at a time step
or more samples • One sample is comprised of • One time step is comprised
multiple time steps of one or more features

Workshop on Deep Learning Time Series by [email protected] July 2022 282

 3D Data Preparation Basics
◦ Three-dimensional structure of input data is often summarized using the array shape notation
[samples, timesteps, features]
◦ Adding the dimension of features

◦ The two-dimensional shape of a dataset has the array shape of

[samples, features]

◦ In time series forecasting problems

◦ The features are observations at time steps
◦ A univariate time series has only one feature

Workshop on Deep Learning Time Series by [email protected] July 2022 283

 3D Data Preparation Basics
◦ Defining the input layer of your LSTM network
◦ The network assumes have one or more samples and requires that specify the number of time steps and the number of
features
◦ Specifying a tuple to the input shape argument
# lstm with an input layer
model = Sequential() Defines an input layer that expects 1 or more samples, 3 time
model.add(LSTM(32, input_shape=(3, 1))) steps, and 1 feature
model.add(Dense(1))

◦ The first layer in the network is actually the first hidden layer
◦ The 32 refers to the number of units in the first hidden layer

◦ The number of units in the first hidden layer is completely unrelated to the number of samples, time steps or
features in the input data

Workshop on Deep Learning Time Series by [email protected] July 2022 284

 3D Data Preparation Basics
◦ Maps onto the univariate time series from the data with 3 input time steps and 1 feature
◦ Loaded the time series dataset from CSV or transformed it to a supervised learning problem in memory
◦ It will have a two-dimensional shape
◦ Convert it to a three-dimensional shape with some number of samples, 3 time steps per sample and 1 feature per time
step, or [?, 3, 1]
◦ Do this by using the reshape() NumPy function

◦ If have 7 samples and 3 time steps per sample for the input element of our time series
◦ Reshape it into [7, 3, 1] by providing a tuple to the reshape() function specifying the desired new shape of (7, 3, 1)
◦ The array must have enough data to support the new shape, which in this case it does as [7, 3] and [7, 3, 1] are functionally
the same thing
# transform input from [samples, features] to [samples, timesteps, features]
X = X.reshape((7, 3, 1))

Workshop on Deep Learning Time Series by [email protected] July 2022 285

 3D Data Preparation Basics
◦ A short-cut in reshaping the array is to use the known shapes
◦ The number of samples and the number of times steps from the array returned from the call to the X.shape property of
the array

The number of rows in a

X.shape[0] The number of samples
2D array

The number of
The number of The number of
X.shape[1] columns in a 2D
feature time steps
array

# transform input from [samples, features] to [samples, timesteps, features]

X = X.reshape((X.shape[0], X.shape[1], 1))

Workshop on Deep Learning Time Series by [email protected] July 2022 286

 3D Data Preparation Basics
◦ Transforming a univariate time series into a three-dimensional array
# transform univariate 2d to 3d
from numpy import array
# split a univariate sequence into samples
def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

# define univariate time series The univariate time series with 10 time steps
series = array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
print(series.shape)
# transform to a supervised learning problem
X, y = split_sequence(series, 3)
print(X.shape, y.shape)
# transform input from [samples, features] to [samples, timesteps, features]
X = X.reshape((X.shape[0], X.shape[1], 1))
print(X.shape) The shape if the input (X) and output (y) elements of each sample after converted into
a supervised learning problem
Workshop on Deep Learning Time Series by [email protected] July 2022 287
 3D Data Preparation Basics
◦ The input element of each sample is reshaped to be three-dimensional suitable for fitting an LSTM or CNN

(10,)
(7, 3) (7,)
(7, 3, 1)

The shape [7, 3, 1]: 7 samples, 3 time steps, 1 feature

Workshop on Deep Learning Time Series by [email protected] July 2022 288

 Data Preparation Example
◦ Consider in the situation:
◦ Have two columns in my data file with 3,650 rows
◦ Column 1 is date (with 1 date interval)
◦ Column 2 is the monitored temperature in Depok City
◦ Trying to forecast the temperature of Depok for future time steps
◦ Set the number of samples, time steps and features in this data for an LSTM?

Workshop on Deep Learning Time Series by [email protected] July 2022 289

 Data Preparation Example
◦ There are few problems here:
◦ Data Shape. LSTMs expect 3D input, and it can be challenging to get the head around this the first time
◦ Sequence Length. LSTMs don't like sequences of more than 200-400 time steps, so the data will need to be split into
subsamples

◦ Croken down into the following 4 steps:

1. Load the 2. Drop the 3. Split Into 4. Reshape

Data Time Column Samples Subsequences

Workshop on Deep Learning Time Series by [email protected] July 2022 290

 1. Load the Data
◦ Load this dataset as a Pandas Series using the function read_csv() as a Pandas DataFrame
# load dataset using read_csv()
from pandas import read_csv
from pandas import DataFrame
series = read_csv('daily-minimum-temperatures.csv',
header=0, index_col=0, parse_dates=True, squeeze=True)

dataframe = DataFrame() Split data into date and temperature

dataframe['date'] = [series.index[i] for i in range(len(series))]
dataframe['temperature'] = [series[i] for i in range(len(series))]

◦ Running this piece both prints the first 5 rows of data

print(dataframe.head(5))
print(dataframe.shape)

Workshop on Deep Learning Time Series by [email protected] July 2022 291

 1. Load the Data
◦ Running this piece both prints the first 5 rows of data
date temperature
0 1981-01-01 20.7
1 1981-01-02 17.9
2 1981-01-03 18.8
3 1981-01-04 14.6
4 1981-01-05 15.8
(3650, 2)

The shape of the loaded data

Have 3,650 rows and 2 columns: a standard univariate time series dataset

Workshop on Deep Learning Time Series by [email protected] July 2022 292

 1. Load the Data
◦ Loading by defining a new dataset in memory with 5,000 time steps
# example of defining a dataset
from numpy import array

# define the dataset

data = list()
n = 5000
for i in range(n):
data.append([i+1, (i+1)*10])

data = array(data)
print(data[:5, :])
print(data.shape)

Workshop on Deep Learning Time Series by [email protected] July 2022 293

 1. Load the Data
◦ Running this piece both prints the first 5 rows of data and the shape of the loaded data
[[ 1 10]
[ 2 20]
[ 3 30]
[ 4 40]
[ 5 50]]
(5000, 2)

A standard univariate time series dataset: 5,000 rows and 2 columns

Workshop on Deep Learning Time Series by [email protected] July 2022 294

 2. Drop the Time Column
◦ If the time series data is uniform over time and there is no missing values
◦ Drop the time column

◦ If not, may want to look at imputing the missing values, resampling the data to a new time scale, or developing a
model that can handle missing values
# example of dropping the time dimension from the dataset
data = array(dataframe)
# drop date Drop the first column
data = data[:, 1]
print(data.shape)

◦ Running the example prints the shape of the dataset after the time column has been removed
(3650,)

Workshop on Deep Learning Time Series by [email protected] July 2022 295

 3. Split Into Samples
◦ LSTMs need to process samples where each sample is a single sequence of observations
◦ Need to split the 3,650 time steps into multiple shorter sub-sequences

◦ There are many ways to do this, and may want to explore some depending on the problem
◦ Need overlapping sequences
◦ Non-overlapping is good but the model needs state across the sub-sequences

Workshop on Deep Learning Time Series by [email protected] July 2022 296

 3. Split Into Samples
◦ Split the 3,650 time steps into 73 sub-sequences of 50 time steps each
◦ Using the old fashioned way rather than using NumPy or Python tricks

# example of splitting a univariate sequence into subsequences

# split into samples (e.g. 3650/50 = 73)
n = len(data)
samples = list()
length = 50
# step over the 3,650 in jumps of 50
for i in range(0,n,length):
# grab from i to i + 50
sample = data[i:i+length]
samples.append(sample)
print(len(samples))

Workshop on Deep Learning Time Series by [email protected] July 2022 297

 4. Reshape Subsequences
◦ The LSTM needs data with the format of [samples, timesteps, features]
◦ There are 73 samples, 50 time steps per sample, and 1 feature
◦ First, need to convert the list of arrays into a 2D NumPy array with the shape [73, 50]
# example of creating an array of subsequence
# convert list of arrays into 2d array
data = array(samples)
print(data.shape)

◦ Running this piece, have 73 rows and 50 columns

◦ Interpreted in a machine learning context
◦ This dataset has 73 samples and 50 features per sample
(73, 50)

Workshop on Deep Learning Time Series by [email protected] July 2022 298

 4. Reshape Subsequences
◦ Use the reshape() function to add one additional dimension for the single feature
◦ Use the existing columns as time steps instead
# example of creating a 3d array of subsequences
# reshape into [samples, timesteps, features]
data = data.reshape((len(samples), length, 1))
print(data.shape)

◦ The data can now be used as an input (X) to an LSTM model, or even a CNN model
(73, 50, 1)

Workshop on Deep Learning Time Series by [email protected] July 2022 299

 Develop MLPs for
Time Series Forecasting
Badan Meteorologi, Klimatologi, dan Geofisika
Juli – Agustus 2022
 MLPs for Time Series Forecasting
◦ MLPs can be applied to time series forecasting
◦ A challenge with using MLPs for time series forecasting is in the preparation of the data
◦ Lag observations must be attened into feature vectors

◦ How to develop a suite of MLP models for a range of standard time series forecasting problems
◦ How to develop MLP models for univariate time series forecasting
◦ How to develop MLP models for multivariate time series forecasting
◦ How to develop MLP models for multi-step time series forecasting

Workshop on Deep Learning Time Series by [email protected] July 2022 301

 MLPs for Time Series Forecasting
Univariate MLP Models
• Data Preparation
• MLP Model
Multivariate MLP Models
• Multiple Input Series
• MLP Model
• Multi-headed MLP Model
• Multiple Parallel Series
• Vector-Output MLP Model
• Multi-output MLP Model
Multi-step MLP Models
• Data Preparation
• Vector Output Model
Multivariate Multi-step MLP Models
• Multiple Input Multi-step Output
• Multiple Parallel Input and Multi-step Output

Workshop on Deep Learning Time Series by [email protected] July 2022 302

 Univariate MLP Models
◦ MLPs can be used to model univariate time series forecasting problems
◦ Univariate time series are a dataset comprised
◦ A single series of observations with a temporal ordering
◦ A model is required to learn from the series of past observations to predict the next value in the sequence

Workshop on Deep Learning Time Series by [email protected] July 2022 303

Data Preparation
Develop MLPs for Time Series Forecasting
 Univariate MLP Models
Data Preparation
◦ The MLP model will learn a function that maps a sequence of past observations as input to an output observation
◦ The sequence of observations must be transformed into multiple examples from which the model can learn

◦ A given univariate sequence:

[10, 20, 30, 40, 50, 60, 70, 80, 90]

◦ Divide the sequence into multiple input/output patterns called samples
◦ Three time steps are used as input
◦ One time step is used as output for the one-step prediction that is being learned

X, y
10, 20, 30, 40
20, 30, 40, 50
30, 40, 50, 60
...

Workshop on Deep Learning Time Series by [email protected] July 2022 305

 Univariate MLP Models
Data Preparation
◦ The split_sequence() function split a given univariate sequence into multiple samples
◦ Each sample has a specified number of time steps and the output is a single time step
# univariate data preparation
from numpy import array
# split a univariate sequence into samples
def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 306

 Univariate MLP Models 1/2
Data Preparation
◦ Transforming a univariate time series into a supervised learning problem
# univariate data preparation
from numpy import array

# split a univariate sequence into samples

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 307

 Univariate MLP Models 2/2
Data Preparation
◦ Transforming a univariate time series into a supervised learning problem
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps = 3

# split into samples

X, y = split_sequence(raw_seq, n_steps)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 308

 Univariate MLP Models
Data Preparation
◦ Running the example splits the univariate series into six samples
[10 20 30] 40
[20 30 40] 50
[30 40 50] 60
[40 50 60] 70
[50 60 70] 80
[60 70 80] 90

y: One output time step

X: Three input time steps

Workshop on Deep Learning Time Series by [email protected] July 2022 309

Univariate MLP Models
Develop MLPs for Time Series Forecasting
 Univariate MLP Models
MLP Model
◦ A simple MLP model
◦ A single hidden layer of nodes
◦ An output layer used to make a prediction

◦ Define an MLP for univariate time series forecasting

# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_steps))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

◦ The number of time steps as input is the number of split_sequence() function

◦ The input dimension for each sample is specified in the input dim argument on the definition of first hidden layer
◦ The model will view each time step as a separate feature instead of separate time steps

Workshop on Deep Learning Time Series by [email protected] July 2022 311

 Univariate MLP Models
MLP Model
◦ The model will expect the input component of training data to have the shape: [samples, features]
◦ The split_sequence() function in the previous section outputs the X with the shape [samples, features] ready
to use for modeling
◦ The model is fit using the eficient Adam version of stochastic gradient descent and optimized using the mean squared
error, or `mse', loss function

# fit model
model.fit(X, y, epochs=2000, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 312

 Univariate MLP Models
MLP Model
◦ Predict the next value in the sequence by providing the input: [70, 80, 90]
◦ Expecting the model to predict something like: [100]
◦ The model expects the input shape to be two-dimensional with [samples, features]
◦ Reshape the single input sample before making the prediction
◦ The shape [1, 3] for 1 sample and 3 time steps used as input features

# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps))
yhat = model.predict(x_input, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 313

 Univariate MLP Models
MLP Model
◦ Running the example, prepares the data, fits the model, and makes a prediction
◦ The model predicts the next value in the sequence
◦ Given the stochastic nature of the algorithm, specific results may vary
◦ Consider running the example a few times
[[100.00006]]

Workshop on Deep Learning Time Series by [email protected] July 2022 314

Multivariate MLP Models
Develop MLPs for Time Series Forecasting
 Multivariate MLP Models
◦ Multivariate time series data means
◦ More than one observation for each time step

◦ Two main models with multivariate time series data

1. Multiple Input Series
2. Multiple Parallel Series

Workshop on Deep Learning Time Series by [email protected] July 2022 316

 Multivariate MLP Models
Multiple Input Series – Data Preparation
◦ A problem may have two or more parallel input time series and an output time series that is dependent on the
input time series
◦ The input time series are parallel because each series has an observation at the same time step

◦ Two parallel input time series where the output series is the simple addition of the input series
# multivariate data preparation
from numpy import hstack

# define input sequence

in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

Workshop on Deep Learning Time Series by [email protected] July 2022 317

 Multivariate MLP Models
Multiple Input Series – Data Preparation
◦ Reshape these three arrays of data as a single dataset
◦ Each row is a time step and each column is a separate time series
◦ This is a standard way of storing parallel time series in a CSV file

from numpy import hstack

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

Workshop on Deep Learning Time Series by [email protected] July 2022 318

 Multivariate MLP Models
Multiple Input Series – Data Preparation
◦ Parallel dependent time series
# multivariate data preparation
from numpy import array
from numpy import hstack

# define input sequence

in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))
print(dataset)

Workshop on Deep Learning Time Series by [email protected] July 2022 319

 Multivariate MLP Models
Multiple Input Series – Data Preparation
◦ Running the example
◦ Prints the dataset with one row per time step and one column for each of the two input and one output parallel time series

[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125]
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 320

 Multivariate MLP Models
Multiple Input Series – Data Preparation
◦ As with the univariate time series, structure these data into samples with input and output samples
◦ Split the data into samples maintaining the order of observations across the two input sequences
◦ Chose three input time steps:
X: Three input time steps

[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105] y: One output time step
[ 60 65 125]
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 321

 Multivariate MLP Models
Multiple Input Series – Data Preparation
◦ The first three time steps of each parallel series are provided as input to the model
◦ The model associates this with the value in the output series at the third time step

◦ Transforming the time series into input/output samples to train the model
◦ Discard some values from the output time series where do not have values in the input time series at prior time steps
◦ The choice of the size of the number of input time steps will have an important effect on how much of the training data is
used

Workshop on Deep Learning Time Series by [email protected] July 2022 322

 Multivariate MLP Models
Multiple Input Series – Data Preparation
◦ Define a function named split_sequences() that will take a dataset
◦ Rows for time steps
◦ Columns for parallel series
◦ Return input/output samples
# multivariate data preparation
from numpy import array

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 323

 Multivariate MLP Models 1/2
Multiple Input Series – Data Preparation
◦ Transforming a parallel dependent time series into a supervised learning problem
# multivariate data preparation
from numpy import array

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

# define input sequence

in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

Workshop on Deep Learning Time Series by [email protected] July 2022 324

 Multivariate MLP Models 2/2
Multiple Input Series – Data Preparation
◦ Transforming a parallel dependent time series into a supervised learning problem
# convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

from numpy import hstack

# horizontally stack columns
dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3

# convert into input/output

X, y = split_sequences(dataset, n_steps)
print(X.shape, y.shape)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 325

 Multivariate MLP Models
Multiple Input Series – Data Preparation
◦ Running the example first prints the shape of the X and y components
The second dimension is the number of time steps per sample, in this case 3
(7, 3, 2) (7,)
[[10 15]
[20 25]
[30 35]] 65 The last dimension specifies the number of parallel time series or the number
[[20 25] of variables, in this case 2 for the two parallel series
[30 35]
[40 45]] 85
[[30 35] The first dimension is the number of samples, in this case 7
[40 45]
[50 55]] 105
[[40 45]
[50 55]
[60 65]] 125
[[50 55]
[60 65]
[70 75]] 145
[[60 65]
[70 75]
[80 85]] 165
[[70 75]
[80 85]
[90 95]] 185

The X component has a three-dimensional structure

Workshop on Deep Learning Time Series by [email protected] July 2022 326

 Multivariate MLP Models
Multiple Input Series – MLP Model
◦ Before fit an MLP on this data, must flatten the shape of the input samples
◦ MLPs require that the shape of the input portion of each sample is a vector
◦ With a multivariate input, have multiple vectors, one for each time step
◦ Flatten the temporal structure of each input sample

[[10 15]
[20 25]
[30 35]]

Flatten

[10, 15, 20, 25, 30, 35]

Workshop on Deep Learning Time Series by [email protected] July 2022 327

 Multivariate MLP Models
Multiple Input Series – MLP Model
◦ Flatten the input
◦ Calculate the length of each input vector as the number of time steps multiplied by the number of features or time series
◦ Use this vector size to reshape the input
# flatten input
n_input = X.shape[1] * X.shape[2]
X = X.reshape((X.shape[0], n_input))

Workshop on Deep Learning Time Series by [email protected] July 2022 328

 Multivariate MLP Models
Multiple Input Series – MLP Model
◦ Define an MLP model for the multivariate input where the vector length is used for the input dimension argument
# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_input))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 329

 Multivariate MLP Models
Multiple Input Series – MLP Model
◦ Making a prediction, the model expects three time steps for two input time series
◦ Predict the next value in the output series proving the input values of
80, 85
90, 95
100, 105

◦ The shape of the 1 sample with 3 time steps and 2 variables would be [1, 3, 2]
◦ Reshape this to be 1 sample with a vector of 6 elements or [1, 6]
◦ Expect the next value in the sequence to be 100 + 105 or 205
# demonstrate prediction
x_input = array([[80, 85], [90, 95], [100, 105]])
x_input = x_input.reshape((1, n_input))
yhat = model.predict(x_input, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 330

 Multivariate MLP Models
Multiple Input Series – MLP Model
◦ Running the example prepares the data, fits the model, and makes a prediction
◦ Given the stochastic nature of the algorithm, your specific results may vary
◦ Consider running the example a few times

[[205.86246]]

Workshop on Deep Learning Time Series by [email protected] July 2022 331

 Multivariate MLP Models
Multi-headed MLP Model
◦ There is another more elaborate way to model the problem - multi-headed input MLP model
◦ Each input series can be handled by a separate MLP
◦ The output of each of these submodels can be combined before a prediction is made for the output sequence
◦ It may offer more flexibility or better performance depending on the specifics of the problem that are being modeled
◦ This type of model can be defined in Keras using the Keras functional API

Define the first input model as an MLP with an input layer that expects vectors
# first input model with n_steps features
visible1 = Input(shape=(n_steps,))
dense1 = Dense(100, activation='relu')(visible1)

define the second input submodel in the same way

# second input model
visible2 = Input(shape=(n_steps,))
dense2 = Dense(100, activation='relu')(visible2)

Workshop on Deep Learning Time Series by [email protected] July 2022 332

 Multivariate MLP Models
Multi-headed MLP Model
◦ Merge the output from each model into one long vector
◦ Can be interpreted before making a prediction for the output sequence

# merge input models

merge = concatenate([dense1, dense2])
output = Dense(1)(merge)

one long vector

◦ Connecting the input and output elements together.

# connect input and output models
model = Model(inputs=[visible1, visible2], outputs=output)

Workshop on Deep Learning Time Series by [email protected] July 2022 333

 Multivariate MLP Models
Multi-headed MLP Model
◦ A schematic of the model, including the shape of the inputs and outputs of each layer

◦ Requires input as a list of two elements, where each element in the list contains data for one of the submodels

Workshop on Deep Learning Time Series by [email protected] July 2022 334

 Multivariate MLP Models
Multi-headed MLP Model
◦ Split the 3D input data into two separate arrays of input data
◦ From one array with the shape [7, 3, 2] to two 2D arrays with the shape [7, 3]
# separate input data
X1 = X[:, :, 0]
X2 = X[:, :, 1]

◦ Data provided in order to fit the model

# fit model
model.fit([X1, X2], y, epochs=2000, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 335

 Multivariate MLP Models
Multi-headed MLP Model
◦ Prepare the data for a single sample as two separate two-dimensional arrays when making a single one-step
prediction
# reshape one sample for making a forecast
x_input = array([[80, 85], [90, 95], [100, 105]])
x1 = x_input[:, 0].reshape((1, n_steps))
x2 = x_input[:, 1].reshape((1, n_steps))

Workshop on Deep Learning Time Series by [email protected] July 2022 336

 Multivariate MLP Models
Multi-headed MLP Model
◦ Running the example prepares the data, fits the model, and makes a prediction
◦ Given the stochastic nature of the algorithm, your specific results may vary
◦ Consider running the example a few times
[[205.9294]]

Workshop on Deep Learning Time Series by [email protected] July 2022 337

 Multivariate MLP Models
Multiple Parallel Series – Data Preparation
◦ An alternate time series problem is the case where there are multiple parallel time series and a value must be
predicted for each
[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125]
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 338

 Multivariate MLP Models
Multiple Parallel Series – Data Preparation
◦ Want to predict the value for each of the three time series for the next time step
◦ This might be referred to as multivariate forecasting
◦ The data must be split into input/output samples in order to train a model
Input

[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125]
Output
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 339

 Multivariate MLP Models
Multiple Parallel Series – Data Preparation
◦ The split sequences() function split multiple parallel time series with rows for time steps and one series per
column into the required input/output shape
# multivariate output data prep
from numpy import array

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 340

 Multivariate MLP Models
Multiple Parallel Series – Data Preparation
◦ Running the example first prints the shape of the prepared X and y components
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3

# convert into input/output

X, y = split_sequences(dataset, n_steps)
print(X.shape, y.shape)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 341

 Multivariate MLP Models
Multiple Parallel Series – Data Preparation
◦ Splitting a multivariate time series into a supervised learning problem
The shape of X is three-dimensional
(6, 3, 3) (6, 3) • The number of samples (6)
• The number of time steps chosen per sample (3)
[[10 15 25] • The number of parallel time series or features (3)
[20 25 45]
[30 35 65]] [40 45 85] The shape of y is two-dimensional
[[20 25 45] • The number of samples (6)
[30 35 65] • The number of time variables per sample to be predicted (3)
[40 45 85]] [ 50 55 105]
[[ 30 35 65]
[ 40 45 85] Input
[ 50 55 105]] [ 60 65 125]
[[ 40 45 85]
[ 50 55 105]
[ 60 65 125]] [ 70 75 145]
[[ 50 55 105]
[ 60 65 125]
[ 70 75 145]] [ 80 85 165] Output
[[ 60 65 125]
[ 70 75 145]
[ 80 85 165]] [ 90 95 185]

Workshop on Deep Learning Time Series by [email protected] July 2022 342

 Multivariate MLP Models
Multiple Parallel Series – Vector-Output MLP Model
◦ Fit an MLP model on the data
◦ Flatten the three dimensional structure of the input data samples to a two dimensional structure of [samples, features],
where lag observations are treated as features by the model
# flatten input
n_input = X.shape[1] * X.shape[2]
X = X.reshape((X.shape[0], n_input))

◦ The model output will be a vector, with one element for each of the three different time series
# determine the number of outputs
n_output = y.shape[1]

Workshop on Deep Learning Time Series by [email protected] July 2022 343

 Multivariate MLP Models
Multiple Parallel Series – Vector-Output MLP Model
◦ Define our model, using the flattened vector length for the input layer and the number of time series as the vector
length when making a prediction
# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_input))
model.add(Dense(n_output))
model.compile(optimizer='adam', loss='mse')

◦ Predict the next value in each of the three parallel series by providing an input of three time steps for each series
70, 75, 145
80, 85, 165
90, 95, 185

Workshop on Deep Learning Time Series by [email protected] July 2022 344

 Multivariate MLP Models
Multiple Parallel Series – Vector-Output MLP Model
◦ The shape of the input for making a single prediction must be 1 sample, 3 time steps and 3 features, or [1, 3, 3]
◦ Flatten this to [1, 9] to meet the expectations of the model
◦ Expect the vector output to be:
[100, 105, 205]

◦ Preparing data for making an out-of-sample forecast with a MLP

# demonstrate prediction
x_input = array([[70,75,145], [80,85,165], [90,95,185]])
x_input = x_input.reshape((1, n_input))
yhat = model.predict(x_input, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 345

 Multivariate MLP Models
Multiple Parallel Series – Vector-Output MLP Model
◦ An MLP for multivariate time series forecasting
# multivariate output mlp example
from numpy import array
from numpy import hstack
from keras.models import Sequential
from keras.layers import Dense

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 346

 Multivariate MLP Models
Multiple Parallel Series – Vector-Output MLP Model
◦ An MLP for multivariate time series forecasting
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3

# convert into input/output

X, y = split_sequences(dataset, n_steps)

# flatten input
n_input = X.shape[1] * X.shape[2]
X = X.reshape((X.shape[0], n_input))
n_output = y.shape[1]

Workshop on Deep Learning Time Series by [email protected] July 2022 347

 Multivariate MLP Models
Multiple Parallel Series – Vector-Output MLP Model
◦ An MLP for multivariate time series forecasting
# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_input))
model.add(Dense(n_output))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=2000, verbose=0)

# demonstrate prediction
x_input = array([[70,75,145], [80,85,165], [90,95,185]])
x_input = x_input.reshape((1, n_input))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 348

 Multivariate MLP Models
Multiple Parallel Series – Vector-Output MLP Model
◦ Running the example prepares the data, fits the model, and makes a prediction
◦ Given the stochastic nature of the algorithm, your specific results may vary
◦ Consider running the example a few times
[[100.10145 104.94529 205.05528]]

Workshop on Deep Learning Time Series by [email protected] July 2022 349

 Multivariate MLP Models
Multiple Parallel Series – Multi-output MLP Model
◦ Multi-output MLP Model
◦ Each output series were handled by a separate output MLP model
◦ Offer more exibility or better performance depending on the specifics of the problem that is being modeled.
◦ This type of model can be defined in Keras using the Keras functional API
◦ Define the input model as an MLP with an input layer that expects flattened feature vectors
# define model
visible = Input(shape=(n_input,))
dense = Dense(100, activation='relu')(visible)

◦ Define one output layer for each of the three series that wish to forecast, where each output submodel will
forecast a single time step
# define output 1
output1 = Dense(1)(dense)
# define output 2
output2 = Dense(1)(dense)
# define output 2
output3 = Dense(1)(dense)

Workshop on Deep Learning Time Series by [email protected] July 2022 350

 Multivariate MLP Models
Multiple Parallel Series – Multi-output MLP Model
◦ Tie the input and output layers together into a single model
# tie together
model = Model(inputs=visible, outputs=[output1, output2, output3])
model.compile(optimizer='adam', loss='mse')

◦ The schematic shows the three separate output layers of the model and the input and output shapes of each
layer

Workshop on Deep Learning Time Series by [email protected] July 2022 351

 Multivariate MLP Models
Multiple Parallel Series – Multi-output MLP Model
◦ When training the model, it will require three separate output arrays per sample
◦ Converting the output training data that has the shape [7, 3] to three arrays with the shape [7, 1]
# separate output
y1 = y[:, 0].reshape((y.shape[0], 1))
y2 = y[:, 1].reshape((y.shape[0], 1))
y3 = y[:, 2].reshape((y.shape[0], 1))

◦ Fitting the multi-output MLP model

# fit model
model.fit(X, [y1,y2,y3], epochs=2000, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 352

 Multivariate MLP Models
Multiple Parallel Series – Multi-output MLP Model
◦ A multi-output MLP for multivariate time series forecasting
# multivariate output mlp example
from numpy import array
from numpy import hstack
from keras.models import Model
from keras.layers import Input
from keras.layers import Dense

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 353

 Multivariate MLP Models
Multiple Parallel Series – Multi-output MLP Model
◦ A multi-output MLP for multivariate time series forecasting
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3

# convert into input/output

X, y = split_sequences(dataset, n_steps)

# flatten input
n_input = X.shape[1] * X.shape[2]
X = X.reshape((X.shape[0], n_input))

Workshop on Deep Learning Time Series by [email protected] July 2022 354

 Multivariate MLP Models
Multiple Parallel Series – Multi-output MLP Model
◦ A multi-output MLP for multivariate time series forecasting
# separate output
y1 = y[:, 0].reshape((y.shape[0], 1))
y2 = y[:, 1].reshape((y.shape[0], 1))
y3 = y[:, 2].reshape((y.shape[0], 1))

# define model
visible = Input(shape=(n_input,))
dense = Dense(100, activation='relu')(visible)

# define output 1
output1 = Dense(1)(dense)
# define output 2
output2 = Dense(1)(dense)
# define output 2
output3 = Dense(1)(dense)

# tie together
model = Model(inputs=visible, outputs=[output1, output2, output3])
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, [y1,y2,y3], epochs=2000, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 355

 Multivariate MLP Models
Multiple Parallel Series – Multi-output MLP Model
◦ A multi-output MLP for multivariate time series forecasting
# demonstrate prediction
x_input = array([[70,75,145], [80,85,165], [90,95,185]])
x_input = x_input.reshape((1, n_input))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 356

 Multivariate MLP Models
Multiple Parallel Series – Multi-output MLP Model
◦ Running the example prepares the data, fits the model, and makes a prediction
◦ Given the stochastic nature of the algorithm, your specific results may vary
◦ Consider running the example a few times

[array([[101.16882]], dtype=float32),
array([[105.94562]], dtype=float32),
array([[207.73004]], dtype=float32)]

Workshop on Deep Learning Time Series by [email protected] July 2022 357

Multi-step MLP Models
Develop MLPs for Time Series Forecasting
 Multi-step MLP Models
◦ Demonstrate the case of developing a multi-step forecast model using a vector model

Workshop on Deep Learning Time Series by [email protected] July 2022 359

 Multi-step MLP Models
Data Preparation
◦ A time series used for multi-step time series forecasting must be split into samples with input and output
components
◦ Both the input and output components will be comprised of multiple time steps and may or may not have the same number
of steps
◦ The univariate time series:
Output for a multi-step forecast

[10, 20, 30, 40, 50, 60, 70, 80, 90]

Input for a multi-step forecast

Workshop on Deep Learning Time Series by [email protected] July 2022 360

 Multi-step MLP Models
Data Preparation
◦ The split_sequence() function split a given univariate time series into samples with a specified number of
input and output time steps
# split a univariate sequence into samples
def split_sequence(sequence, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the sequence
if out_end_ix > len(sequence):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 361

 Multi-step MLP Models
Data Preparation
◦ Demonstrate the function on the small contrived dataset
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# split into samples

X, y = split_sequence(raw_seq, n_steps_in, n_steps_out)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 362

 Multi-step MLP Models
Data Preparation
◦ Running the example, splits the univariate series into input and output time steps
◦ Prints the input and output components of each
Output for a multi-step forecast

[10 20 30] [40 50]

[20 30 40] [50 60]
[30 40 50] [60 70]
[40 50 60] [70 80]
[50 60 70] [80 90]

Input for a multi-step forecast

Workshop on Deep Learning Time Series by [email protected] July 2022 363

 Multi-step MLP Models
Vector Output Model
◦ The MLP can output a vector directly that can be interpreted as a multi-step forecast
◦ One time step of each output time series was forecasted as a vector
◦ With the number of input and output steps specified in the n_steps_in and n_steps_out variables

◦ Define a multi-step time-series forecasting model

# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_steps_in))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 364

 Multi-step MLP Models
Vector Output Model
◦ The model can make a prediction for a single sample
◦ Predict the next two steps beyond the end of the dataset by providing the input:
[70, 80, 90]

◦ Expect the predicted output to be:

[100, 110]

◦ The shape of the single sample of input data must be [1, 3] for the 1 sample and 3 time steps (features) of the
input and the single feature
# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps_in))
yhat = model.predict(x_input, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 365

 Multi-step MLP Models 1/2
Vector Output Model
◦ The MLP for multi-step forecasting with a univariate time series
# univariate multi-step vector-output mlp example
from numpy import array
from keras.models import Sequential
from keras.layers import Dense

# split a univariate sequence into samples

def split_sequence(sequence, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the sequence
if out_end_ix > len(sequence):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 366

 Multi-step MLP Models 2/2
Vector Output Model
◦ The MLP for multi-step forecasting with a univariate time series
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# split into samples

X, y = split_sequence(raw_seq, n_steps_in, n_steps_out)

# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_steps_in))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=2000, verbose=0)

# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps_in))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 367

 Multi-step MLP Models
Vector Output Model
◦ Running the example forecasts and prints the next two time steps in the sequence
◦ Given the stochastic nature of the algorithm, specific results may vary
◦ Consider running the example a few times

[[102.572365 113.88405 ]]

Workshop on Deep Learning Time Series by [email protected] July 2022 368

Multivariate Multi-step MLP
Models
Develop MLPs for Time Series Forecasting
 Multivariate Multi-step MLP Models
◦ It is possible to mix and match the different types of MLP models presented the different problems
◦ Applies to time series forecasting problems that involve multivariate and multi-step forecasting
◦ It may be a little more challenging, particularly in preparing the data and defining the shape of inputs and outputs for the
model

◦ Data preparation and modeling for multivariate multi-step time series forecasting
1. Multiple Input Multi-step Output
2. Multiple Parallel Input and Multi-step Output

Workshop on Deep Learning Time Series by [email protected] July 2022 370

 Multivariate Multi-step MLP Models
Multiple Input Multi-step Output
◦ Multivariate time series forecasting problems
◦ The output series is separate but dependent upon the input time series
◦ Multiple time steps are required for the output series

◦ Consider a multivariate time series with a dependent series:

◦ Use three prior time steps of each of the two input time series to predict two time steps of the output time series
Input for a multi-step forecast for a dependent series

[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125]
[ 70 75 145] Output for a multi-step forecast for a dependent series
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 371

 Multivariate Multi-step MLP Models
Multiple Input Multi-step Output
◦ The split_sequences() function implements splitting for multivariate sequence into samples
# split a multivariate sequence into samples
def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out-1
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 372

 Multivariate Multi-step MLP Models 1/2
Multiple Input Multi-step Output
◦ Demonstrate on the contrived dataset
# multivariate multi-step data preparation
from numpy import array
from numpy import hstack

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out-1
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

# define input sequence

in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

Workshop on Deep Learning Time Series by [email protected] July 2022 373

 Multivariate Multi-step MLP Models 2/2
Multiple Input Multi-step Output
◦ Demonstrate on the contrived dataset
# convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# convert into input/output

X, y = split_sequences(dataset, n_steps_in, n_steps_out)
print(X.shape, y.shape)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 374

 Multivariate Multi-step MLP Models
Multiple Input Multi-step Output
◦ Running the example and prints the shape of the prepared training data
The shape of the input portion of the samples is three-dimensional
(6, 3, 2) (6, 2) Comprised of six samples, with three time steps and two variables for the two input time series

[[10 15]
The output portion of the samples is two-dimensional for the six samples and the two time steps
[20 25]
for each sample to be predicted
[30 35]] [65 85]
[[20 25]
[30 35] Input for a multi-step forecast for a dependent series
[40 45]] [ 85 105]
[[30 35]
[40 45]
[50 55]] [105 125]
[[40 45]
[50 55]
[60 65]] [125 145] Output for a multi-step forecast for a dependent series
[[50 55]
[60 65]
[70 75]] [145 165]
[[60 65]
[70 75]
[80 85]] [165 185]

Workshop on Deep Learning Time Series by [email protected] July 2022 375

 Multivariate Multi-step MLP Models 1/3
Multiple Input Multi-step Output
◦ Develop an MLP model for multi-step predictions using a vector output
# multivariate multi-step mlp example
from numpy import array
from numpy import hstack
from keras.models import Sequential
from keras.layers import Dense

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out-1
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 376

 Multivariate Multi-step MLP Models 2/3
Multiple Input Multi-step Output
◦ Develop an MLP model for multi-step predictions using a vector output
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# convert into input/output

X, y = split_sequences(dataset, n_steps_in, n_steps_out)

# flatten input
n_input = X.shape[1] * X.shape[2]
X = X.reshape((X.shape[0], n_input))

Workshop on Deep Learning Time Series by [email protected] July 2022 377

 Multivariate Multi-step MLP Models 3/3
Multiple Input Multi-step Output
◦ Develop an MLP model for multi-step predictions using a vector output
# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_input))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=2000, verbose=0)

# demonstrate prediction
x_input = array([[70, 75], [80, 85], [90, 95]])
x_input = x_input.reshape((1, n_input))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 378

 Multivariate Multi-step MLP Models
Multiple Input Multi-step Output
◦ Running the example, fits the model and predicts the next two time steps of the output sequence beyond the
dataset
◦ Expect the next two steps to be [185, 205]

[[187.13754 208.63889]]

Workshop on Deep Learning Time Series by [email protected] July 2022 379

 Multivariate Multi-step MLP Models
Multiple Parallel Input and Multi-step Output
◦ A problem with parallel time series may require the prediction of multiple time steps of each time series
◦ Consider a multivariate time series
◦ Use the last three time steps from each of the three time series as input to the model and predict the next time steps of
each of the three time series as output
Input for the first a multivariate time series
[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125]
[ 70 75 145]
[ 80 85 165] Output for the first a multivariate time series
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 380

 Multivariate Multi-step MLP Models
Multiple Parallel Input and Multi-step Output
◦ The split_sequences() function split a multivariate sequence into samples
# split a multivariate sequence into samples
def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 381

 Multivariate Multi-step MLP Models 1/2
Multiple Parallel Input and Multi-step Output
◦ Demonstrate the function on the small contrived dataset
# multivariate multi-step data preparation
from numpy import array
from numpy import hstack

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 382

 Multivariate Multi-step MLP Models 2/2
Multiple Parallel Input and Multi-step Output
◦ Demonstrate the function on the small contrived dataset
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# convert into input/output

X, y = split_sequences(dataset, n_steps_in, n_steps_out)
print(X.shape, y.shape)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 383

 Multivariate Multi-step MLP Models
Multiple Parallel Input and Multi-step Output
◦ Running the example and prints the shape of the prepared training dataset

(5, 3, 3) (5, 2, 3)

[[10 15 25] Both the input (X) and output (Y ) elements of the dataset are three dimensional for
[20 25 45] the number of samples, time steps, and variables or parallel time series
[30 35 65]] [[ 40 45 85]
[ 50 55 105]]
[[20 25 45]
[30 35 65]
[40 45 85]] [[ 50 55 105]
[ 60 65 125]]
[[ 30 35 65] Input for multi-step forecasting for a multivariate series
[ 40 45 85]
[ 50 55 105]] [[ 60 65 125]
[ 70 75 145]]
[[ 40 45 85]
[ 50 55 105]
[ 60 65 125]] [[ 70 75 145]
[ 80 85 165]] Output for multi-step forecasting for a multivariate series
[[ 50 55 105]
[ 60 65 125]
[ 70 75 145]] [[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 384

 Multivariate Multi-step MLP Models
Multiple Parallel Input and Multi-step Output
◦ Develop an MLP model to make multivariate multi-step forecasts
◦ Flattening the shape of the input data, must also flatten the three-dimensional structure of the output data
◦ The MLP model is only capable of taking vector inputs and outputs
# flatten input
n_input = X.shape[1] * X.shape[2]
X = X.reshape((X.shape[0], n_input))

# flatten output
n_output = y.shape[1] * y.shape[2]
y = y.reshape((y.shape[0], n_output))

Workshop on Deep Learning Time Series by [email protected] July 2022 385

 Multivariate Multi-step MLP Models 1/3
Multiple Parallel Input and Multi-step Output
◦ An MLP model for multi-step forecasting for a multivariate series
# multivariate multi-step mlp example
from numpy import array
from numpy import hstack
from keras.models import Sequential
from keras.layers import Dense

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 386

 Multivariate Multi-step MLP Models 2/3
Multiple Parallel Input and Multi-step Output
◦ An MLP model for multi-step forecasting for a multivariate series
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# convert into input/output

X, y = split_sequences(dataset, n_steps_in, n_steps_out)

# flatten input
n_input = X.shape[1] * X.shape[2]
X = X.reshape((X.shape[0], n_input))

Workshop on Deep Learning Time Series by [email protected] July 2022 387

 Multivariate Multi-step MLP Models 3/3
Multiple Parallel Input and Multi-step Output
◦ An MLP model for multi-step forecasting for a multivariate series
# flatten output
n_output = y.shape[1] * y.shape[2]
y = y.reshape((y.shape[0], n_output))

# define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_input))
model.add(Dense(n_output))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=2000, verbose=0)

# demonstrate prediction
x_input = array([[60, 65, 125], [70, 75, 145], [80, 85, 165]])
x_input = x_input.reshape((1, n_input))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 388

 Multivariate Multi-step MLP Models
Multiple Parallel Input and Multi-step Output
◦ Running the example, fits the model and predicts the values for each of the three time steps for the next two time
steps beyond the end of the dataset
◦ Expect the values for these series and time steps to be as follows
90, 95, 185
100, 105, 205

◦ The model forecast gets reasonably close to the expected values

[[ 89.49298 95.07807 186.1331
100.96125 105.88323 206.4983 ]]

Workshop on Deep Learning Time Series by [email protected] July 2022 389

 MLP Project
Develop MLPs for Time Series Forecasting
 MLP Project
◦ Create Minimum Daily Temperature Forecasting Model using MLP!
◦ Train the model using data over 8 years (1981-1988)
◦ Test the model using data over 2 years (1989-1990)
◦ Find the performance using RMSE
◦ Compare with the proper Classical Methods

Workshop on Deep Learning Time Series by [email protected] July 2022 391

Develop CNNs for Time Series
Forecasting
Badan Meteorologi, Klimatologi, dan Geofisika
Juli – Agustus 2022
 Developing CNNs for Time Series Forecasting
◦ Convolutional Neural Network models (CNNs) can be applied to time series forecasting
◦ There are many types of CNN models that can be used for each specific type of time series forecasting problem
◦ How to develop a suite of CNN models for a range of standard time series forecasting problems

◦ Provide standalone examples of each model on each type of time series problem as a template that can copy and
adapt for the specific time series forecasting problem
◦ How to develop CNN models for univariate time series forecasting
◦ How to develop CNN models for multivariate time series forecasting
◦ How to develop CNN models for multi-step time series forecasting

Workshop on Deep Learning Time Series by [email protected] July 2022 393

 CNN Model for Time Series Forecasting
1. Univariate CNN Models

• Data Preparation
• CNN Model

2. Multivariate CNN Models

• Multiple Input Series

• Multiple Parallel Series

3. Multi-step CNN Models

4. Multivariate Multi-step CNN Models

Workshop on Deep Learning Time Series by [email protected] July 2022 394

Univariate CNN Models
Develop CNNs for Time Series Forecasting
 1. Univariate CNN Models
Data Preparation
◦ The CNN model learn a function that maps a sequence of past observations as input to an output observation
◦ The sequence of observations must be transformed into multiple examples from which the model can learn
A given univariate sequence
[10, 20, 30, 40, 50, 60, 70, 80, 90]

◦ Divide the sequence into multiple input/output patterns called samples for the one-step prediction that is being
learned
Three time steps are used as input

X, y
10, 20, 30, 40
20, 30, 40, 50
30, 40, 50, 60 One time step is used as output
...

Workshop on Deep Learning Time Series by [email protected] July 2022 396

 1. Univariate CNN Models
Data Preparation
◦ The split_sequence() function split a given univariate sequence into multiple samples
◦ Each sample has a specified number of time steps
◦ The output is a single time step
# univariate data preparation
from numpy import array

# split a univariate sequence into samples

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 397

 1. Univariate CNN Models
Data Preparation
◦ Transforming a univariate time series into a supervised learning problem
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps = 3

# split into samples

X, y = split_sequence(raw_seq, n_steps)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 398

 1. Univariate CNN Models
Data Preparation
◦ Running the example
◦ Splits the univariate series into six samples
◦ Each sample has three input time steps and one output time step

[10 20 30] 40
[20 30 40] 50
[30 40 50] 60
[40 50 60] 70
[50 60 70] 80
[60 70 80] 90

Workshop on Deep Learning Time Series by [email protected] July 2022 399

 1. Univariate CNN Models
CNN Model
◦ A 1D-CNN is a CNN model that has a convolutional hidden layer that operates over a 1D sequence

◦ Perhaps a second convolutional layer has very long input sequences

◦ A pooling layer distill the output of the convolutional layer to the most salient elements

◦ The convolutional and pooling layers are followed by a dense fully connected layer that interprets the features
extracted by the convolutional part of the model

◦ A flatten layer is used between the convolutional layers and the dense layer to reduce the feature maps to a
single one-dimensional vector

Workshop on Deep Learning Time Series by [email protected] July 2022 400

 1. Univariate CNN Models
CNN Model
◦ Define a 1D CNN Model for univariate time series forecasting
# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 401

 1. Univariate CNN Models
CNN Model
◦ Key in the definition is the shape of the input
◦ The model expects as input for each sample in terms of the number of time steps and the number of features

◦ Working with a univariate series

◦ The number of features is one, for one variable
◦ The number of time steps as input is the number as an argument to the split_sequence() function

Workshop on Deep Learning Time Series by [email protected] July 2022 402

 1. Univariate CNN Models
CNN Model
◦ The input shape for each sample is specified in the input shape argument on the definition of the first hidden
layer
◦ Always have multiple samples
◦ The model will expect the input component of training data to have the dimensions or shape: [samples,timesteps,
features]

◦ The split_sequence() function in the previous section outputs the X with the shape [samples,
timesteps]
◦ Reshape it to have an additional dimension for the one feature
# reshape from [samples, timesteps] into [samples, timesteps, features]
n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

Workshop on Deep Learning Time Series by [email protected] July 2022 403

 1. Univariate CNN Models
CNN Model
◦ The CNN does not actually view the data as having time steps
◦ it is treated as a sequence over which convolutional read operations can be performed, like a one-dimensional image
◦ Define a convolutional layer with 64 filter maps and a kernel size of 2
◦ Followed by a max pooling layer and a dense layer to interpret the input feature

◦ An output layer is specified that predicts a single numerical value

◦ The model is fit using the eficient Adam version of stochastic gradient descent and optimized using the mean squared
error loss function
◦ Once the model is defined, fit it on the training dataset
# fit model
model.fit(X, y, epochs=1000, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 404

 1. Univariate CNN Models
CNN Model
◦ After the model is fit, use it to make a prediction
◦ Predict the next value in the sequence by providing the input: [70, 80, 90]
◦ Expecting the model to predict something like: [100]

◦ The model expects the input shape to be 3D with [samples, timesteps, features]
◦ Reshape the single input sample before making the prediction
# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 405

 1. Univariate CNN Models
CNN Model
◦ A 1D CNN model for univariate time series forecasting and make a single prediction
# univariate cnn example
from numpy import array
from keras import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers.convolutional import Conv1D
from keras.layers.convolutional import MaxPooling1D

# split a univariate sequence into samples

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 406

 1. Univariate CNN Models
CNN Model
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps = 3

# split into samples

X, y = split_sequence(raw_seq, n_steps)

# reshape from [samples, timesteps] into [samples, timesteps, features]

n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 407

 1. Univariate CNN Models
CNN Model
# fit model
model.fit(X, y, epochs=1000, verbose=0)

# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 408

 1. Univariate CNN Models
CNN Model
◦ Running the example, prepares the data, fits the model, and makes a prediction
◦ The model predicts the next value in the sequence
[[101.4986]]

Workshop on Deep Learning Time Series by [email protected] July 2022 409

Multivariate CNN Models
Develop CNNs for Time Series Forecasting
 Multivariate CNN Models
◦ Multivariate time series data means data where there is more than one observation for each time step

◦ There are two main models

1. Multiple Input Series
2. Multiple Parallel Series

Workshop on Deep Learning Time Series by [email protected] July 2022 411

 2. Multivariate CNN Models
Multiple Input Series – Data Preparation
◦ A problem may have
◦ Two or more parallel input time series
◦ Single output time series that is dependent on the input time series

◦ The input time series are parallel

◦ Each series has observations at the same time steps
◦ The input is two parallel time series
◦ The output series is single as the simple addition of the input series

# define input sequence

in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

Workshop on Deep Learning Time Series by [email protected] July 2022 412

 2. Multivariate CNN Models
Multiple Input Series – Data Preparation
◦ Reshape these three arrays of data as a single dataset
◦ Each row is a time step
◦ Each column is a separate time series
# convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

Workshop on Deep Learning Time Series by [email protected] July 2022 413

 2. Multivariate CNN Models
Multiple Input Series – Data Preparation
◦ Defining a dependent time series dataset
# multivariate data preparation
from numpy import array
from numpy import hstack

# define input sequence

in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))
print(dataset)

Workshop on Deep Learning Time Series by [email protected] July 2022 414

 2. Multivariate CNN Models
Multiple Input Series – Data Preparation
◦ Running the example
◦ One row per time step
◦ One column for each of the two input
◦ One output parallel time series
[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125]
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 415

 2. Multivariate CNN Models
Multiple Input Series – Data Preparation
◦ As the univariate time series
◦ Must structure these data into samples with input and output samples
◦ A 1D CNN model needs sucient context to learn a mapping from an input sequence to an output value
◦ CNNs can support parallel input time series as separate channels, like red, green, and blue components of an image
◦ Need to split the data into samples maintaining the order of observations across the two input sequences
X: Three input time steps

[[ 10 15 25]
The first three time steps of each parallel series are provided as input to the model
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105] y: One output time step
[ 60 65 125]
[ 70 75 145] The model associates this with the value in the output series at the third time step
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 416

 2. Multivariate CNN Models
Multiple Input Series – Data Preparation
◦ Transforming the time series into input/output samples to train the model
◦ Have to discard some values from the output time series where do not have values in the input time series at prior time
steps
◦ The choice of the size of the number of input time steps will have an important effect on how much of the training data is
used
◦ Define a function named split_sequences() that take a dataset with rows for time steps and columns for
parallel series and return input/output samples
# split a multivariate sequence into samples
def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 417

 2. Multivariate CNN Models
Multiple Input Series – Data Preparation
◦ Test this function on the dataset using three time steps for each input time series as input
# multivariate data preparation
from numpy import array
from numpy import hstack

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

# define input sequence

in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

Workshop on Deep Learning Time Series by [email protected] July 2022 418

 2. Multivariate CNN Models
Multiple Input Series – Data Preparation
# convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3

# convert into input/output

X, y = split_sequences(dataset, n_steps)
print(X.shape, y.shape)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 419

 2. Multivariate CNN Models
Multiple Input Series – Data Preparation
◦ Running the example, prints the shape of the X and y components
The X component has a three-dimensional structure
(7, 3, 2) (7,)

[[10 15] • The first dimension is the number of samples, 7

[20 25]
[30 35]] 65
• The second dimension is the number of time steps per sample, 3, the value
[[20 25] specified to the function
[30 35] • The last dimension specifies the number of parallel time series or the number of
[40 45]] 85 variables, 2, for the two parallel series
[[30 35]
[40 45]
[50 55]] 105
[[40 45]
[50 55]
[60 65]] 125 The exact three-dimensional structure expected by a 1D CNN as input
[[50 55]
[60 65]
[70 75]] 145 The three time steps for each of the two input series
[[60 65]
[70 75]
[80 85]] 165
[[70 75]
[80 85]
[90 95]] 185

Workshop on Deep Learning Time Series by [email protected] July 2022 420

 2. Multivariate CNN Models
Multiple Input Series – CNN Model
◦ Fit a 1D CNN model on this data
◦ Specifying the expected number of time steps and features to expect for each input sample

# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 421

 2. Multivariate CNN Models
Multiple Input Series – CNN Model
◦ When making a prediction
◦ The model expects three time steps for two input time series
◦ Can predict the next value in the output series providing the input values of
80, 85
90, 95
100, 105

◦ The shape of the one sample with three time steps and two variables must be [1, 3, 2]
◦ Expect the next value in the sequence to be 100 + 105 or 205
# demonstrate prediction
x_input = array([[80, 85], [90, 95], [100, 105]])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 422

 2. Multivariate CNN Models
Multiple Input Series – CNN Model
◦ A CNN model for forecasting a dependent time series
# multivariate cnn example
from numpy import array
from numpy import hstack
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers.convolutional import Conv1D
from keras.layers.convolutional import MaxPooling1D

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 423

 2. Multivariate CNN Models
Multiple Input Series – CNN Model
◦ A CNN model for forecasting a dependent time series
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3

# convert into input/output

X, y = split_sequences(dataset, n_steps)

# the dataset knows the number of features, e.g. 2

n_features = X.shape[2]

Workshop on Deep Learning Time Series by [email protected] July 2022 424

 2. Multivariate CNN Models
Multiple Input Series – CNN Model
◦ A CNN model for forecasting a dependent time series
# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=1000, verbose=0)

# demonstrate prediction
x_input = array([[80, 85], [90, 95], [100, 105]])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 425

 2. Multivariate CNN Models
Multiple Input Series – CNN Model
◦ Running the example, prepares the data, fits the model, and makes a prediction
[[207.40111]]

Workshop on Deep Learning Time Series by [email protected] July 2022 426

 2. Multivariate CNN Models
Multi-headed CNN Model
◦ More elaborate way to model the problem - Multi-headed CNN Model
◦ Each input series can be handled by a separate CNN
◦ The output of each of these submodels can be combined before a prediction is made for the output sequence
◦ It may offer more flexibility or better performance depending on the specifics of the problem that is being modeled

◦ It allows to configure each submodel differently for each input series

◦ The number of filter maps
◦ The kernel size

◦ This type of model can be defined in Keras using the Keras functional API

Workshop on Deep Learning Time Series by [email protected] July 2022 427

 2. Multivariate CNN Models
Multi-headed CNN Model
◦ Define the first input model as a 1D CNN with an input layer that expects vectors with n_steps and 1 feature
# first input model
visible1 = Input(shape=(n_steps, n_features))
cnn1 = Conv1D(64, 2, activation='relu')(visible1)
cnn1 = MaxPooling1D()(cnn1)
cnn1 = Flatten()(cnn1)

◦ Define the second input submodel

# second input model
visible2 = Input(shape=(n_steps, n_features))
cnn2 = Conv1D(64, 2, activation='relu')(visible2)
cnn2 = MaxPooling1D()(cnn2)
cnn2 = Flatten()(cnn2)

Workshop on Deep Learning Time Series by [email protected] July 2022 428

 2. Multivariate CNN Models
Multi-headed CNN Model
◦ Both input submodels merge the output from each model into one long vector which can be interpreted before
making a prediction for the output sequence
# merge input models
merge = concatenate([cnn1, cnn2])
dense = Dense(50, activation='relu')(merge)
output = Dense(1)(dense)

◦ Tie the inputs and outputs together

# connect input and output models
model = Model(inputs=[visible1, visible2], outputs=output)

Workshop on Deep Learning Time Series by [email protected] July 2022 429

 2. Multivariate CNN Models
Multi-headed CNN Model
◦ Plot of Multi-headed 1D CNN for Multivariate Time Series Forecasting
The shape of the inputs

The shape of the Output

Workshop on Deep Learning Time Series by [email protected] July 2022 430

 2. Multivariate CNN Models
Multi-headed CNN Model
◦ This model requires input to be provided as a list of two elements
◦ Each element in the list contains data for one of the submodels
◦ Can split the 3D input data into two separate arrays of input data from one array with the shape [7, 3, 2] to two 3D arrays
with [7, 3, 1]
# one time series per head
n_features = 1

# separate input data

X1 = X[:, :, 0].reshape(X.shape[0], X.shape[1], n_features)
X2 = X[:, :, 1].reshape(X.shape[0], X.shape[1], n_features)

◦ These data can then be provided in order to fit the model

# fit model
model.fit([X1, X2], y, epochs=1000, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 431

 2. Multivariate CNN Models
Multi-headed CNN Model
◦ Prepare the data for a single sample as two separate two-dimensional arrays when making a single one-step
prediction
# reshape one sample for making a prediction
x_input = array([[80, 85], [90, 95], [100, 105]])
x1 = x_input[:, 0].reshape((1, n_steps, n_features))
x2 = x_input[:, 1].reshape((1, n_steps, n_features))

Workshop on Deep Learning Time Series by [email protected] July 2022 432

 2. Multivariate CNN Models
Multi-headed CNN Model
◦ A Multi-headed CNN for forecasting a dependent time series
# multivariate multi-headed 1d cnn example
from numpy import array
from numpy import hstack
from keras.models import Model
from keras.layers import Input, Dense, Flatten, concatenate
from keras.layers.convolutional import Conv1D, MaxPooling1D

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 433

 2. Multivariate CNN Models
Multi-headed CNN Model
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
# convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))
# horizontally stack columns
dataset = hstack((in_seq1, in_seq2, out_seq))
# choose a number of time steps
n_steps = 3
# convert into input/output
X, y = split_sequences(dataset, n_steps)
# one time series per head
n_features = 1
# separate input data
X1 = X[:, :, 0].reshape(X.shape[0], X.shape[1], n_features)
X2 = X[:, :, 1].reshape(X.shape[0], X.shape[1], n_features)

Workshop on Deep Learning Time Series by [email protected] July 2022 434

 2. Multivariate CNN Models
Multi-headed CNN Model
# first input model
visible1 = Input(shape=(n_steps, n_features))
cnn1 = Conv1D(64, 2, activation='relu')(visible1)
cnn1 = MaxPooling1D()(cnn1)
cnn1 = Flatten()(cnn1)
# second input model
visible2 = Input(shape=(n_steps, n_features))
cnn2 = Conv1D(64, 2, activation='relu')(visible2)
cnn2 = MaxPooling1D()(cnn2)
cnn2 = Flatten()(cnn2)
# merge input models
merge = concatenate([cnn1, cnn2])
dense = Dense(50, activation='relu')(merge)
output = Dense(1)(dense)
model = Model(inputs=[visible1, visible2], outputs=output)
model.compile(optimizer='adam', loss='mse')
# fit model
model.fit([X1, X2], y, epochs=1000, verbose=0)
# demonstrate prediction
x_input = array([[80, 85], [90, 95], [100, 105]])
x1 = x_input[:, 0].reshape((1, n_steps, n_features))
x2 = x_input[:, 1].reshape((1, n_steps, n_features))
yhat = model.predict([x1, x2], verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 435

 2. Multivariate CNN Models
Multi-headed CNN Model
◦ Running the example, prepares the data, fits the model, and makes a prediction
[[205.78104]]

Workshop on Deep Learning Time Series by [email protected] July 2022 436

 2. Multivariate CNN Models
Multiple Parallel Series
◦ An alternate time series problem
◦ There are multiple parallel time series and a value must be predicted for each

◦ Want to predict the value for each of the three time series for the next time step
◦ Referred to as multivariate forecasting
◦ The data must be split into input/output samples in order to train a model

X: Input

[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125] y: Output
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 437

 2. Multivariate CNN Models
Multiple Parallel Series
◦ The split_sequences() function split multiple parallel time series with rows for time steps and one series
per column into the required input/output shape
# split a multivariate sequence into samples
def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 438

 2. Multivariate CNN Models
Multiple Parallel Series
◦ Demonstrate on the contrived problem
# multivariate output data prep
from numpy import array
from numpy import hstack

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 439

 2. Multivariate CNN Models
Multiple Parallel Series
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
# convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))
# horizontally stack columns
dataset = hstack((in_seq1, in_seq2, out_seq))
# choose a number of time steps
n_steps = 3
# convert into input/output
X, y = split_sequences(dataset, n_steps)
print(X.shape, y.shape)
# summarize the data
for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 440

 2. Multivariate CNN Models
Multiple Parallel Series
◦ Running the example, prints the shape of the prepared X and y components
The X component has a three-dimensional structure
(6, 3, 3) (6, 3)

[[10 15 25] • The number of samples (6)

[20 25 45] • The number of time steps chosen per sample (3)
[30 35 65]] [40 45 85] • The number of parallel time series or features (3)
[[20 25 45]
[30 35 65] The shape of y is two-dimensional
[40 45 85]] [ 50 55 105]
[[ 30 35 65]
• The number of samples (6)
[ 40 45 85]
• The number of time variables per sample to be predicted (3)
[ 50 55 105]] [ 60 65 125]
[[ 40 45 85]
[ 50 55 105]
The exact three-dimensional structure expected by a 1D CNN as input X
[ 60 65 125]] [ 70 75 145]
[[ 50 55 105]
[ 60 65 125]
[ 70 75 145]] [ 80 85 165]
[[ 60 65 125]
[ 70 75 145]
Two-dimensional output shapes y
[ 80 85 165]] [ 90 95 185]

Workshop on Deep Learning Time Series by [email protected] July 2022 441

 2. Multivariate CNN Models
Multiple Parallel Series – Vector-Output CNN Model
◦ Ready to fit a 1D CNN model on this data
◦ The number of time steps and parallel series (features) are specified for the input layer via the input shape argument
◦ The number of parallel series is also used in the specification of the number of values to predict by the model in the output
layer

# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(n_features))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 442

 2. Multivariate CNN Models
Multiple Parallel Series – Vector-Output CNN Model
◦ Predict the next value in each of the three parallel series by providing an input of three time steps for each series
70, 75, 145
80, 85, 165
90, 95, 185

◦ The shape of the input for making a single prediction must be 1 sample, 3 time steps, and 3 features, or [1, 3, 3]
# demonstrate prediction
x_input = array([[70,75,145], [80,85,165], [90,95,185]])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)

◦ Expect the vector output to be: [100, 105, 205]

Workshop on Deep Learning Time Series by [email protected] July 2022 443

 2. Multivariate CNN Models
Multiple Parallel Series – Vector-Output CNN Model
◦ Tie all of this together and demonstrate a 1D CNN for multivariate output time series forecasting
# multivariate output 1d cnn example
from numpy import array
from numpy import hstack
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.layers.convolutional import Conv1D, MaxPooling1D

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 444

 2. Multivariate CNN Models
Multiple Parallel Series – Vector-Output CNN Model
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
# convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))
# horizontally stack columns
dataset = hstack((in_seq1, in_seq2, out_seq))
# choose a number of time steps
n_steps = 3
# convert into input/output
X, y = split_sequences(dataset, n_steps)
# the dataset knows the number of features, e.g. 2
n_features = X.shape[2]
# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(n_features))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 445

 2. Multivariate CNN Models
Multiple Parallel Series – Vector-Output CNN Model
# fit model
model.fit(X, y, epochs=3000, verbose=0)

# demonstrate prediction
x_input = array([[70,75,145], [80,85,165], [90,95,185]])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 446

 2. Multivariate CNN Models
Multiple Parallel Series – Vector-Output CNN Model
◦ Running the example, prepares the data, fits the model and makes a prediction
[[101.36304 106.10809 207.5859 ]]

Workshop on Deep Learning Time Series by [email protected] July 2022 447

 2. Multivariate CNN Models
Multiple Parallel Series – Multi-output CNN Model
◦ As with multiple input series
◦ Each output series can be handled by a separate output CNN model
◦ It may offer more flexibility or better performance depending on the specifics of the problem that is being modeled
◦ This type of model can be defined in Keras using the Keras functional API

◦ Define the input model as a 1D CNN model

# define model
visible = Input(shape=(n_steps, n_features))
cnn = Conv1D(64, 2, activation='relu')(visible)
cnn = MaxPooling1D()(cnn)
cnn = Flatten()(cnn)
cnn = Dense(50, activation='relu')(cnn)

Workshop on Deep Learning Time Series by [email protected] July 2022 448

 2. Multivariate CNN Models
Multiple Parallel Series – Multi-output CNN Model
◦ Define one output layer for each of the three series that we wish to forecast
◦ Each output submodel forecast a single time step
# define output 1
output1 = Dense(1)(cnn)
# define output 2
output2 = Dense(1)(cnn)
# define output 3
output3 = Dense(1)(cnn)

◦ Tie the input and output layers together into a single model
# tie together
model = Model(inputs=visible, outputs=[output1, output2, output3])
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 449

 2. Multivariate CNN Models
Multiple Parallel Series – Multi-output CNN Model
◦ Plot of Multi-output 1D CNN for Multivariate Time Series Forecasting

Workshop on Deep Learning Time Series by [email protected] July 2022 450

 2. Multivariate CNN Models
Multiple Parallel Series – Multi-output CNN Model
◦ Training the model
◦ Require three separate output arrays per sample
◦ Achieve this by converting the output training data that has the shape [7, 3] to three arrays with the shape [7, 1]
# separate output
y1 = y[:, 0].reshape((y.shape[0], 1))
y2 = y[:, 1].reshape((y.shape[0], 1))
y3 = y[:, 2].reshape((y.shape[0], 1))

◦ Provided to the model during training

# fit model
model.fit(X, [y1,y2,y3], epochs=2000, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 451

 2. Multivariate CNN Models
Multiple Parallel Series – Multi-output CNN Model
◦ A Multi-output CNN model for forecasting multiple parallel time series
# multivariate output 1d cnn example
from numpy import array
from numpy import hstack
from keras.models import Model
from keras.layers import Input, Dense, Flatten
from keras.layers.convolutional import Conv1D, MaxPooling1D

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 452

 2. Multivariate CNN Models
Multiple Parallel Series – Multi-output CNN Model
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3

# convert into input/output

X, y = split_sequences(dataset, n_steps)

# the dataset knows the number of features, e.g. 2

n_features = X.shape[2]

Workshop on Deep Learning Time Series by [email protected] July 2022 453

 2. Multivariate CNN Models
Multiple Parallel Series – Multi-output CNN Model
# separate output
y1 = y[:, 0].reshape((y.shape[0], 1))
y2 = y[:, 1].reshape((y.shape[0], 1))
y3 = y[:, 2].reshape((y.shape[0], 1))

# define model
visible = Input(shape=(n_steps, n_features))
cnn = Conv1D(64, 2, activation='relu')(visible)
cnn = MaxPooling1D()(cnn)
cnn = Flatten()(cnn)
cnn = Dense(50, activation='relu')(cnn)

# define output 1
output1 = Dense(1)(cnn)
# define output 2
output2 = Dense(1)(cnn)
# define output 3
output3 = Dense(1)(cnn)

# tie together
model = Model(inputs=visible, outputs=[output1, output2, output3])
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 454

 2. Multivariate CNN Models
Multiple Parallel Series – Multi-output CNN Model
# fit model
model.fit(X, [y1,y2,y3], epochs=2000, verbose=0)

# demonstrate prediction
x_input = array([[70,75,145], [80,85,165], [90,95,185]])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 455

 2. Multivariate CNN Models
Multiple Parallel Series – Multi-output CNN Model
◦ Running the example, prepares the data, fits the model, and makes a prediction
[array([[100.82]], dtype=float32),
array([[105.979256]], dtype=float32),
array([[207.18721]], dtype=float32)]

Workshop on Deep Learning Time Series by [email protected] July 2022 456

Multi-step CNN Models
Develop CNNs for Time Series Forecasting
 3. Multi-step CNN Models
Data Preparation
◦ There is little difference
◦ A 1D CNN model in predicting a vector output that represents different output variables
◦ A vector output that represents multiple time steps of one variable

◦ There are subtle and important differences in the way the training data is prepared
◦ Demonstrate the case of developing a multi-step forecast model using a vector model

Workshop on Deep Learning Time Series by [email protected] July 2022 458

 3. Multi-step CNN Models
Data Preparation
◦ As with one-step forecasting, a time series used for multi-step time series forecasting must be split into samples
with input and output components
◦ Both the input and output components weew comprised of multiple time steps and may or may not have the same number
of steps
◦ Given the univariate time series:
[10, 20, 30, 40, 50, 60, 70, 80, 90]

◦ Use the last three time steps as input and forecast the next two time steps
X: Input

[10, 20, 30, 40, 50, 60, 70, 80, 90]

y: Output

Workshop on Deep Learning Time Series by [email protected] July 2022 459

 3. Multi-step CNN Models
Data Preparation
◦ The split_sequence() function split a given univariate time series into samples with a specified number of
input and output time steps
# split a univariate sequence into samples
def split_sequence(sequence, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the sequence
if out_end_ix > len(sequence):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 460

 3. Multi-step CNN Models
Data Preparation
◦ Demonstrate the function on the small contrived dataset
# multi-step data preparation
from numpy import array

# split a univariate sequence into samples

def split_sequence(sequence, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the sequence
if out_end_ix > len(sequence):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 461

 3. Multi-step CNN Models
Data Preparation
◦ Demonstrate the function on the small contrived dataset
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# split into samples

X, y = split_sequence(raw_seq, n_steps_in, n_steps_out)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 462

 3. Multi-step CNN Models
Data Preparation
◦ Running the example splits the univariate series into input and output time steps
X: Input time steps

[10 20 30] [40 50]

[20 30 40] [50 60]
[30 40 50] [60 70]
[40 50 60] [70 80]
[50 60 70] [80 90]

y: Output time steps

Workshop on Deep Learning Time Series by [email protected] July 2022 463

 3. Multi-step CNN Models
Vector Output Model
◦ The 1D CNN can output a vector directly that can be interpreted as a multi-step forecast
◦ One time step of each output time series was forecasted as a vector
◦ As with the 1D CNN models for univariate data in a prior section, the prepared samples must be reshaped

◦ The CNN expects data to have a 3D structure of [samples, timesteps, features]

◦ The reshape is straightforward
# reshape from [samples, timesteps] into [samples, timesteps, features]
n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

Workshop on Deep Learning Time Series by [email protected] July 2022 464

 3. Multi-step CNN Models
Vector Output Model
◦ With the number of input and output steps specified in the n_steps_in and n_steps_out variables
◦ Define a multi-step time-series forecasting model
# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 465

 3. Multi-step CNN Models
Vector Output Model
◦ The model can make a prediction for a single sample
◦ Predict the next two steps beyond the end of the dataset by providing the input: [70, 80, 90]
◦ Expect the predicted output to be: [100, 110]
◦ The shape of the single sample of input data when making the prediction must be [1, 3, 1] for the 1 sample, 3 time steps of
the input, and the single feature

# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 466

 3. Multi-step CNN Models
Vector Output Model
◦ A vector-output CNN for multi-step forecasting
# univariate multi-step vector-output 1d cnn example
from numpy import array
from keras.models import Sequential
from keras.layers import Dense,Flatten
from keras.layers.convolutional import Conv1D, MaxPooling1D

# split a univariate sequence into samples

def split_sequence(sequence, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the sequence
if out_end_ix > len(sequence):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

# define input sequence

raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

Workshop on Deep Learning Time Series by [email protected] July 2022 467

 3. Multi-step CNN Models
Vector Output Model
# choose a number of time steps
n_steps_in, n_steps_out = 3, 2

# split into samples

X, y = split_sequence(raw_seq, n_steps_in, n_steps_out)

# reshape from [samples, timesteps] into [samples, timesteps, features]

n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 468

 3. Multi-step CNN Models
Vector Output Model
# fit model
model.fit(X, y, epochs=2000, verbose=0)

# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 469

 3. Multi-step CNN Models
Vector Output Model
◦ Running the example, forecasts and prints the next two time steps in the sequence
[[102.59686 114.22529]]

Workshop on Deep Learning Time Series by [email protected] July 2022 470

Multivariate Multi-step CNN
Models
Develop CNNs for Time Series Forecasting
 4. Multivariate Multi-step CNN Models
◦ It is possible to mix and match the different types of 1D CNN models presented
◦ Applies to time series forecasting problems that involve multivariate and multi-step forecasting

4. Multivariate Multi-step CNN Models

4.1. Multiple Input and Multi-step 4.2. Multiple Parallel Input and Multi-
Output step Output

Workshop on Deep Learning Time Series by [email protected] July 2022 472

 4. Multivariate Multi-step CNN Models
4.1. Multiple Input and Multi-step Output
◦ Multivariate time series forecasting problems
◦ The output series is separate but dependent upon the input time series
◦ Multiple time steps are required for the output series
X: Three prior time steps of each of the two input time series

[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125] y: two time steps of the output time series
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 473

 4. Multivariate Multi-step CNN Models
4.1. Multiple Input and Multi-step Output
◦ The split_sequences() function transform a multivariate time series into samples for multi-step forecasting
# multivariate output data prep
from numpy import array
from numpy import hstack

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 474

 4. Multivariate Multi-step CNN Models
4.1. Multiple Input and Multi-step Output
◦ Example preparing a multivariate input dependent time series with multi-step forecasts
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
# convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))
# horizontally stack columns
dataset = hstack((in_seq1, in_seq2, out_seq))
# choose a number of time steps
n_steps = 3
# convert into input/output
X, y = split_sequences(dataset, n_steps)
print(X.shape, y.shape)
# summarize the data
for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 475

 4. Multivariate Multi-step CNN Models
4.1. Multiple Input Multi-step Output
◦ Running the example, prints the shape of the prepared training data
The output samples is two-dimensional
(6, 3, 3) (6, 3)  The six samples
 The two time steps for each sample to be predicted
[[10 15 25]
[20 25 45] The shape of the input samples is three-dimensional
 Comprised of six samples,
[30 35 65]] [40 45 85]  Three time steps
[[20 25 45]  Two variables for the two input time series
[30 35 65]
[40 45 85]] [ 50 55 105]
[[ 30 35 65]
[ 40 45 85]
[ 50 55 105]] [ 60 65 125]
[[ 40 45 85]
[ 50 55 105]
[ 60 65 125]] [ 70 75 145]
[[ 50 55 105]
[ 60 65 125]
[ 70 75 145]] [ 80 85 165]
[[ 60 65 125]
[ 70 75 145]
[ 80 85 165]] [ 90 95 185]

Workshop on Deep Learning Time Series by [email protected] July 2022 476

 4. Multivariate Multi-step CNN Models
4.1. Multiple Input and Multi-step Output
◦ Example of a CNN model for multivariate dependent time series with multi-step forecasts
# multivariate multi-step 1d cnn example
from numpy import array
from numpy import hstack
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers.convolutional import Conv1D
from keras.layers.convolutional import MaxPooling1D

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out-1
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 477

 4. Multivariate Multi-step CNN Models
4.1. Multiple Input and Multi-step Output
# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=2000, verbose=0)

# demonstrate prediction
x_input = array([[70, 75], [80, 85], [90, 95]])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 478

 4. Multivariate Multi-step CNN Models
4.1. Multiple Input and Multi-step Output
# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=2000, verbose=0)

# demonstrate prediction
x_input = array([[70, 75], [80, 85], [90, 95]])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 479

 4. Multivariate Multi-step CNN Models
4.1. Multiple Input and Multi-step Output
◦ Running the example, fits the model and predicts the next two time steps of the output sequence beyond the
dataset
◦ Expect the next two steps to be [185, 205]

[[185.5557 208.05902]]

Workshop on Deep Learning Time Series by [email protected] July 2022 480

 4. Multivariate Multi-step CNN Models
4.2. Multiple Parallel Input and Multi-step Output
◦ A problem with parallel time series may require the prediction of multiple time steps of each time series

X: The last three time steps from each of the three time series
[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125] y: The next time steps of each of the three time series
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 481

 4. Multivariate Multi-step CNN Models
4.2. Multiple Parallel Input and Multi-step Output
◦ The split_sequences() function transform a multivariate time series into samples for multi-step forecasting
# multivariate multi-step data preparation
from numpy import array
from numpy import hstack

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 482

 4. Multivariate Multi-step CNN Models
4.2. Multiple Parallel Input and Multi-step Output
◦ Example preparing a multivariate parallel time series with multi-step forecasts
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# convert into input/output

X, y = split_sequences(dataset, n_steps_in, n_steps_out)
print(X.shape, y.shape)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 483

 4. Multivariate Multi-step CNN Models
4.2. Multiple Parallel Input and Multi-step Output
◦ Running the example, prints the shape of the prepared training dataset
X & Y: The input and output are three dimensional
 The number of samples
(5, 3, 3) (5, 2, 3)
[[10 15 25]  Time steps
[20 25 45]  Variables or parallel time series
[30 35 65]] [[ 40 45 85]
[ 50 55 105]]
[[20 25 45]
[30 35 65]
[40 45 85]] [[ 50 55 105]
[ 60 65 125]]
[[ 30 35 65]
[ 40 45 85]
[ 50 55 105]] [[ 60 65 125]
[ 70 75 145]]
[[ 40 45 85]
[ 50 55 105]
[ 60 65 125]] [[ 70 75 145]
[ 80 85 165]]
[[ 50 55 105]
[ 60 65 125]
[ 70 75 145]] [[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 484

 4. Multivariate Multi-step CNN Models
4.2. Multiple Parallel Input and Multi-step Output
◦ Develop a 1D CNN model using a vector-output model
◦ Flatten the three-dimensional structure of the output of each sample in order to train the model
◦ The model is trained on and expected to predict a vector of six numbers directly

# flatten output
n_output = y.shape[1] * y.shape[2]
y = y.reshape((y.shape[0], n_output))

Workshop on Deep Learning Time Series by [email protected] July 2022 485

 4. Multivariate Multi-step CNN Models
4.2. Multiple Parallel Input and Multi-step Output
◦ Example of a CNN model for multivariate parallel time series with multi-step forecasts
# multivariate output multi-step 1d cnn example
from numpy import array
from numpy import hstack
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers.convolutional import Conv1D
from keras.layers.convolutional import MaxPooling1D

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 486

 4. Multivariate Multi-step CNN Models
4.2. Multiple Parallel Input and Multi-step Output
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# convert into input/output

X, y = split_sequences(dataset, n_steps_in, n_steps_out)

# flatten output
n_output = y.shape[1] * y.shape[2]
y = y.reshape((y.shape[0], n_output))

Workshop on Deep Learning Time Series by [email protected] July 2022 487

 4. Multivariate Multi-step CNN Models
4.2. Multiple Parallel Input and Multi-step Output
# the dataset knows the number of features, e.g. 2
n_features = X.shape[2]

# define model
model = Sequential()
model.add(Conv1D(64, 2, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(MaxPooling1D())
model.add(Flatten())
model.add(Dense(50, activation='relu'))
model.add(Dense(n_output))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=7000, verbose=0)

# demonstrate prediction
x_input = array([[60, 65, 125], [70, 75, 145], [80, 85, 165]])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 488

 4. Multivariate Multi-step CNN Models
4.2. Multiple Parallel Input and Multi-step Output
◦ Running the example fits the model and predicts the values for each of the three time steps for the next two time
steps beyond the end of the dataset
◦ Expect the values for these series and time steps
90, 95, 185
100, 105, 205

◦ The model forecast gets reasonably close to the expected values

[[ 91.51622 97.380585 188.90419 102.8417 108.48057 211.89542 ]]

Workshop on Deep Learning Time Series by [email protected] July 2022 489

Develop LSTMs for Time
Series Forecasting
Badan Meteorologi, Klimatologi, dan Geofisika
Juli – Agustus 2022
 Develop LSTMs for Time Series Forecasting
◦ Types of LSTM models that can be used for each specific type of time series forecasting problem
◦ How to develop LSTM models for univariate time series forecasting
◦ How to develop LSTM models for multivariate time series forecasting
◦ How to develop LSTM models for multi-step time series forecasting

Workshop on Deep Learning Time Series by [email protected] July 2022 491

 Different Types of LSTM Models

1. Univariate LSTM Models 2. Multivariate LSTM Models

• 1.1. Vanilla LSTM • 2.1. Multiple Input Series.
• 1.2. Stacked LSTM • 2.2. Multiple Parallel Series.
• 1.3. Bidirectional LSTM
• 1.4. CNN-LSTM
• 1.5. ConvLSTM

3. Multi-step LSTM Models 4. Multivariate Multi-step LSTM Models

• 3.1. Vector Output Model • 4.1. Multiple Input Multi-step Output.
• 3.2. Encoder-Decoder Model • 4.2. Multiple Parallel Input and Multi-step Output.

Workshop on Deep Learning Time Series by [email protected] July 2022 492

 LSTM Models
◦ LSTM is an artificial neural network used in the fields of artificial intelligence and deep learning
◦ Unlike standard feedforward neural networks, LSTM has feedback connections
◦ Such a recurrent neural network (RNN) can process not only single data points (such as images), but also entire sequences
of data (such as speech or video)

Workshop on Deep Learning Time Series by [email protected] July 2022 493

 LSTM Models
◦ The name of LSTM refers to the analogy that a standard RNN has both "long-term memory" and "short-term
memory“
◦ The connection weights and biases in the network change once per episode of training, analogous to how physiological
changes in synaptic strengths store long-term memories
◦ The activation patterns in the network change once per time-step, analogous to how the moment-to-moment change in
electric firing patterns in the brain store short-term memories

◦ The LSTM architecture aims to provide a short-term memory for RNN that can last thousands of timesteps, thus
"long short-term memory"

Workshop on Deep Learning Time Series by [email protected] July 2022 494

 LSTM Models
◦ A common LSTM unit is composed
◦ A cell
◦ An input gate
◦ An output gate
◦ A forget gate

◦ The cell remembers values over arbitrary time intervals

◦ The three gates regulate the flow of information into and out of the cell

Workshop on Deep Learning Time Series by [email protected] July 2022 495

 LSTM Models
◦ LSTM networks are well-suited to classifying, processing and making predictions based on time series data
◦ There can be lags of unknown duration between important events in a time series
◦ LSTMs were developed to deal with the vanishing gradient problem that can be encountered when training traditional RNNs
◦ Relative insensitivity to gap length is an advantage of LSTM over RNNs, hidden Markov models and other sequence
learning methods in numerous applications

Workshop on Deep Learning Time Series by [email protected] July 2022 496

Univariate LSTM Models
Develop LSTMs for Time Series Forecasting
 1. Univariate LSTM Models
Data Preparation
◦ The LSTM model learn a function that maps a sequence of past observations as input to an output observation
◦ The sequence of observations must be transformed into multiple examples
◦ Consider a given univariate sequence
◦ Divide the sequence into multiple input/output patterns called sample

X: Three input time steps

[10, 20, 30, 40, 50, 60, 70, 80, 90]

y: One output time step

X, y
10, 20, 30, 40
20, 30, 40, 50
30, 40, 50, 60
...

Workshop on Deep Learning Time Series by [email protected] July 2022 498

 1. Univariate LSTM Models
Data Preparation
◦ The split_sequence() function split a given univariate sequence into multiple samples
◦ Each sample has a specified number of time steps and the output is a single time step
# univariate data preparation
from numpy import array

# split a univariate sequence into samples

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 499

 1. Univariate LSTM Models
Data Preparation
◦ Demonstrate the function on the small contrived dataset
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps = 3

# split into samples

X, y = split_sequence(raw_seq, n_steps)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 500

 1. Univariate LSTM Models
Data Preparation
◦ Running the example, splits the univariate series into six samples
X: Three input time steps

[10 20 30] 40
[20 30 40] 50
[30 40 50] 60 y: One output time step
[40 50 60] 70
[50 60 70] 80
[60 70 80] 90

Workshop on Deep Learning Time Series by [email protected] July 2022 501

 1. Univariate LSTM Models
1.1. Vanilla LSTM
◦ A Vanilla LSTM has
◦ A single hidden layer of LSTM units
◦ Single output layer

Workshop on Deep Learning Time Series by [email protected] July 2022 502

 1. Univariate LSTM Models
1.1. Vanilla LSTM
◦ Key to LSTMs is that they offer native support for sequences
◦ Unlike a CNN that reads across the entire input vector
◦ The LSTM model reads one time step of the sequence at a time
◦ The LSTM model builds up an internal state representation that can be used as a learned context for making a prediction
# define model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

◦ Key in the definition is the shape of the input

◦ What is the model expects as input for each sample in terms of the number of time steps and the number of features
◦ Working with a univariate series, the number of features is one, for one variable
◦ The number of time steps as input is the number chose when preparing our dataset as an argument to the
split_sequence() function

Workshop on Deep Learning Time Series by [email protected] July 2022 503

 1. Univariate LSTM Models
1.1. Vanilla LSTM
◦ The shape of the input for each sample is specified in the input shape argument on the definition of first hidden
layer
◦ Almost always have multiple samples
◦ The model expect the input component of training data to have the dimensions or shape: [samples, timesteps,
features]
◦ The split_sequence() function in the previous section outputs the X with the shape [samples, timesteps]

◦ Reshape to have an additional dimension for the one feature

# reshape from [samples, timesteps] into [samples, timesteps, features]
n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

Workshop on Deep Learning Time Series by [email protected] July 2022 504

 1. Univariate LSTM Models
1.1. Vanilla LSTM
◦ Define a model with 50 LSTM units in the hidden layer and single output layer
◦ Predicts a single numerical value
◦ The model is fit using the eficient Adam version of stochastic gradient descent and optimized using the mean squared
error loss function

◦ Once the model is defined, can fit it on the training dataset

# fit model
model.fit(X, y, epochs=200, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 505

 1. Univariate LSTM Models
1.1. Vanilla LSTM
◦ After the model is fit, can use it to make a prediction
◦ Can predict the next value in the sequence by providing the input: [70, 80, 90]
◦ Expecting the model to predict something like: [100]

◦ The model expects the input shape to be 3D with [samples, timesteps, features]
◦ Reshape the single input sample before making the prediction
# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 506

 1. Univariate LSTM Models
1.1. Vanilla LSTM
◦ A Vanilla LSTM for univariate time series forecasting and make a single prediction
# univariate lstm example
from numpy import array
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense

# split a univariate sequence into samples

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 507

 1. Univariate LSTM Models
1.1. Vanilla LSTM
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps = 3

# split into samples

X, y = split_sequence(raw_seq, n_steps)

# reshape from [samples, timesteps] into [samples, timesteps, features]

n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

# define model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 508

 1. Univariate LSTM Models
1.1. Vanilla LSTM
# fit model
model.fit(X, y, epochs=200, verbose=0)

# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 509

 1. Univariate LSTM Models
1.1. Vanilla LSTM
◦ Running the example, prepares the data, fits the model, and makes a prediction
[[102.88385]]

Workshop on Deep Learning Time Series by [email protected] July 2022 510

 1. Univariate LSTM Models
1.2. Stacked LSTM
◦ A Stacked LSTM architecture is an LSTM model comprised of multiple LSTM layers
◦ The Stacked LSTM is an extension to vanilla model that has multiple hidden LSTM layers where
each layer contains multiple memory cells
◦ An LSTM layer above provides a sequence output rather than a single value output to the LSTM
layer below
◦ One output per input time step, rather than one output time step for all input time steps

◦ Stacking LSTM hidden layers makes the model deeper, more accurately earning the
description as a deep learning technique

Workshop on Deep Learning Time Series by [email protected] July 2022 511

 1. Univariate LSTM Models
1.2. Stacked LSTM
◦ Multiple hidden LSTM layers can be stacked one on top of another as a Stacked LSTM model
◦ An LSTM layer requires a 3D input
◦ LSTMs by default will produce a 2D output as an interpretation from the end of the sequence
◦ Address this by having the LSTM output a value for each time step in the input data by setting the
return_sequences=True argument on the layer

◦ Have 3D output from hidden LSTM layer as input to the next

# define model
model = Sequential()
model.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))
model.add(LSTM(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 512

 1. Univariate LSTM Models
1.2. Stacked LSTM
◦ A Stacked LSTM for univariate time series forecasting
# univariate stacked lstm example
from numpy import array
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense

# split a univariate sequence

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 513

 1. Univariate LSTM Models
1.2. Stacked LSTM
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps = 3

# split into samples

X, y = split_sequence(raw_seq, n_steps)

# reshape from [samples, timesteps] into [samples, timesteps, features]

n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

# define model
model = Sequential()
model.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))
model.add(LSTM(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 514

 1. Univariate LSTM Models
1.2. Stacked LSTM
# fit model
model.fit(X, y, epochs=200, verbose=0)

# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 515

 1. Univariate LSTM Models
1.2. Stacked LSTM
◦ Running the example, predicts the next value in the sequence, which we expect would be 100
[[102.80462]]

Workshop on Deep Learning Time Series by [email protected] July 2022 516

 1. Univariate LSTM Models
1.3. Bidirectional LSTM
◦ On some sequence prediction problems
◦ It can be beneficial to allow the LSTM model to learn the input sequence both forward and backwards and concatenate
both interpretations
◦ Implement a Bidirectional LSTM for univariate time series forecasting by wrapping the first hidden layer in a wrapper layer
called Bidirectional

Workshop on Deep Learning Time Series by [email protected] July 2022 517

 1. Univariate LSTM Models
1.3. Bidirectional LSTM
◦ An example of defining a Bidirectional LSTM to read input both forward and backward
# define model
model = Sequential()
model.add(Bidirectional(LSTM(50, activation='relu'), input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 518

 1. Univariate LSTM Models
1.3. Bidirectional LSTM
◦ A Bidirectional LSTM for univariate time series forecasting
# univariate bidirectional lstm example
from numpy import array
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense
from keras.layers import Bidirectional

# split a univariate sequence

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 519

 1. Univariate LSTM Models
1.3. Bidirectional LSTM
◦ A Bidirectional LSTM for univariate time series forecasting
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps = 3

# split into samples

X, y = split_sequence(raw_seq, n_steps)

# reshape from [samples, timesteps] into [samples, timesteps, features]

n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

# define model
model = Sequential()
model.add(Bidirectional(LSTM(50, activation='relu'), input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 520

 1. Univariate LSTM Models
1.3. Bidirectional LSTM
◦ A Bidirectional LSTM for univariate time series forecasting
# fit model
model.fit(X, y, epochs=200, verbose=0)

# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 521

 1. Univariate LSTM Models
1.3. Bidirectional LSTM
◦ Running the example predicts the next value in the sequence, which we expect would be 100
[[101.8872]]

Workshop on Deep Learning Time Series by [email protected] July 2022 522

 1. Univariate LSTM Models
1.4. CNN-LSTM
◦ A CNN is a type of neural network developed for working with 2D image data
◦ The CNN can be very effective at automatically extracting and learning features from 1D sequence data such as univariate
time series data

◦ A CNN model can be used in a hybrid model with an LSTM backend

◦ The CNN is used to interpret subsequences of input that together are provided as a sequence to an LSTM model to
interpret
◦ This hybrid model is called a CNN-LSTM

Workshop on Deep Learning Time Series by [email protected] July 2022 523

 1. Univariate LSTM Models
1.4. CNN-LSTM

Workshop on Deep Learning Time Series by [email protected] July 2022 524

 1. Univariate LSTM Models
1.4. CNN-LSTM
◦ The first step is to split the input sequences into subsequences that can be processed by the CNN model
◦ Split the univariate time series data into input/output samples with four steps as input and one as output
◦ Each sample can then be split into two sub-samples, each with two time steps

◦ The CNN can interpret each subsequence of two time steps and provide a time series of interpretations of the
subsequences to the LSTM model to process as input
◦ Parameterize this and define the number of subsequences as n_seq and the number of time steps per subsequence as n
steps.
◦ The input data can then be reshaped to have the required structure: [samples, subsequences, timesteps,
features]

Workshop on Deep Learning Time Series by [email protected] July 2022 525

 1. Univariate LSTM Models
1.4. CNN-LSTM
◦ Reshaping data for a CNN-LSTM model
# choose a number of time steps
n_steps = 4
# split into samples
X, y = split_sequence(raw_seq, n_steps)
# reshape from [samples, timesteps] into [samples, subsequences, timesteps, features]
n_features = 1
n_seq = 2
n_steps = 2
X = X.reshape((X.shape[0], n_seq, n_steps, n_features))

Workshop on Deep Learning Time Series by [email protected] July 2022 526

 1. Univariate LSTM Models
1.4. CNN-LSTM
◦ Reuse the same CNN model when reading in each sub-sequence of data separately
◦ Achieved by wrapping the entire CNN model in a TimeDistributed wrapper that will apply the entire model once per input, in
this case, once per input subsequence
◦ The CNN model has a convolutional layer for reading across the subsequence that requires a number of filters and a kernel
size to be specified
◦ The number of filters is the number of reads or interpretations of the input sequence
◦ The kernel size is the number of time steps included of each read operation of the input sequence
◦ The convolution layer is followed by a max pooling layer that distills the filter maps down to ¼ of their size that includes
the most salient features
◦ These structures are then flattened down to a single one-dimensional vector to be used as a single input time step to the
LSTM layer
# define the input cnn model
model.add(TimeDistributed(Conv1D(64, 1, activation='relu'), input_shape=(None, n_steps, n_features)))
model.add(TimeDistributed(MaxPooling1D()))
model.add(TimeDistributed(Flatten()))

Workshop on Deep Learning Time Series by [email protected] July 2022 527

 1. Univariate LSTM Models
1.4. CNN-LSTM
◦ Define the LSTM part of the model that interprets the CNN model's read of the input sequence and makes a
prediction

# define the output model

model.add(LSTM(50, activation='relu'))
model.add(Dense(1))

Workshop on Deep Learning Time Series by [email protected] July 2022 528

 1. Univariate LSTM Models
1.4. CNN-LSTM
◦ A CNN-LSTM for univariate time series forecasting
# univariate cnn lstm example
from numpy import array
from keras.models import Sequential
from keras.layers import LSTM, Dense, Flatten, TimeDistributed
from keras.layers.convolutional import Conv1D, MaxPooling1D

# split a univariate sequence into samples

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 529

 1. Univariate LSTM Models
1.4. CNN-LSTM
◦ A CNN-LSTM for univariate time series forecasting
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps = 4

# split into samples

X, y = split_sequence(raw_seq, n_steps)

# reshape from [samples, timesteps] into [samples, subsequences, timesteps, features]

n_features = 1
n_seq = 2
n_steps = 2
X = X.reshape((X.shape[0], n_seq, n_steps, n_features))

Workshop on Deep Learning Time Series by [email protected] July 2022 530

 1. Univariate LSTM Models
1.4. CNN-LSTM
◦ A CNN-LSTM for univariate time series forecasting
# define model
model = Sequential()
model.add(TimeDistributed(Conv1D(64, 1, activation='relu'), input_shape=(None, n_steps, n_features)))
model.add(TimeDistributed(MaxPooling1D()))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=500, verbose=0)

# demonstrate prediction
x_input = array([60, 70, 80, 90])
x_input = x_input.reshape((1, n_seq, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 531

 1. Univariate LSTM Models
1.4. CNN-LSTM
◦ Running the example predicts the next value in the sequence, which we expect would be 100
[[100.68751]]

Workshop on Deep Learning Time Series by [email protected] July 2022 532

 1. Univariate LSTM Models
1.5. ConvLSTM
◦ The ConvLSTM
◦ The convolutional reading of input is built directly into each LSTM unit
◦ Developed for reading 2D spatial-temporal data, but can be adapted for use with univariate time series forecasting
◦ The layer expects input as a sequence of 2D images
◦ The shape of input data must be: [samples, timesteps, rows, columns, features]

Workshop on Deep Learning Time Series by [email protected] July 2022 533

 1. Univariate LSTM Models
1.5. ConvLSTM

Workshop on Deep Learning Time Series by [email protected] July 2022 534

 1. Univariate LSTM Models
1.5. ConvLSTM
◦ Split each sample into subsequences
◦ Timesteps become the number of subsequences, or n_seq,
◦ Columns be the number of time steps for each subsequence, or n steps
◦ The number of rows is fixed at 1 as working with 1D data
◦ Reshape the prepared samples into the required structure
# choose a number of time steps
n_steps = 4

# split into samples

X, y = split_sequence(raw_seq, n_steps)

# reshape from [samples, timesteps] into [samples, timesteps, rows, columns, features]
n_features = 1
n_seq = 2
n_steps = 2
X = X.reshape((X.shape[0], n_seq, 1, n_steps, n_features))

Workshop on Deep Learning Time Series by [email protected] July 2022 535

 1. Univariate LSTM Models
1.5. ConvLSTM
◦ Define the ConvLSTM
◦ A single layer in terms of the number of filters
◦ A two-dimensional kernel size in terms of (rows, columns)
◦ The number of rows is always fixed to 1 in the kernel
◦ The output of the model be flattened before interpreting and a prediction made

# define the input cnnlstm model

model.add(ConvLSTM2D(64, (1,2), activation='relu', input_shape=(n_seq, 1, n_steps, n_features)))
model.add(Flatten())

Workshop on Deep Learning Time Series by [email protected] July 2022 536

 1. Univariate LSTM Models
1.5. ConvLSTM
◦ A ConvLSTM for univariate time series forecasting
# univariate convlstm example
from numpy import array
from keras.models import Sequential
from keras.layers import Dense, Flatten, ConvLSTM2D

# split a univariate sequence into samples

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 537

 1. Univariate LSTM Models
1.5. ConvLSTM
◦ A ConvLSTM for univariate time series forecasting
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps = 4

# split into samples

X, y = split_sequence(raw_seq, n_steps)

# reshape from [samples, timesteps] into [samples, timesteps, rows, columns, features]
n_features = 1
n_seq = 2
n_steps = 2
X = X.reshape((X.shape[0], n_seq, 1, n_steps, n_features))

Workshop on Deep Learning Time Series by [email protected] July 2022 538

 1. Univariate LSTM Models
1.5. ConvLSTM
◦ A ConvLSTM for univariate time series forecasting
# define model
model = Sequential()
model.add(ConvLSTM2D(64, (1,2), activation='relu', input_shape=(n_seq, 1, n_steps, n_features)))
model.add(Flatten())
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=500, verbose=0)

# demonstrate prediction
x_input = array([60, 70, 80, 90])
x_input = x_input.reshape((1, n_seq, 1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 539

 1. Univariate LSTM Models
1.5. ConvLSTM
◦ Running the example predicts the next value in the sequence, which we expect would be 100
[[102.88765]]

Workshop on Deep Learning Time Series by [email protected] July 2022 540

Multivariate LSTM Models
Develop LSTMs for Time Series Forecasting
 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ A problem may have two or more parallel input time series and an output time series
◦ Dependent on the input time series
◦ The input time series are parallel because each series has an observation at the same time steps

◦ A simple example of two parallel input time series where the output series is the simple addition of the input
series
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

Workshop on Deep Learning Time Series by [email protected] July 2022 542

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ Reshape these three arrays of data as a single dataset
◦ Each row is a time step
◦ Each column is a separate time series

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

Workshop on Deep Learning Time Series by [email protected] July 2022 543

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ Defining a dependent series dataset
# multivariate data preparation
from numpy import array
from numpy import hstack

# define input sequence

in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))
print(dataset)

Workshop on Deep Learning Time Series by [email protected] July 2022 544

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ Running the example, prints the dataset
◦ One row per time step
◦ One column for each of the two input
◦ One output parallel time series
[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125]
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 545

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ As with the univariate time series
◦ Must structure these data into samples with input and output elements
◦ An LSTM model needs sufficient context to learn a mapping from an input sequence to an output value
◦ LSTMs can support parallel input time series as separate variables or features
◦ Need to split the data into samples maintaining the order of observations across the two input sequences
X: Three input time steps

[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105] y: One output time step
[ 60 65 125]
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 546

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ A function named split_sequences() take a dataset with rows for time steps and columns for parallel series
and return input/output samples
# multivariate data preparation
from numpy import array, hstack

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 547

 2. Multivariate LSTM Models
2.1. Multiple Input Series
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3

# convert into input/output

X, y = split_sequences(dataset, n_steps)
print(X.shape, y.shape)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])
Workshop on Deep Learning Time Series by [email protected] July 2022 548
 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ Running the example, prints the shape of the X and y components
The X component has a three-dimensional structure
 The first dimension is the number of samples
(7, 3, 2) (7,)  The second dimension is the number of time steps per sample
 The last dimension specfies the number of parallel time series or the number of variables
[[10 15]
[20 25]
[30 35]] 65
[[20 25]
[30 35]
[40 45]] 85
[[30 35]
[40 45] The y is output three time step
[50 55]] 105
...
[[70 75]
[80 85]
[90 95]] 185

Workshop on Deep Learning Time Series by [email protected] July 2022 549

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ Fit an LSTM model on this data
◦ Any of the varieties of LSTMs such as a Vanilla, Stacked, Bidirectional, CNN, or ConvLSTM model

◦ Use a Vanilla LSTM

◦ The number of time steps and parallel series (features) are specified for the input layer via the input shape argument
# define model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 550

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ When making a prediction
◦ The model expects three time steps for two input time series
◦ Predict the next value in the output series providing the input values
80, 85
90, 95
100, 105

◦ The shape of the one sample with three time steps and two variables must be [1, 3, 2]
# demonstrate prediction
x_input = array([[80, 85], [90, 95], [100, 105]])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)

◦ Expect the next value in the sequence to be 100 + 105, or 205

Workshop on Deep Learning Time Series by [email protected] July 2022 551

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ A Vanilla LSTM for multivariate dependent time series forecasting
# multivariate lstm example
from numpy import array
from numpy import hstack
from keras.models import Sequential
from keras.layers import LSTM, Dense

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 552

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ A Vanilla LSTM for multivariate dependent time series forecasting
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3

# convert into input/output

X, y = split_sequences(dataset, n_steps)

# the dataset knows the number of features, e.g. 2

n_features = X.shape[2]

Workshop on Deep Learning Time Series by [email protected] July 2022 553

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ A Vanilla LSTM for multivariate dependent time series forecasting
# define model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=200, verbose=0)

# demonstrate prediction
x_input = array([[80, 85], [90, 95], [100, 105]])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 554

 2. Multivariate LSTM Models
2.1. Multiple Input Series
◦ Running the example prepares the data, fits the model, and makes a prediction
[[206.12552]]

Workshop on Deep Learning Time Series by [email protected] July 2022 555

 2. Multivariate LSTM Models
2.2. Multiple Parallel Series
◦ There are multiple parallel time series and a value must be predicted for each

X: Three input time steps

[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125] y: One output time step
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 556

 2. Multivariate LSTM Models
2.2. Multiple Parallel Series
◦ The split_sequences() function split multiple parallel time series with rows for time steps and one series
per column into the required input/output shape
# multivariate output data prep
from numpy import array
from numpy import hstack

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 557

 2. Multivariate LSTM Models
2.2. Multiple Parallel Series
◦ Splitting a multivariate parallel time series onto samples
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
# convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))
# horizontally stack columns
dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3
# convert into input/output
X, y = split_sequences(dataset, n_steps)
print(X.shape, y.shape)
# summarize the data
for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 558

 2. Multivariate LSTM Models
2.2. Multiple Parallel Series
◦ Running the example, prints the shape of the prepared X and y components
The X component has a three-dimensional structure
 The number of samples
(6, 3, 3) (6, 3)  The number of time steps per sample
 The number of parallel time series or features
[[10 15 25]
[20 25 45] The shape of y is two-dimensional
[30 35 65]] [40 45 85]  The number of samples
[[20 25 45]  The number of time variables per sample to be predicted
[30 35 65]
[40 45 85]] [ 50 55 105]
...
[[ 60 65 125]
[ 70 75 145]
[ 80 85 165]] [ 90 95 185]

Workshop on Deep Learning Time Series by [email protected] July 2022 559

 2. Multivariate LSTM Models
2.2. Multiple Parallel Series
◦ Ready to fit an LSTM model
◦ Any of the varieties of LSTMs such as a Vanilla, Stacked, Bidirectional, CNN, or ConvLSTM model
◦ Use a Stacked LSTM where the number of time steps and parallel series (features) are specified for the input layer via the
input shape argument
◦ The number of parallel series is also used in the specification of the number of values to predict by the model in the output
layer
# define model
model = Sequential()
model.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))
model.add(LSTM(100, activation='relu'))
model.add(Dense(n_features))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 560

 2. Multivariate LSTM Models
2.2. Multiple Parallel Series
◦ Predict the next value in each of the three parallel series by providing an input of three time steps for each series
70, 75, 145
80, 85, 165
90, 95, 185

◦ The shape of the input for making a single prediction must be 1 sample, 3 time steps, and 3 features, or [1, 3, 3]
# demonstrate prediction
x_input = array([[70,75,145], [80,85,165], [90,95,185]])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)

◦ Expect the vector output to be

[100, 105, 205]

Workshop on Deep Learning Time Series by [email protected] July 2022 561

 2. Multivariate LSTM Models
2.2. Multiple Parallel Series
◦ A Stacked LSTM for multivariate output time series forecasting
# multivariate output stacked lstm example
from numpy import array, hstack
from keras.models import Sequential
from keras.layers import LSTM, Dense

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 562

 2. Multivariate LSTM Models
2.2. Multiple Parallel Series
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps = 3

# convert into input/output

X, y = split_sequences(dataset, n_steps)

# the dataset knows the number of features, e.g. 2

n_features = X.shape[2]

Workshop on Deep Learning Time Series by [email protected] July 2022 563

 2. Multivariate LSTM Models
2.2. Multiple Parallel Series
# define model
model = Sequential()
model.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))
model.add(LSTM(100, activation='relu'))
model.add(Dense(n_features))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=400, verbose=0)

# demonstrate prediction
x_input = array([[70,75,145], [80,85,165], [90,95,185]])
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 564

 2. Multivariate LSTM Models
2.2. Multiple Parallel Series
◦ Running the example, prepares the data, fits the model, and makes a prediction
[[100.53121 106.33091 205.00043]]

Workshop on Deep Learning Time Series by [email protected] July 2022 565

Multi-step LSTM Models
Develop LSTMs for Time Series Forecasting
 3. Multi-step LSTM Models
3.1. Vector Output Model - Data Preparation
◦ As with one-step forecasting
◦ A time series used for multi-step time series forecasting must be split into samples with input and output components
◦ Both the input and output components be comprised of multiple time steps and may or may not have the same number of
steps
X: The last three time steps as input

[10, 20, 30, 40, 50, 60, 70, 80, 90]

y: The next two time steps

Workshop on Deep Learning Time Series by [email protected] July 2022 567

 3. Multi-step LSTM Models
3.1. Vector Output Model - Data Preparation
◦ The split_sequence() function split a given univariate time series into samples with a specified number of
input and output time steps
# multi-step data preparation
from numpy import array

# split a univariate sequence into samples

def split_sequence(sequence, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the sequence
if out_end_ix > len(sequence):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 568

 3. Multi-step LSTM Models
3.1. Vector Output Model - Data Preparation
◦ Splitting a univariate series for multi-step forecasting into samples
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# split into samples

X, y = split_sequence(raw_seq, n_steps_in, n_steps_out)

# summarize the data

for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 569

 3. Multi-step LSTM Models
3.1. Vector Output Model - Data Preparation
◦ Running the example
◦ Splits the univariate series into input and output time steps
◦ Prints the input and output components of each

[10 20 30] [40 50]

[20 30 40] [50 60]
[30 40 50] [60 70]
[40 50 60] [70 80]
[50 60 70] [80 90]

Workshop on Deep Learning Time Series by [email protected] July 2022 570

 3. Multi-step LSTM Models
3.1. Vector Output Model
◦ The LSTM can output a vector directly that can be interpreted as a multi-step forecast
◦ One time step of each output time series was forecasted as a vector
◦ As with the LSTMs for univariate data in a prior section, the prepared samples must first be reshaped
◦ The LSTM expects data to have a three-dimensional structure of [samples, timesteps, features]
◦ Only have one feature so the reshape is straightforward
# reshape from [samples, timesteps] into [samples, timesteps, features]
n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

Workshop on Deep Learning Time Series by [email protected] July 2022 571

 3. Multi-step LSTM Models
3.1. Vector Output Model
◦ With the number of input and output steps specified in the n_steps_in and n_steps_out variables
◦ Define a multi-step time-series forecasting model
◦ Any of the presented LSTM model types could be used, such as Vanilla, Stacked, Bidirectional, CNN-LSTM, or ConvLSTM

# define model
model = Sequential()
model.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps_in, n_features)))
model.add(LSTM(100, activation='relu'))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 572

 3. Multi-step LSTM Models
3.1. Vector Output Model
◦ The model can make a prediction for a single sample
◦ Predict the next two steps beyond the end of the dataset by providing the input
[70, 80, 90]

◦ Expect the predicted output to be

[100, 110]

◦ The shape of the single sample of input data when making the prediction must be [1, 3, 1] for the 1 sample, 3
time steps of the input, and the single feature
# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)

Workshop on Deep Learning Time Series by [email protected] July 2022 573

 3. Multi-step LSTM Models
3.1. Vector Output Model
◦ The Stacked LSTM for multi-step forecasting with a univariate time series
# univariate multi-step vector-output stacked lstm example
from numpy import array
from keras.models import Sequential
from keras.layers import LSTM, Dense

# split a univariate sequence into samples

def split_sequence(sequence, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the sequence
if out_end_ix > len(sequence):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 574

 3. Multi-step LSTM Models
3.1. Vector Output Model
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# split into samples

X, y = split_sequence(raw_seq, n_steps_in, n_steps_out)

# reshape from [samples, timesteps] into [samples, timesteps, features]

n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

# define model
model = Sequential()
model.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps_in, n_features)))
model.add(LSTM(100, activation='relu'))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 575

 3. Multi-step LSTM Models
3.1. Vector Output Model
# fit model
model.fit(X, y, epochs=50, verbose=0)

# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 576

 3. Multi-step LSTM Models
3.1. Vector Output Model
◦ Running the example forecasts and prints the next two time steps in the sequence
[[118.37873 135.12431]]

Workshop on Deep Learning Time Series by [email protected] July 2022 577

 3. Multi-step LSTM Models
3.2. Encoder-Decoder Model
◦ Encoder-Decoder LSTM: A model specifically developed for forecasting variable length output sequences
◦ The model was designed for prediction problems
◦ There are both input and output sequences, so-called sequence-to-sequence problems, such as translating text from one
language to another
◦ This model can be used for multi-step time series forecasting
◦ The model is comprised of two sub-models: the encoder and the decoder

Workshop on Deep Learning Time Series by [email protected] July 2022 578

 3. Multi-step LSTM Models
3.2. Encoder-Decoder Model

Workshop on Deep Learning Time Series by [email protected] July 2022 579

 3. Multi-step LSTM Models
3.2. Encoder-Decoder Model
◦ The encoder is a model responsible for reading and interpreting the input sequence
◦ The output of the encoder is a fixed length vector that represents the model's interpretation of the sequence
◦ The encoder is traditionally a Vanilla LSTM model
◦ Other encoder models can be used such as Stacked, Bidirectional, and CNN models

# define encoder model

model.add(LSTM(100, activation='relu', input_shape=(n_steps_in, n_features)))

Workshop on Deep Learning Time Series by [email protected] July 2022 580

 3. Multi-step LSTM Models
3.2. Encoder-Decoder Model
◦ The decoder uses the output of the encoder as an input
◦ The fixed-length output of the encoder is repeated, once for each required time step in the output sequence
# repeat encoding
model.add(RepeatVector(n_steps_out))

◦ This sequence is then provided to an LSTM decoder model

◦ The model must output a value for each value in the output time step, which can be interpreted by a single output model
# define decoder model
model.add(LSTM(100, activation='relu', return_sequences=True))

◦ Use the same output layer or layers to make each one-step prediction in the output sequence
◦ Achieved by wrapping the output part of the model in a TimeDistributed wrapper
# define model output
model.add(TimeDistributed(Dense(1)))

Workshop on Deep Learning Time Series by [email protected] July 2022 581

 3. Multi-step LSTM Models
3.2. Encoder-Decoder Model
◦ The full definition for an Encoder-Decoder model for multi-step time series forecasting
# define model
model = Sequential()
model.add(LSTM(100, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(RepeatVector(n_steps_out))
model.add(LSTM(100, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.compile(optimizer='adam', loss='mse')

◦ The input data must be reshaped into the expected three-dimensional shape of [samples, timesteps, features]\
# reshape input training data
X = X.reshape((X.shape[0], X.shape[1], n_features))

◦ the output must also have this shape

◦ Predict a given number of time steps with a given number of features for each input sample
# reshape output training data
y = y.reshape((y.shape[0], y.shape[1], n_features))

Workshop on Deep Learning Time Series by [email protected] July 2022 582

 3. Multi-step LSTM Models
3.2. Encoder-Decoder Model
◦ An Encoder-Decoder LSTM for multi-step time series forecasting
# univariate multi-step encoder-decoder lstm example
from numpy import array
from keras.models import Sequential
from keras.layers import LSTM, Dense, RepeatVector, TimeDistributed

# split a univariate sequence into samples

def split_sequence(sequence, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the sequence
if out_end_ix > len(sequence):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 583

 3. Multi-step LSTM Models
3.2. Encoder-Decoder Model
◦ An Encoder-Decoder LSTM for multi-step time series forecasting
# define input sequence
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# split into samples

X, y = split_sequence(raw_seq, n_steps_in, n_steps_out)

# reshape from [samples, timesteps] into [samples, timesteps, features]

n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))
y = y.reshape((y.shape[0], y.shape[1], n_features))

# define model
model = Sequential()
model.add(LSTM(100, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(RepeatVector(n_steps_out))
model.add(LSTM(100, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.compile(optimizer='adam', loss='mse')

Workshop on Deep Learning Time Series by [email protected] July 2022 584

 3. Multi-step LSTM Models
3.2. Encoder-Decoder Model
◦ An Encoder-Decoder LSTM for multi-step time series forecasting
# fit model
model.fit(X, y, epochs=100, verbose=0)

# demonstrate prediction
x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 585

 3. Multi-step LSTM Models
3.2. Encoder-Decoder Model
◦ Running the example, forecasts and prints the next two time steps in the sequence
[[[103.93782]
[117.1069 ]]]

Workshop on Deep Learning Time Series by [email protected] July 2022 586

Multivariate Multi-step
LSTM Models
Develop LSTMs for Time Series Forecasting
 4. Multivariate Multi-step LSTM Models
4.1. Multiple Input Multi-step Output
◦ A multivariate dependent time series
Three prior time steps of each of the two input time series

[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105] Two time steps of the output time series
[ 60 65 125]
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 588

 4. Multivariate Multi-step LSTM Models
4.1. Multiple Input Multi-step Output
◦ The split_sequences() function for splitting a dependent series for multi-step forecasting into samples
# split a multivariate sequence into samples
def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out-1
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 589

 4. Multivariate Multi-step LSTM Models 1/2
4.1. Multiple Input Multi-step Output
◦ Splitting a parallel series for multi-step forecasting into samples
# multivariate multi-step data preparation
from numpy import array
from numpy import hstack

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out-1
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 590

 4. Multivariate Multi-step LSTM Models 2/2
4.1. Multiple Input Multi-step Output
◦ Splitting a parallel series for multi-step forecasting into samples
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))
# choose a number of time steps
n_steps_in, n_steps_out = 3, 2
# covert into input/output
X, y = split_sequences(dataset, n_steps_in, n_steps_out)
print(X.shape, y.shape)
# summarize the data
for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 591

 4. Multivariate Multi-step LSTM Models 2/2
4.1. Multiple Input Multi-step Output
◦ Running the example, prints the shape of the prepared training data
(6, 3, 2) (6, 2)
[[10 15]
[20 25] The output portion of the samples is two-dimensional for the
[30 35]] [65 85] six samples and the two time steps for each sample
[[20 25]
[30 35] The shape of the input portion of the samples is three-dimensional
[40 45]] [ 85 105]  Six samples
[[30 35]  Three time steps
[40 45]  Two variables for the 2 input time series
[50 55]] [105 125]
[[40 45]
[50 55]
[60 65]] [125 145]
[[50 55]
[60 65]
[70 75]] [145 165]
[[60 65]
[70 75]
[80 85]] [165 185]

Workshop on Deep Learning Time Series by [email protected] July 2022 592

 4. Multivariate Multi-step LSTM Models 1/3
4.1. Multiple Input Multi-step Output
◦ An LSTM model for multi-step predictions
# multivariate multi-step stacked lstm example
from numpy import array, hstack
from keras.models import Sequential
from keras.layers import LSTM, Dense

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out-1
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 593

 4. Multivariate Multi-step LSTM Models 2/3
4.1. Multiple Input Multi-step Output
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# covert into input/output

X, y = split_sequences(dataset, n_steps_in, n_steps_out)

# the dataset knows the number of features, e.g. 2

n_features = X.shape[2]

Workshop on Deep Learning Time Series by [email protected] July 2022 594

 4. Multivariate Multi-step LSTM Models 3/3
4.1. Multiple Input Multi-step Output
# define model
model = Sequential()
model.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps_in,
n_features)))
model.add(LSTM(100, activation='relu'))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=200, verbose=0)

# demonstrate prediction
x_input = array([[70, 75], [80, 85], [90, 95]])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 595

 4. Multivariate Multi-step LSTM Models 3/3
4.1. Multiple Input Multi-step Output
◦ Running the model and predicts the next two time steps of the output sequence beyond the dataset
◦ Expect the next two steps to be: [185, 205]
◦ It is a challenging framing of the problem with very little data, and the arbitrarily configured version of the model gets close
[[187.38196 208.30263]]

Workshop on Deep Learning Time Series by [email protected] July 2022 596

 4. Multivariate Multi-step LSTM Models
4.2. Multiple Parallel Input and Multi-step Output
◦ A multivariate parallel time series dataset

The last three time steps from each of the three time series

[[ 10 15 25]
[ 20 25 45]
[ 30 35 65]
[ 40 45 85]
[ 50 55 105]
[ 60 65 125] The next time steps of each of the three time series
[ 70 75 145]
[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 597

4. Multivariate Multi-step LSTM Models
4.2. Multiple Parallel Input and Multi-step Output
◦ The split_sequences() function for splitting a parallel dataset for multi-step forecasting into samples
# split a multivariate sequence into samples
def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

July 2022 598

 4. Multivariate Multi-step LSTM Models
4.2. Multiple Parallel Input and Multi-step Output
◦ Splitting a parallel series for multi-step forecasting into samples
# multivariate multi-step data preparation
from numpy import array
from numpy import hstack

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 599

 4. Multivariate Multi-step LSTM Models
4.2. Multiple Parallel Input and Multi-step Output
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))
# choose a number of time steps
n_steps_in, n_steps_out = 3, 2
# covert into input/output
X, y = split_sequences(dataset, n_steps_in, n_steps_out)
print(X.shape, y.shape)
# summarize the data
for i in range(len(X)):
print(X[i], y[i])

Workshop on Deep Learning Time Series by [email protected] July 2022 600

 4. Multivariate Multi-step LSTM Models
4.2. Multiple Parallel Input and Multi-step Output
◦ Running the example, prints the shape of the prepared training dataset
(5, 3, 3) (5, 2, 3)
[[10 15 25]
[20 25 45] Both the input (X) and output (Y ) elements of the dataset are three dimensional
[30 35 65]] [[ 40 45 85]  Number of samples
[ 50 55 105]]  Time steps
[[20 25 45]  Variables or parallel time series
[30 35 65]
[40 45 85]] [[ 50 55 105]
[ 60 65 125]]
[[ 30 35 65]
[ 40 45 85]
[ 50 55 105]] [[ 60 65 125]
[ 70 75 145]]
[[ 40 45 85]
[ 50 55 105]
[ 60 65 125]] [[ 70 75 145]
[ 80 85 165]]
[[ 50 55 105]
[ 60 65 125]
[ 70 75 145]] [[ 80 85 165]
[ 90 95 185]]

Workshop on Deep Learning Time Series by [email protected] July 2022 601

 4. Multivariate Multi-step LSTM Models 1/3
4.2. Multiple Parallel Input and Multi-step Output
◦ An Encoder-Decoder LSTM for multi-step forecasting for parallel series
# multivariate multi-step encoder-decoder lstm example
from numpy import array, hstack
from keras.models import Sequential
from keras.layers import LSTM, Dense, RepeatVector, TimeDistributed

# split a multivariate sequence into samples

def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
# gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

Workshop on Deep Learning Time Series by [email protected] July 2022 602

 4. Multivariate Multi-step LSTM Models 2/3
4.2. Multiple Parallel Input and Multi-step Output
# define input sequence
in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])

# convert to [rows, columns] structure

in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))

# horizontally stack columns

dataset = hstack((in_seq1, in_seq2, out_seq))

# choose a number of time steps

n_steps_in, n_steps_out = 3, 2

# covert into input/output

X, y = split_sequences(dataset, n_steps_in, n_steps_out)
# the dataset knows the number of features, e.g. 2
n_features = X.shape[2]

Workshop on Deep Learning Time Series by [email protected] July 2022 603

 4. Multivariate Multi-step LSTM Models 3/3
4.2. Multiple Parallel Input and Multi-step Output
# define model
model = Sequential()
model.add(LSTM(200, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(RepeatVector(n_steps_out))
model.add(LSTM(200, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(n_features)))
model.compile(optimizer='adam', loss='mse')

# fit model
model.fit(X, y, epochs=300, verbose=0)

# demonstrate prediction
x_input = array([[60, 65, 125], [70, 75, 145], [80, 85, 165]])
x_input = x_input.reshape((1, n_steps_in, n_features))
yhat = model.predict(x_input, verbose=0)
print(yhat)

Workshop on Deep Learning Time Series by [email protected] July 2022 604

 4. Multivariate Multi-step LSTM Models
4.2. Multiple Parallel Input and Multi-step Output
◦ Running the example, fits the model and predicts the values for each of the three time steps for the next two time
steps beyond the end of the dataset
90, 95, 185
100, 105, 205

◦ Output from an Encoder-Decoder Output LSTM for multi-step forecasting for parallel series
[[[ 90.46589 95.4396 185.87837]
[100.56111 105.4582 206.39198]]]

Workshop on Deep Learning Time Series by [email protected] July 2022 605