Neural Computing and Applications manuscript No.

(will be inserted by the editor)

Multivariate Gaussian and Student−t process regression

for multi-output prediction

Zexun Chen · Bo Wang · Alexander N. Gorban

Received: date / Accepted: date

arXiv:1703.04455v5 [stat.ML] 18 Jan 2018

Abstract Gaussian process for vector-valued function model has been shown to be
a useful method for multi-output prediction. The existing method for this model is
to re-formulate the matrix-variate Gaussian distribution as a multivariate normal
distribution. Although it is effective in many cases, re-formulation is not always
workable and difficult to extend because not all matrix-variate distributions can be
transformed to related multivariate distributions, such as the case for matrix-variate
Student−t distribution. In this paper, we propose a new derivation of multivariate
Gaussian process regression (MV-GPR), where the model settings, derivations and
computations are all directly performed in matrix form, rather than vectorizing the
matrices as done in the existing methods. Furthermore, we introduce the multivari-
ate Student−t process and then derive a new method, multivariate Student−t pro-
cess regression (MV-TPR) for multi-output prediction. Both MV-GPR and MV-TPR
have closed form expressions for the marginal likelihoods and predictive distribu-
tions. The usefulness of the proposed methods are illustrated through several sim-
ulated examples. In particular, we verify empirically that MV-TPR has superiority
for the datasets considered, including air quality prediction and bike rent predic-
tion. At last, the proposed methods are shown to produce profitable investment
strategies in the stock markets.

Keywords multivariate Gaussian process · multivariate Student−t process ·

Gaussian process regression · Student−t process regression · multi-output
prediction · stock investment strategy · industrial sector · time series prediction

Corresponding author: Zexun Chen

Department of Mathematics, University of Leicester, Leicester, LE1 7RH, UK
Tel.: +44-7767940465
E-mail: [email protected]
Bo Wang
Department of Mathematics, University of Leicester, Leicester, LE1 7RH, UK
Alexander N. Gorban
Department of Mathematics, University of Leicester, Leicester, LE1 7RH, UK
2 Zexun Chen et al.

1 Introduction

Over the last few decades, Gaussian processes regression (GPR) has been proven
to be a powerful and effective method for non-linear regression problems due to
many favorable properties, such as simple structure of obtaining and expressing
uncertainty in predictions, the capability of capturing a wide variety of behaviour
by parameters, and a natural Bayesian interpretation [21, 4]. In 1996, Neal [20] re-
vealed that many Bayesian regression models based on neural network converge to
Gaussian processes (GP) in the limit of an infinite number of hidden units [26]. GP
has been suggested as a replacement for supervised neural networks in non-linear
regression [17,27] and classification [17]. Furthermore, GP has excellent capability
for forecasting time series [6,5].
Despite the popularity of GPR in various modelling tasks, there still exists a
conspicuous imperfection, that is, the majority of GPR models are implemented
for single response variables or considered independently for multiple responses
variables without consideration of their correlation [4, 25]. In order to resolve multi-
output prediction problem, Gaussian process regression for vector-valued function
is proposed and regarded as a pragmatic and straightforward method. The core of
this method is to vectorize the multi-response variables and construct a ”big” co-
variance, which describes the correlations between the inputs as well as between the
outputs [4,25,7,2]. This modelling strategy is feasible due to the fact that the matrix-
variate Gaussian distributions can be re-formulated as multivariate Gaussian distri-
butions [7,14]. Intrinsically, Gaussian process regression for vector-valued function
is still a conventional Gaussian process regression model since it merely vectorizes
multi-response variables, which are assumed to follow a developed case of GP with
a reproduced kernel. As an extension, it is natural to consider more general elliptical
processes models for multi-output prediction. However, the vectorization method
cannot be used to extend multi-output GPR because the equivalence between vec-
torized matrix-variate and multivariate distributions only exists in Gaussian cases
[14].
To overcome this drawback, we propose another derivation of dependent Gaus-
sian process regression, named as multivariate Gaussian process regression (MV-
GPR), where the model settings, derivations and computations are all directly per-
formed in matrix form, rather than vectorizing the matrices as done in the existing
methods. MV-GPR is a more straightforward method, and can be implemented in
the same way as the conventional GPR. In fact, our proposed MV-GPR not only a
new derivation of Gaussian process for multi-output, but also a simple framework
to derive more general elliptical processes models for multi-output. Hence, we fur-
ther introduce the multivariate Student−t process and then derive a new consider-
able method, multivariate Student−t process regression (MV-TPR) for multi-output
prediction. The usefulness of proposed methods are illustrated through several sim-
ulated examples. Furthermore, we also verify empirically that MV-TPR has supe-
riority in the prediction based on some widely-used datasets, including air qual-
ity prediction and bike rent prediction. The proposed methods are then applied to
stock market modelling and shown to make the profitable stock investment strate-
gies.
In summary, the main contributions of this paper are concluded in the follow-
ing. (1), We propose a concise and straightforward derivation of dependent Gaus-
sian process regression, MV-GPR. (2), Based on the framework of MV-GPR deriva-
Multivariate Gaussian and Student−t process regression for multi-output prediction 3

tion, we achieve a new method for multi-output prediction, MV-TPR. (3), The ef-
fectiveness of the proposed MV-GPR and MV-TPR are illustrated through several
simulated examples. (4), MV-TPR shows its superiority in the air quality predic-
tion and the bike rent prediction. (5), We apply MV-GPR and MV-TPR to produce
profitable investment strategies in the stock markets.
The paper is organized as follows. Section 2 introduces some preliminaries of
matrix-variate Gaussian and Student−t distributions with their useful properties.
In Section 3 multivariate Gaussian process and multivariate Student−t process are
defined and their regression models are presented. Some numerical experiments by
simulated data, stock market investment, and real datasets are presented in Section
4. Section 5 concludes the paper.

2 Preliminaries

Matrix-variate Gaussian and Student−t distributions have many useful properties,

as discussed in the literatures [14,9,30]. Below we list some of them which are used
in this paper.

2.1 Matrix-variate Gaussian distribution

Definition 1 The random matrix X ∈ Rn×d is said to have a matrix-variate Gaussian

distribution with mean matrix M ∈ Rn×d and covariance matrix Σ ∈ Rn×n , Ω ∈ Rd×d
if and only if its probability density function is given by

dn d n 1
p( X | M, Σ, Ω) = (2π )− det(Σ)− 2 det(Ω)− 2 etr(− Ω−1 ( X − M)T Σ−1 ( X − M )),
2
2
(1)
where etr(·) is exponential of matrix trace, Ω and Σ are positive semi-definite. It is denoted
X ∼ MN n,d ( M, Σ, Ω). Without loss of clarity, it is denoted X ∼ MN ( M, Σ, Ω).

Like multivariate Gaussian distribution, matrix-variate Gaussian distribution also

possesses several important properties as follows.

Theorem 1 (Transposable) If X ∼ MN n,d ( M, Σ, Ω), then X T ∼ MN d,n ( MT , Ω, Σ).

The matrix-variate Gaussian is related to the multivariate Gaussian in the following

way.

Theorem 2 (Vectorizable) X ∼ MN n,d ( M, Σ, Ω) if and only if

vec( X T ) ∼ Nnd (vec( MT ), Σ ⊗ Ω),

where vec(·) is vector operator and ⊗ is Kronecker product (or called tensor product).

Furthermore, the matrix-variate Gaussian distribution is consistent under the marginal-

ization and conditional distribution.
4 Zexun Chen et al.

Theorem 3 (Marginalization and conditional distribution) Let X ∼ MN n,d ( M, Σ, Ω),

and partition X, M, Σ and Ω as
   
X1r n1 M1r n1
X= = [ X1c X2c ] , M= = [ M1c M2c ]
X2r n2 M2r n2
d1 d2 d1 d2

Σ = Σ11 Σ12 Ω = Ω11 Ω12

   
n1 and d1 ,
Σ21 Σ22 n2 Ω21 Ω22 d2
n1 n2 d1 d2

where n1 , n2 , d1 , d2 is the column or row length of the corresponding vector or matrix. Then,
1. X1r ∼ MN n1 ,d ( M1r , Σ11 , Ω),
 
−1
X2r | X1r ∼ MN n2 ,d M2r + Σ21 Σ11 ( X1r − M1r ), Σ22·1 , Ω ;

2. X1c ∼ MN n,d1 ( M1c , Σ, Ω11 ),

 
−1
X2c | X1c ∼ MN n,d2 M2c + ( X1c − M1c )Ω11 Ω12 , Σ, Ω22·1 ;

where Σ22·1 and Ω22·1 are the Schur complement [29] of Σ11 and Ω11 , respectively,
−1 −1
Σ22·1 = Σ22 − Σ21 Σ11 Σ12 , Ω22·1 = Ω22 − Ω21 Ω11 Ω12 .

2.2 Matrix-variate Student−t distribution

Definition 2 The random matrix X ∈ Rn×d is said to have a matrix-variate Student−t

distribution with the mean matrix M ∈ Rn×d and covariance matrix Σ ∈ Rn×n , Ω ∈
Rd×d and the degree of freedom ν if and only if the probability density function is given by

Γn [ 21 (ν + d + n − 1)] d n
p( X |ν, M, Σ, Ω) = 1 dn
det(Σ)− 2 det(Ω)− 2 ×
π 2 Γn [ 21 (ν + n − 1)]
1
det(In + Σ−1 ( X − M)Ω−1 ( X − M )T )− 2 (ν+d+n−1) , (2)

where Ω and Σ are positive semi-definite, and

n
1 i
Γn (λ) = π n(n−1)/4 ∏ Γ(λ + − ).
i =1
2 2

We denote this by X ∼ MT n,d (ν, M, Σ, Ω). Without loss of clarity, it is denoted X ∼

MT (ν, M, Σ, Ω).
Theorem 4 (Expectation and covariance) Let X ∼ MT (ν, M, Σ, Ω), then
1
E( X ) = M, cov(vec( X T )) = Σ ⊗ Ω, ν > 2.
ν−2
Theorem 5 (Transposable) If X ∼ MT n,d (ν, M, Σ, Ω), then X T ∼ MT d,n (ν, MT , Ω, Σ).
Multivariate Gaussian and Student−t process regression for multi-output prediction 5

d
Theorem 6 (Asymptotics) Let X ∼ MT n,d (ν, M, Σ, Ω),then X → MN n,d ( M, Σ, Ω)
d
as ν → ∞, where ”→” denotes convergence in distribution.
Theorem 7 (Marginalization and conditional distribution) Let X ∼ MT n,d (ν, M, Σ, Ω),
and partition X, M, Σ and Ω as
   
X1r n1 M1r n1
X= = [ X1c X2c ] , M= = [ M1c M2c ]
X2r n2 M2r n2
d1 d2 d1 d2

Σ = Σ11 Σ12 Ω = Ω11 Ω12

   
n1 and d1 ,
Σ21 Σ22 n2 Ω21 Ω22 d2
n1 n2 d1 d2

where n1 , n2 , d1 , d2 is the column or row length of the corresponding vector or matrix. Then,
1. X1r ∼ MT n1 ,d (ν, M1r , Σ11 , Ω),

−1
X2r | X1r ∼MT n2 ,d ν + n1 , M2r + Σ21 Σ11 ( X1r − M1r ), Σ22·1 ,

−1
Ω + ( X1r − M1r )T Σ11 ( X1r − M1r ) ;

2. X1c ∼ MT n,d1 (, ν, M1c , Σ, Ω11 ),


−1
X2c | X1c ∼MT n,d2 ν + d1 , M2c + ( X1c − M1c )Ω11 Ω12 ,

−1
Σ + ( X1c − M1c )Ω11 ( X1c − M1c )T , Ω22·1 ;

where Σ22·1 and Ω22·1 are the Schur complement of Σ11 and Ω11 , respectively,
−1 −1
Σ22·1 = Σ22 − Σ21 Σ11 Σ12 , Ω22·1 = Ω22 − Ω21 Ω11 Ω12 .

3 Multivariate Gaussian and Student−t process regression models

Like the derivation of GPR, we have to define multivariate Gaussian process before
the derivation of MV-GPR. Following the definition of GP, a multivariate Gaussian
process should be a collection of random vector-valued variables, any finite number
of which have matrix-variate Gaussian distribution. Therefore, we define a multi-
variate Gaussian process as follows.
Definition 3 (MV-GP) f is a multivariate Gaussian process on X with vector-valued
mean function u : X 7→ Rd , covariance function (also called kernel) k : X × X 7→ R and
positive semi-definite parameter matrix Ω ∈ Rd×d if any finite collection of vector-valued
variables have a joint matrix-variate Gaussian distribution,

[f ( x1 )T , . . . , f ( xn )T ]T ∼ MN ( M, Σ, Ω), n ∈ N,
where f , u ∈ Rd are row vectors whose components are the functions { f i }id=1 and {µi }id=1
respectively. Furthermore, M ∈ Rn×d with Mij = µ j ( xi ), and Σ ∈ Rn×n with Σij =
k( xi , x j ). Sometimes Σ is called column covariance matrix while Ω is row covariance ma-
trix. We denote f ∼ MGP (u, k, Ω).
6 Zexun Chen et al.

3.1 Multivariate Gaussian process regression

For a GPR, the noisy model, y = f ( x ) + ε is usually considered. However, for a

MV-TPR, there may exist the analytically intractable problems. In order to unify
the model derivation, we adopt the same method used in [24] for both MV-GPR
and MV-TPR and consider the noise free regression model with noise-incorporated
kernel .
Given n pairs of observations {( xi , yi )}in=1 , xi ∈ R p , yi ∈ Rd , we assume the
following model

f ∼ MGP (u, k0 , Ω),

yi = f ( xi ), for i = 1, · · · , n,

where k0 = k( xi , x j ) + δij σn2 , δij = 1 if i = j, otherwise δij = 0. We assume u = 0 as

commonly done in GPR.
By the definition of multivariate Gaussian process, it yields that the collection
of functions [f ( x1 ), . . . , f ( xn )] follow a matrix-variate Gaussian distribution

[f ( x1 )T , . . . , f ( xn )T ]T ∼ MN (0, K 0 , Ω),

where K 0 is the n × n covariance matrix of which the (i, j)-th element [K 0 ]ij =
k 0 ( x i , x j ).
To predict a new variable f∗ = [ f ∗1 , . . . , f ∗m ]T at the test locations X∗ = [ xn+1 , . . . , xn+m ]T ,
the joint distribution of the training observations Y = [y1T , · · · , ynT ]T and the predic-
tive targets f∗ are given by

K 0 ( X, X ) K 0 ( X∗ , X )T
     
Y
∼ MN 0, ,Ω , (3)
f∗ K 0 ( X∗ , X ) K 0 ( X∗ , X∗ )

where K 0 ( X, X ) is an n × n matrix of which the (i, j)-th element [K 0 ( X, X )]ij =

k0 ( xi , x j ), K 0 ( X∗ , X ) is an m × n matrix of which the (i, j)-th element [K 0 ( X∗ , X )]ij =
k0 ( xn+i , x j ), and K 0 ( X∗ , X∗ ) is an m × m matrix with the (i, j)-th element [K 0 ( X∗ , X∗ )]ij =
k0 ( xn+i , xn+ j ). Thus, taking advantage of conditional distribution of multivariate
Gaussian process, the predictive distribution is

p(f∗ | X, Y, X∗ ) = MN ( M̂, Σ̂, Ω̂), (4)

where

M̂ = K 0 ( X∗ , X )T K 0 ( X, X )−1 Y, (5)
0 0 T 0 −1 0
Σ̂ = K ( X∗ , X∗ ) − K ( X∗ , X ) K ( X, X ) K ( X∗ , X ), (6)
Ω̂ = Ω. (7)

Additionally, the expectation and the covariance are obtained,

E[f∗ ] = M̂ = K 0 ( X∗ , X )T K 0 ( X, X )−1 Y, (8)

cov(vec(f∗T )) = Σ̂ ⊗ Ω̂ = [K 0 ( X∗ , X∗ ) − K 0 ( X∗ , X )T K 0 ( X, X )−1 K 0 ( X∗ , X )] ⊗ Ω.(9)
Multivariate Gaussian and Student−t process regression for multi-output prediction 7

3.1.1 Kernel

Despite there are two covariance matrices, column covariance and row covariance
in MV-GPR, only the column covariance depended on inputs is considered as kernel
since it contains our presumptions about the function we wish to learn and define
the closeness and similarity between data points [22]. Of course, the choice of kernel
also has a profound impact on the performance of a multivariate Gaussian process
as well as multivariate Student−t process. The most widely-used kernel is Squared
Exponential (SE), which is defined as

k x − x 0 k2
k SE ( x, x 0 ) = s2f exp(− ),
2`2
where s2f is the signal variance and can also be considered as an output-scale am-
plitude and the parameter ` is the input (length or time) scale [23]. The kernel can
also be defined by Automatic Relevance Determination (ARD)

(x − x0 )T Θ −1 (x − x0 )
k SEard (x, x0 ) = s2f exp(− ),
2
p
where Θ is a diagonal matrix with the element components {`2i }i=1 , which are the
length scales for each corresponding input dimension.

3.1.2 Parameter estimation

Similar to GPR models, the hyper-parameters involved in the kernel of MV-GPR

need to be estimated from the training data. Although Monte Carlo methods can
perform GPR without the need of estimating hyperparameters [27, 5, 19, 18], the
common approach is to estimate them by means of maximum marginal likelihood
due to the high computational cost of Monte Carlo methods.
Compared with the conventional GPR model, Ω is an extra parameters so that
the undetermined parameters, which need to be estimated, contain the hyper-parameters
in the kernel, noisy level σn2 and the row covariance parameter matrix Ω. Due to
positive semi-definite, Ω can be denoted Ω = ΦΦT , where
φ11 0 · · · 0
 
 φ21 φ22 · · · 0 
Φ= . . . . .
 
 .. .. . . .. 
φd1 φd2 · · · φdd
To guarantee the uniqueness of Φ, the diagonal elements are restricted to be positive
and denote ϕii = ln(φii ) for i = 1, 2, · · · , d.
In MV-GPR model, the observations are followed by a matrix-variate Gaussian
distribution Y ∼ MN n,d (0, K 0 , Ω) where K 0 is the noisy column covariance matrix
with element [K 0 ]ij = k0 ( xi , x j ) so that K 0 = K + σn2 I where K is noise-free col-
umn covariance matrix with element [K ]ij = k( xi , x j ). As we know there are hyper-
parameters in the kernel k so that we can denote K = Kθ . The hyper-parameter set
denotes Θ = {θ1 , θ2 , . . .}, thus
∂K 0 ∂K 0 ∂Kθ
= In , =
∂σn2 ∂θi ∂θi
8 Zexun Chen et al.

According to the matrix-variate distribution, the negative log marginal likeli-

hood of observations is

nd d n 1
L= ln(2π ) + ln det(K 0 ) + ln det(Ω) + tr((K 0 )−1 YΩ−1 Y T ). (10)
2 2 2 2

The derivatives of the negative log marginal likelihood with respect to parame-
ter σn2 , θi , φij and ϕii are as follows

∂L d 1
= tr((K 0 )−1 ) − tr(αK0 Ω−1 αTK0 ),
∂σn2 2 2
∂L
   
d ∂K 1 ∂K
= tr (K 0 )−1 θ − tr αK0 Ω−1 αTK0 θ ,
∂θi 2 ∂θi 2 ∂θi
∂L n 1
= tr[Ω−1 (Eij ΦT + ΦEij )] − tr[αΩ (K 0 )−1 αTΩ (Eij ΦT + ΦEij )],
∂φij 2 2
∂L n 1
= tr[Ω−1 (Jii ΦT + ΦJii )] − tr[αΩ (K 0 )−1 αTΩ (Jii ΦT + ΦJii )],
∂ϕii 2 2

where αK0 = (K 0 )−1 Y, αΩ = Ω−1 Y T , Eij is the d × d elementary matrix having unity
in the (i,j)-th element and zeros elsewhere, and Jii is the same as Eij but with the
unity being replaced by e ϕii . The details can be found in A.2.

3.2 Comparison

In fact, by comparing the results in the existing methods [25, 2] and our proposed
methods (Equation (8) and Equation (9)), the predictive distributions are the same
if we consider the noise-free regression model. However, there is no doubt that
our proposed MV-GPR has more straightforward formulation, where the model
settings, derivations and computations are all directly performed in matrix form,
rather than vectorizing the matrices.
Furthermore, it is not difficult to find that Theorem 2 (vectorizable property)
plays an essential role in the derivation of the existing methods of multi-output
Gaussian process. In other words, the classical derivation does rely on the equiv-
alence between vectorized matrix-variate Gaussian distribution and multivariate
Gaussian distribution. Unfortunately, book [14] presents a simple example to show
that the similar result between vectorized matrix-variate Student−t distribution
and multivariate Student−t distribution does not exist. That is to say, the structure
of the derivation of the existing methods cannot be applied to the Student−t ver-
sion. However, the derivation of our proposed MV-GPR is based on the calculations
in the matrix form. That is to say, the derivation does not rely on the equivalence
in Theorem 2. As a consequence, the structure of our proposed MV-GPR is able
to be applied to derive MV-TPR naturally and thus it is easy to consider a depen-
dent Student−t process regression model for multi-output prediction by defining a
multivariate Student−t process.
To conclude, our proposed MV-GPR provides not only a new derivation of
Gaussian process for multi-output, but also a simple framework to derive more
general elliptical processes models for multi-output, that is, MV-TPR.
Multivariate Gaussian and Student−t process regression for multi-output prediction 9

3.3 Multivariate Student−t process regression

As an extension to multi-output GPR, it is natural to consider more general elliptical

processes models for multi-output prediction, for example, it is probably MV-TPR.
As the advantage of our proposed MV-GPR compared with the existing methods,
the framework of derivation in MV-GPR is able to be easily applied to derive MV-
TPR. First of all, we can define multivariate Student−t process based on the defini-
tion of MV-GP and TP proposed by [24].
Definition 4 (MV-TP) f is a multivariate Student−t process on X with parameter ν >
2, vector-valued mean function u : X 7→ Rd , covariance function (also called kernel)
k : X × X 7→ R and positive semi-definite parameter matrix Ω ∈ Rd×d if any finite
collection of vector-valued variables have a joint matrix-variate Student−t distribution,

[f ( x1 )T , . . . , f ( xn )T ]T ∼ MT (ν, M, Σ, Ω), n ∈ N,

where f , u ∈ Rd are row vectors whose components are the functions { f i }id=1 and {µi }id=1
respectively. Furthermore, M ∈ Rn×d with Mij = µ j ( xi ), and Σ ∈ Rn×n with Σij =
k( xi , x j ). We denote f ∼ MT P (ν, u, k, Ω).
Without much modifications, MV-TPR model can be formulated along the same
line as MV-GPR based on the definition of multivariate Student−t process and it
briefly presented below.
Given n pairs of observations {( xi , yi )}in=1 , xi ∈ R p , yi ∈ Rd , we assume

f ∼ MT P (ν, u, k0 , Ω), ν > 2,

yi = f ( xi ), for i = 1, · · · , n,

where ν is the degree of freedom of Student−t process and the remaining parame-
ters have the same meaning of MV-GPR model. Consequently, the predictive distri-
bution is obtained as

p(f∗ | X, Y, X∗ ) = MT (ν̂, M̂, Σ̂, Ω̂), (11)

where

ν̂ = ν + n, (12)
0 T 0 −1
M̂ = K ( X∗ , X ) K ( X, X ) Y, (13)
Σ̂ = K 0 ( X∗ , X∗ ) − K 0 ( X∗ , X )T K 0 ( X, X )−1 K 0 ( X∗ , X ), (14)
T 0 −1
Ω̂ = Ω + Y K ( X, X ) Y. (15)

According to the expectation and the covariance of matrix-variate Student−t distri-

bution, the predictive mean and covariance are given by

E[f∗ ] = M̂ = K 0 ( X∗ , X )T K 0 ( X, X )−1 Y, (16)

1
cov(vec(f∗T )) = Σ̂ ⊗ Ω̂
ν+n−2
1
= [K 0 ( X∗ , X∗ ) − K 0 ( X∗ , X )T K 0 ( X, X )−1 K 0 ( X∗ , X )]
ν+n−2
⊗ (Ω + YT K 0 ( X, X )−1 Y ). (17)
10 Zexun Chen et al.

In the MV-TPR model, the observations are followed by a matrix-variate Student−t

distribution Y ∼ MT n,d (ν, 0, K 0 , Ω). The negative log marginal likelihood is
1 d n
L= (ν + d + n − 1) ln det(In + (K 0 )−1 YΩ−1 YT ) + ln det(K 0 ) + ln det(Ω)
2    2  2
1 1 1
+ ln Γn (ν + n − 1) − ln Γn (ν + d + n − 1) + dn ln π
2 2 2
1 ν + n − 1
= (ν + d + n − 1) ln det(K 0 + YΩ−1 YT ) − ln det(K 0 )
2    2 
1 1 n 1
+ ln Γn (ν + n − 1) − ln Γn (ν + d + n − 1) + ln det(Ω) + dn ln π.
2 2 2 2
Therefore the parameters of MV-TPR contains the all the parameters in MV-GPR
and one more parameter: the degree of freedom ν. The derivatives of the negative
log marginal likelihood with respect to parameter ν,σn2 , θi , φij and ϕii are as follows

∂L
 
1 1 1 1 1 1
= ln det(U ) − ln det(K 0 ) + ψn ( τ ) − ψn (τ + d) ,
∂ν 2 2 2 2 2 2
∂L (τ + d) τ
= tr(U −1 ) − tr((K 0 )−1 ),
∂σn2 2 2
∂L
   
(τ + d) −1 ∂K τ −1 ∂Kθ
= tr U θ
− tr Σ ,
∂θi 2 ∂θi 2 ∂θi
∂L (τ + d) n
=− tr[U −1 αTΩ (Eij ΦT + ΦEij )αΩ ] + tr[Ω−1 (Eij ΦT + ΦEij )],
∂φij 2 2
∂L (τ + d) n
=− tr[U −1 αTΩ (Jii ΦT + ΦJii )αΩ ] + tr[Ω−1 (Jii ΦT + ΦJii )],
∂ϕii 2 2

where U = K 0 + YΩ−1 Y T , τ = ν + n − 1 and ψn (·) is the derivative of the function

ln Γn (·) with respect to ν. The details can be found in A.3.
Remark 1 As we know, the marginal likelihood function in GPR model is not convex with
respect to the hyper-parameters, therefore the optimization algorithm may converge to a lo-
cal optimum whereas the global one provides better result [11]. Consequently, the optimized
hyper-parameters achieved by maximum likelihood estimation and the performance of GPR
may depend on the initial values of the optimization algorithm [27, 5, 28, 18]. A common
strategy adopted by most GPR practitioners is a heuristic method. That is, the optimization
is repeated using several initial values generated randomly from a prior distribution based
on their expert opinions and experiences. Usually, 10 times randomly selections is used
from a simple prior, such as uniform distribution (0,1). The final estimates of the hyper-
parameters are the ones with the largest likelihood values after convergence [27, 5, 28]. Fur-
ther discussions of how to select a suitable initial hyper-parameters can be found in [28, 8].
For both MV-GPR and MV-TPR, the same heuristic method is applied.

4 Experiments

In this section, we demonstrate the usefulness of the matrix-variate process regres-

sion models using some numerical examples, including simulated data and real
data.
Multivariate Gaussian and Student−t process regression for multi-output prediction 11

4.1 Simulated Example

We first consider a simulated data from two specific functions. The true model used
to generate data is given by,

y = [ f 1 ( x ), f 2 ( x )] + [ε(1) , ε(2) ],
f 1 ( x ) = 2x · cos( x ), f 2 ( x ) = 1.5x · cos( x + π/5),

where the vector noise is produced from a sample of multivariate Gaussian process
[ε(1) , ε(2) ] ∼ MGP (0, k SE , Ω) . We select k SE with parameter [`, s2f ] = [ln(1.001), ln(5)]
and Ω = 0.25 1 0.25 . The covariate x has 100 equally spaced values in [-10, 10] so that


1
a sample of 100 observations for y1 and y2 are obtained.
For model training, we try to use fewer points with one part missing so that
the zth data points where z = {3r + 1}12 32
r =1 ∪ {3r + 2}r =22 are selected for both y1
and y2 . The prediction is performed at all 100 covariate values equally spaced in
[-10, 10]. The RMSEs between the predicted values and the true ones from f 1 ( x )
and f 2 ( x ) are calculated. At the same time, the conventional GPR and TPR models
are conducted for the two outputs independently, which are used to compare with
the proposed models. The process above is repeated 1000 times and the results are
reported in Table 1 and an example of prediction is given in Figure 1. The ARMSE
(Average Root Mean Square Error) for 100 points predictions repeated 1000 times is
defined by
!1
1 1000 1 100
2
ARMSE = ∑
1000 i=1 100 j=1 ∑ (ŷij − yij ) 2
,

where yij is the jth observation in the ith experiment while ŷij is the jth prediction
in the ith experiment.

Table 1 The ARMSE by the different models (multivariate Gaussian noisy data)

Output 1 (y1 ) Output 2 (y2 )

MV-GP GP MV-TP TP MV-GP GP MV-TP TP
1.540 1.594 1.258 1.585 1.749 2.018 1.518 2.017

The same experiment is also conducted for the case where the vector noise is
a sample from multivariate Student−t process [ε(1) , ε(2) ] ∼ MT P (3, 0, k SE , Ω).We
select k SE with parameter [`, s2f ] = [ln(1.001), ln(5)] and Ω = [1, 0.25; 0.25, 1]. The
result of ARMSEs are presented in Table 2 and an example of prediction is demon-
strated in Figure 2.
From the tables and figures above, it can be seen that the multivariate process
regression models are able to discover a more desired pattern in the gap compared
with using the conventional GPR and TPR model independently. It also reveals that
taking correlations between the two outputs into consideration improves the accu-
racy of prediction compared with the methods of modeling each output indepen-
dently. In particular, MV-TPR performs better than MV-GPR in the predictions of
12 Zexun Chen et al.

MV-GP, y 1 MV-TP, y 1
20 25

15 20

15
10

10
5
5
0
0
-5
-5

-10
-10

-15 -15

-20 -20
-10 -8 -6 -4 -2 0 2 4 6 8 10 -10 -8 -6 -4 -2 0 2 4 6 8 10
(a) MV-GPR (y1 ) (b) MV-TPR (y1 )

GP, y 1 TP, y 1
20 20

15 15

10 10

5 5

0 0

-5 -5

-10 -10

-15 -15

-20 -20
-10 -8 -6 -4 -2 0 2 4 6 8 10 -10 -8 -6 -4 -2 0 2 4 6 8 10
(c) GPR (y1 ) (d) TPR (y1 )

MV-GP, y 2 MV-TP, y 2
15 20

15
10

10

5
5

0 0

-5
-5

-10

-10
-15

-15 -20
-10 -8 -6 -4 -2 0 2 4 6 8 10 -10 -8 -6 -4 -2 0 2 4 6 8 10
(e) MV-GPR (y2 ) (f) MV-TPR (y2 )

GP, y 2 TP, y 2
20 20

15 15

10 10

5 5

0 0

-5 -5

-10 -10

-15 -15
-10 -8 -6 -4 -2 0 2 4 6 8 10 -10 -8 -6 -4 -2 0 2 4 6 8 10
(g) GPR (y2 ) (h) TPR (y2 )

Fig. 1 Predictions for MV-GP noise data using different models. From panels (a) to (d): predictions
for y1 by MV-GPR, MV-TPR, GPR and TPR. From panels (e) to (h): predictions for y2 by MV-GPR,
MV-TPR, GPR and TPR. The solid blue lines are predictions, the solid red lines are the true functions
and the circles are the observations. The dash lines represent the 95% confidence intervals
Multivariate Gaussian and Student−t process regression for multi-output prediction 13

MV-GP, y 1 MV-TP, y 1
25 25

20 20

15 15

10 10

5 5

0 0

-5 -5

-10 -10

-15 -15

-20 -20

-25 -25
-10 -8 -6 -4 -2 0 2 4 6 8 10 -10 -8 -6 -4 -2 0 2 4 6 8 10
(a) MV-GPR (y1 ) (b) MV-TPR (y1 )

GP, y 1 TP, y 1
25 25

20 20

15 15

10 10

5 5

0 0

-5 -5

-10 -10

-15 -15

-20 -20

-25 -25
-10 -8 -6 -4 -2 0 2 4 6 8 10 -10 -8 -6 -4 -2 0 2 4 6 8 10
(c) GPR (y1 ) (d) TPR (y1 )

MV-GP, y 2 MV-TP, y 2
20 20

15 15

10 10

5 5

0 0

-5 -5

-10 -10

-15 -15

-20 -20
-10 -8 -6 -4 -2 0 2 4 6 8 10 -10 -8 -6 -4 -2 0 2 4 6 8 10
(e) MV-GPR (y2 ) (f) MV-TPR (y2 )

GP, y 2 TP, y 2
20 20

15 15

10 10

5 5

0 0

-5 -5

-10 -10

-15 -15

-20 -20

-25 -25
-10 -8 -6 -4 -2 0 2 4 6 8 10 -10 -8 -6 -4 -2 0 2 4 6 8 10
(g) GPR (y2 ) (h) TPR (y2 )

Fig. 2 Predictions for MV-TP noise data using different models. From panels (a) to (d): predictions
for y1 by MV-GPR, MV-TPR, GPR and TPR. From panels (e) to (h): predictions for y2 by MV-GPR,
MV-TPR, GPR and TPR. The solid blue lines are predictions, the solid red lines are the true functions
and the circles are the observations. The dash lines represent the 95% confidence intervals
14 Zexun Chen et al.

Table 2 The ARMSE by the different models (multivariate Student−t noisy data)

Output 1 (y1 ) Output 2 (y2 )

MV-GP GP MV-TP TP MV-GP GP MV-TP TP
1.441 1.505 1.238 1.503 1.636 1.941 1.464 1.940

both types of noisy data. This may be explained by the fact that MV-TPR has a bet-
ter modeling flexibility with one more parameter which can capture the degree of
freedom of the data and take the correlations between two responses into account.
The reason remains to be studied in future.
It is notable that the predictive variance of MV-GPR is much smaller than the
independent GPR model. This is likely caused by the loss of information in the in-
dependent model. As discussed in [15], the prediction uncertainty of GPR is useful
in building the predicting model by ensemble learning.

4.2 Real data examples

We further test our proposed methods on two real datasets 1 . The selected mean
function is zero-offset and the selected kernel is SEard. Before the experiments, all
the data have been normalized by
yi − µ
ỹi = ,
σ
where µ and σ are the sample mean and standard deviation of the data {yi }in=1
respectively.

4.2.1 Air quality prediction

The dataset contains 9358 instances of hourly averaged responses from an array of
5 metal oxide chemical sensors embedded in an Air Quality Chemical Multisen-
sor Device with 15 attributes [10]. We delete all the points with missing attributes
(887 points remaining). The first 864 points are considered in our experiment be-
cause the data is hourly observed (1 day = 24 hours) and the whole data set is
divided into 9 subsets (each subset has 4-days’ data points, totally 864 data points).
In the experiment, the input is comprised of 9 attributes, including time, true hourly
averaged concentration CO in mg/m3 (COGT), true hourly averaged overall Non
Metanic HydroCarbons concentration in microg/m3 (NMHCGT), true hourly aver-
aged Benzene concentration in microg/m3 (C6H6GT), true hourly averaged NOx
concentration in ppb (NOx), true hourly averaged NO2 concentration in microg/m3
(NO2), absolute humidity (AH), temperature (T) and relative humidity (RH). The
output consists of 5 attributes, including PT08.S1 (tin oxide) hourly averaged sen-
sor response, PT08.S2 (titania) hourly averaged sensor response, PT08.S3 (tungsten
oxide) hourly averaged sensor response, PT08.S4 (tungsten oxide) hourly averaged
sensor response and PT08.S5 (indium oxide) hourly averaged sensor response.
1 These data sets are from the UC Irvine Machine Learning Repository: https://archive.ics.

uci.edu/ml/index.php.
Multivariate Gaussian and Student−t process regression for multi-output prediction 15

The cross-validation method is taken as k −fold, where k = 9. Each subset is

considered as test set and the remaining subsets are considered as training set. Four
models, containing MV-GPR, MV-TPR, GPR (predict each output independently)
and TPR (predict each output independently) are applied to make multi-output
prediction based on the divided training and test sets. The process is repeated for 9
times.
For each subset’s prediction, MSE (mean square error) and MAE(mean absolute
error) are calculated and the median of the 9 MSEs and MAEs is used to evaluate
each output. Finally, the maximum median of all the outputs (MMO) is used to eval-
uate the multi-dimensional prediction. The results are shown in Table 3 and we can
verify empirically that MV- TPR performs the best since it has smallest maximum
error in terms of MSE and MAE.

Table 3 Air quality prediction results based on MSEs and MAEs

(a) MSE
MV-GPR MV-TPR GPR TPR
PT08S1CO 0.091 0.065 0.079 0.074
Outputs PT08S2NMHC 8.16 × 10−5 3.42 × 10−5 1.91 × 10−7 7.32 × 10−8
(Median of 9 PT08S3NOx 0.036 0.027 0.022 0.025
subsets’ MSEs) PT08S4NO2 0.015 0.014 0.010 0.009
PT08S5O3 0.092 0.073 0.060 0.067
MMO 0.092 0.073 0.079 0.074

(b) MAE
MV-GPR MV-TPR GPR TPR
PT08S1CO 0.240 0.204 0.212 0.223
Outputs PT08S2NMHC 6.39 × 10−3 1.15 × 10−2 1.80 × 10−4 9.26 × 10−5
(Median of 9 PT08S3NOx 0.141 0.122 0.115 0.120
subsets’ MAEs) PT08S4NO2 0.095 0.089 0.079 0.073
PT08S5O3 0.231 0.210 0.199 0.205
MMO 0.240 0.210 0.212 0.223

4.2.2 Bike rent prediction

This dataset contains the hourly and daily count of rental bikes between years 2011
and 2012 in Capital bikeshare system with the corresponding weather and seasonal
information [12]. There are 16 attributes. We test our proposed methods for multi-
output prediction based on daily count dataset. After deleting all the points with
missing attributes, we use the first 168 data points in the season Autumn because
the data is daily observed (1 week = 7 days) and the whole dataset is divided into
8 subsets (each subset has 3 weeks’ data points). In the experiment, the input is
comprised of 8 attributes, including normalized temperature, normalized feeling
temperature, normalized humidity, normalized wind speed, whether day is holiday
or not, day of the week, working day or not and weathersit. The output consists of
2 attributes, including the count of casual users (Casual) and the count of registered
users (Registered).
16 Zexun Chen et al.

The cross-validation method is taken as k−fold, where k = 8. All the remaining

steps are the same as the air quality prediction experiment, except k is 8 so that the
process is repeated 8 times.

Table 4 Bike rent prediction results based on MSEs and MAEs

(a) MSE
MV-GPR MV-TPR GPR TPR
Outputs (median Casual 0.411 0.334 0.424 0.397
of 8 subsets’ MSEs) Registered 0.982 0.903 1.134 1.111
MMO 0.982 0.903 1.134 1.111

(b) MAE
MV-GPR MV-TPR GPR TPR
Outputs (median Casual 0.558 0.488 0.540 0.546
of 8 subsets’ MAEs) Registered 0.897 0.855 0.916 0.907
MMO 0.897 0.855 0.916 0.907

The results are shown in Table 4 and we can also verify empirically that MV-
TPR performs the best in terms of MSE and MAE.

4.3 Application to stock market investment

In the previous subsections, the examples show the usefulness of our proposed
methods in terms of more accurate prediction. Furthermore, our proposed meth-
ods can be applied to produce trading strategies in the stock market investment.
It is known that the accurate prediction of future for an equity market is almost
impossible. Admittedly, the more realistic idea is to make a strategy based on the
Buy&Sell signal in the different prediction models [1]. In this paper, we consider a
developed Dollar 100 (dD100) as a criterion of the prediction models. The dD100
criterion is able to reflect the theoretical future value of $100 invested at the begin-
ning, and traded according to the signals constructed by predicted value and the
reality. The details of dD100 criterion are described in Section 4.3.2.
Furthermore, the equity index is an important measurement of the value of a
stock market and is used by many investors making trades and scholars studying
stock markets. The index is computed from the weighted average of the selected
stocks prices, so it is able to describe how the whole stock market in the considera-
tion performs in a period and thus many trading strategies of a stock or a portfolio
have to take the information of the index into account. As a result, our experimental
predictions for specific stocks are based on the indices as well.

4.3.1 Data preparation

We obtain daily price data, containing opening, closing, and adjusted closing for
the stocks (the details are shown in Section 4.3.3 and Section 4.3.4) and three main
Multivariate Gaussian and Student−t process regression for multi-output prediction 17

indices in the US, Dow Jones Industrial Average (INDU), S&P500 (SPX), and NAS-
DAQ (NDX) from Yahoo Finance in the period of 2013 – 2014. The log returns of the
adjusted closing price and inter-day log returns are obtained by defining

ACPi
Log return: LRi = ln ,
ACPi−1
CPi
Inter-day log return: ILRi = ln ,
OPi

where ACPi is the adjusted closing price of the ith day (i > 1), CPi is the closing
price of the ith day, and OPi is the opening price of the ith day. Therefore, there are
totally 503 daily log returns and log inter-day returns for all the stocks and indices
from 2013 to 2014.

4.3.2 Prediction model and strategy

The sliding windows method is used for our prediction models, including GPR,
TPR, MV-GPR, and MV-TPR, based on the indices, INDU, SPX, and NDX. The train-
ing sample is set as 303, which is used to forecast for the next 10 days, and the train-
ing set is updated by dropping off the earliest 10 days and adding on the latest 10
days when the window is moved. The sliding-forward process was run 20 times,
resulting in a total of 200 prediction days, in groups of 10. The updated training set
allows all the models and parameters to adapt the dynamic structure of the equity
market [1]. Specifically, the inputs consist of the log returns of 3 indices, the targets
are multiple stocks’ log returns and Standard Exponential with automatic relevance
determination (SEard) is used as the kernel for all of these prediction models.
It is noteworthy that the predicted log returns of stocks are used to produce a
buy or sell signal for trading rather than to discover an exact pattern of the future.
The signal BS produced by the predicted log returns of the stocks is defined by

ˆ i − LRi + ILRi , i = 1, · · · , 200,

BSi = LR

where { LRˆ i }200 are the predicted log returns of a specific stock, { LRi }200 are the
i =1 i =1
true log returns while { ILRi }200
i =1 are the inter-day log returns. The Buy&Sell strat-
egy relying on the signal BS is described in Table 5.

Table 5 Buy&Sell strategy of dD100 investment

Decision Condition
Buy ˆ i > 0, & BSi > 0 & we have the position of cash
LR
Sell ˆ i < 0, & BSi < 0 & we have the position of share
LR
Keep No action is taken for the rest of the option

It is noted that the stocks in our experiment are counted in Dollar rather than
the number of shares, which means in theory we can precisely buy or sell a specific
Dollar valued stock. For example, if the stock price is $37 when we only have $20,
we can still buy $20 valued stock rather than borrowing $17 and then buy 1 share.
18 Zexun Chen et al.

Furthermore, it is also necessary to explain why we choose the signal BS. By the
definition, we rewrite it as
ˆ i
ACP ACPi CP
BSi = ln( ) − ln( ) + ln( i )
ACPi−1 ACPi−1 OPi
ˆ i
ACP ACPi ACPi
= ln( ) − ln( ) + ln( )
ACPi−1 ACPi−1 AOPi
ˆ i
ACP
= ln( ),
AOPi

where { ACPi }200 200

i =0 are the last 201 adjusted closing prices for a stock, {CP }i =1 are the
200
last 200 closing prices, and { AOP}i=1 are the adjusted opening prices. If BSi > 0,
the predicted closing price should be higher than the adjusted opening price, which
means we can obtain the inter-day profit by buying the shares at the opening price 2
as long as the signal based on our predictions is accurate. Meanwhile, the opposite
manipulation based on BS strategy means that we can avoid the inter-day loss by
selling decisively at the opening price. Furthermore, the reasonable transaction fee
0.025% is considered in the experiment since the strategy might trade frequently 3 .
As a result, this is a reasonable strategy since we can definitely obtain a profit by
buying the shares and cut the loss by selling the shares in time only if our prediction
has no serious problem. It is also an executable strategy because the decision is
made based on the next day’s reality and our prediction models.
At last, BS signal varies in different prediction models so that we denote these
Buy&Sell strategies based on MV-GPR, MV-TPR, GPR, and TPR model as MV-GPR
strategy, MV-TPR strategy, GPR strategy, and TPR strategy, respectively.

4.3.3 Chinese companies in NASDAQ

In recent years, the ”Chinese concepts stock” has received an extensive attention
among international investors owing to the fast development of Chinese economy
and an increasing number of Chinese firms have been traded in the international
stock markets [16]. The ”Chinese concepts stock” refers to the stock issued by firms
whose asset or earning have essential activities in Mainland China. Undoubtedly,
all these ”Chinese concept stocks” are heavily influenced by the political and eco-
nomic environment of China together. For this reason, all these stocks have the po-
tential and unneglectable correlation theoretically, which is probably reflected in the
movement of stock prices. The performance of multiple targets prediction, which
takes the potential relationship into consideration, should be better. Therefore, the
first real data example is based on 3 biggest Chinese companies described in Table
6.
We apply MV-GPR, MV-TPR, GPR and TPR strategies and the results are demon-
strated in Figure 3. Furthermore, Table B1, Table B2 and Table B3 in the appendix
summarize the results by period for each stock respectively. In particular, the Buy&Sell
2 Actually, the value has to be considered as adjusted opening price since all the shares are

counted in Dollar. The adjusted opening price is also easy to compute based on the real opening
price and the dividend information
3 The figure 0.025% is comprehensive consideration referred to NASDAQ website:http://

nasdaq.cchwallstreet.com/
Multivariate Gaussian and Student−t process regression for multi-output prediction 19

Table 6 Three biggest ”Chinese concept” stocks

Ticker Exchange Company

BIDU NASDAQ Baidu, Inc.
CTRP NASDAQ Ctrip.com International, Ltd.
NTES NASDAQ NetEase, Inc.

signal examples for each stock are shown in Table B4, Table B5 and Table B6 respec-
tively, along with other relevant details.

$100 Investment on BIDU $100 Investment on NTES

300 190

MV-GPR Strategy MV-GPR Strategy

180
MV-TPR Strategy MV-TPR Strategy
GPR Strategy GPR Strategy
250 170
TPR Strategy TPR Strategy
Buy-and-Hold Stragety Buy-and-Hold Stragety
160
Dow Jones Index Dow Jones Index
NASDAQ NASDAQ
200 S&P500
150 S&P500

140

150 130

120

100 110

100

50 90
Mar 2014 May 2014 Jul 2014 Sep 2014 Nov 2014 Jan 2015 Mar 2014 May 2014 Jul 2014 Sep 2014 Nov 2014 Jan 2015

$100 Investment on CTRP

240

MV-GPR Strategy
220 MV-TPR Strategy
GPR Strategy
TPR Strategy
200
Buy-and-Hold Stragety
Dow Jones Index
180 NASDAQ
S&P500

160

140

120

100

80
Mar 2014 May 2014 Jul 2014 Sep 2014 Nov 2014 Jan 2015

Fig. 3 The movement of invested $100 in 200 days for 3 Chinese stocks in the US market. The top 4
lines in legend are Buy&Sell strategies based on 4 prediction models, MV-GPR, MV-TPR, GPR, and
TPR, respectively. The last 4 lines are Buy&Hold strategies for the stock and for the three indices,
INDU, NASDAQ, and NDX, respectively

From the view of Figure 3, there is no doubt that a $100 investment for each
stock has sharply increased over 200 days period using Buy&Sell strategies no
matter whether the stock went up or down during this period. In particular, the
stock prices of BIDU and NTES rose up gradually while CTRP hit the peak and
then decreased in a large scale. Anyway, the Buy&Sell strategies based on dif-
ferent prediction models have still achieved considerable profits compared with
Buy&Hold strategies for the corresponding investments. However, the different
prediction models have diverse performances for each stock. For BIDU, GPR-based
models, including MV-GPR and GPR, outperform TPR-based models, including
MV-TPR and TPR. For NTES, all the models for Buy&Sell strategy have the sim-
20 Zexun Chen et al.

ilar performance. Admittedly, TPR-based models, especially MV-TPR, have an out-

standing performance for stock CTRP.

4.3.4 Diverse sectors in Dow 30

Owing to the globalization of capital, there has been a significant shift in the relative
importance of national and economic influences in the world’s largest equity mar-
kets and the impact of industrial sector effects is now gradually replacing that of
country effects in these markets [3]. Therefore, a further example is carried out un-
der the diverse industrial sectors in Dow 30 from New York Stock Exchange (NYSE)
and NASDAQ.
Initially, the classification of stocks based on diverse industrial sectors in Dow
30 has to be done. There are two main industry classification taxonomies, including
Industry Classification Benchmark (ICB) and Global Industry Classification Stan-
dard (GICS). In our research, ICB is used to segregate markets into sectors within
the macro economy. The stocks in Dow 30 are classified in Table 7. Due to the multi-

Table 7 Stock components of Dow 30

Ticker Company Exchange Industry Industry 4 (ICB)

DD DuPont NYSE Chemical industry Basic Materials
KO Coca-Cola NYSE Beverages Consumer Goods
PG Procter & Gamble NYSE Consumer goods Consumer Goods
MCD McDonald’s NYSE Fast food Consumer Goods
NKE Nike NYSE Apparel Consumer Services
DIS Walt Disney NYSE Broadcasting and entertainment Consumer Services
HD The Home Depot NYSE Home improvement retailer Consumer Services
WMT Wal-Mart NYSE Retail Consumer Services
JPM JPMorgan Chase NYSE Banking Financials
GS Goldman Sachs NYSE Banking, Financial services Financials
V Visa NYSE Consumer banking Financials
AXP American Express NYSE Consumer finance Financials
TRV Travelers NYSE Insurance Financials
UNH UnitedHealth Group NYSE Managed health care Health Care
JNJ Johnson & Johnson NYSE Pharmaceuticals Health Care
MRK Merck NYSE Pharmaceuticals Health Care
PFE Pfizer NYSE Pharmaceuticals Health Care
BA Boeing NYSE Aerospace and defense Industrials
MMM 3M NYSE Conglomerate Industrials
GE General Electric NYSE Conglomerate Industrials
UTX United Technologies NYSE Conglomerate Industrials
CAT Caterpillar NYSE Construction and mining equipment Industrials
CVX Chevron NYSE Oil & gas Oil & Gas
XOM ExxonMobil NYSE Oil & gas Oil & Gas
CSCO Cisco Systems NASDAQ Computer networking Technology
IBM IBM NYSE Computers and technology Technology
AAPL Apple NASDAQ Consumer electronics Technology
INTC Intel NASDAQ Semiconductors Technology
MSFT Microsoft NASDAQ Software Technology
VZ Verizon NYSE Telecommunication Telecommunications

variate process models considering at least two related stocks in one group, the first
(Basic Materials) and the last industrial sector (Telecommunications), each consist-
ing of only one stock, are excluded. As a result, our experiments are performed
4 Note that the terms ”industry” and ”sector” are reversed from the Global Industry Classifica-

tion Standard (GICS) taxonomy.

Multivariate Gaussian and Student−t process regression for multi-output prediction 21

seven times for the seven grouped industrial sector stocks, including Oil&Gas, In-
dustrial, Consumer Goods, Health Care, Consumer Services, Financials and Tech-
nology, respectively.
Secondly, the 4 models, MV-GPR, MV-TPR, GPR and TPR, are applied in the
same way as in Section 4.3.3 and the ranking of stock investment performance is
listed in Table 8 (the details are summarized in Table C1). On the whole, for each
stock, there is no doubt that using Buy&Sell strategy is much better than using
Buy&Hold strategy regardless of the industrial sector. Specifically, MV-GPR makes
a satisfactory performance overall in the sectors, Industrials, Consumer Services
and Financials while MV-TPR has a higher ranking in Health Care in general.

Table 8 Stock investment performance ranking under different strategies

Buy&Sell Strategy Buy&Hold Stragegy

Ticker Industry
MV-GPR MV-TPR GPR TPR Stock INDU NDX SPX
CVX Oil & Gas 3rd 4th 2nd 1st 8th 7th 5th 6th
XOM Oil & Gas 4th 2nd 3rd 1st 8th 7th 5th 6th
MMM Industrials 2nd 3rd 1st 4th 5th 8th 6th 7th
BA Industrials 1st 2nd 3rd 4th 8th 7th 5th 6th
CAT Industrials 3rd 4th 2nd 1st 8th 7th 5th 6th
GE Industrials 2nd 4th 3rd 1st 8th 7th 5th 6th
UTX Industrials 2nd 4th 3rd 1st 8th 7th 5th 6th
KO Consumer Goods 2nd 1st 3rd 4th 6th 8th 5th 7th
MCD Consumer Goods 2nd 4th 1st 3rd 8th 7th 5th 6th
PG Consumer Goods 3rd 4th 1st 2nd 5th 8th 6th 7th
JNJ Health Care 3rd 2nd 1st 4th 6th 8th 5th 7th
MRK Health Care 3rd 2nd 4th 1st 8th 7th 5th 6th
PFE Health Care 4th 1st 3rd 2nd 8th 7th 5th 6th
UNH Health Care 2nd 3rd 1st 4th 5th 8th 6th 7th
HD Consumer Services 1st 4th 3rd 2nd 5th 8th 6th 7th
NKE Consumer Services 2nd 3rd 4th 1st 5th 8th 6th 7th
WMT Consumer Services 1st 4th 3rd 2nd 5th 8th 6th 7th
DIS Consumer Services 3rd 2nd 1st 4th 5th 8th 6th 7th
AXP Financials 2nd 4th 1st 3rd 8th 7th 5th 6th
GS Financials 2nd 1st 3rd 4th 5th 8th 6th 7th
JPM Financials 2nd 4th 1st 3rd 6th 8th 5th 7th
TRV Financials 2nd 3rd 1st 4th 5th 8th 6th 7th
V Financials 1st 4th 3rd 2nd 5th 8th 6th 7th
AAPL Technology 4th 2nd 3rd 1st 5th 8th 6th 7th
CSCO Technology 2nd 1st 3rd 4th 5th 8th 6th 7th
IBM Technology 4th 1st 2nd 3rd 8th 7th 5th 6th
INTC Technology 3rd 4th 2nd 1st 5th 8th 6th 7th
MSFT Technology 2nd 4th 1st 3rd 5th 8th 6th 7th

Further analysis is considered using industrial sector portfolios, which consists

of these grouped stocks by the same weight investment on each stock. For example,
the Oil & Gas portfolio investment is $100 with $50 shares CVX and $50 shares XOM
while the Technology portfolio investment is $100 with the same $20 investment on
each stock in the industrial sector Technology. The diverse industry portfolio invest-
ment performance ranking is listed in Table 9 (the details are described in Table C2).
Apparently, the Buy&Sell strategies performed better than the Buy&Hold strate-
gies. MV-GPR suits better in three industries, including Consumer Goods, Con-
sumer Services, and Financials, followed by TPR which performed best in Oil&Gas
22 Zexun Chen et al.

and Industrials. The optimal investment strategy in Health Care is MV-TPR while
in Technology industry, using GPR seems to be the most profitable.

Table 9 Industry portfolio investment performance ranking under different strategies

Buy&Sell Strategy Buy&Hold Stragegy

Industry Portfolio
MV-GPR MV-TPR GPR TPR Stock INDU NDX SPX
Oil & Gas 4th 3rd 2nd 1st 8th 7th 5th 6th
Industrials 2nd 4th 3rd 1st 8th 7th 5th 6th
Consumer Goods 1st 4th 2nd 3rd 7th 8th 5th 6th
Health Care 4th 1st 3rd 2nd 6th 8th 5th 7th
Consumer Services 1st 4th 3rd 2nd 5th 8th 6th 7th
Financials 1st 4th 2nd 3rd 5th 8th 6th 7th
Technology 4th 3rd 1st 2nd 5th 8th 6th 7th

5 Conclusion and discussion

In this paper, we propose an another derivation of dependent Gaussian process

regression for multi-output prediction, where the model settings, derivations and
computations are all directly performed in matrix form. MV-GPR is a more straight-
forward method and can be implemented in the same way as conventional GPR.
Like the conventional Gaussian process for vector-valued function, our models are
also able to learn the correlations between inputs and outputs. In comparison to
the conventional Gaussian process for vector-valued function, our formulations are
more convenient and flexible.
Furthermore, our proposed MV-GPR provides not only a new derivation of
Gaussian process for multi-output, but also a simple framework to derive more
general elliptical processes models for multi-output, that is, MV-TPR. We define the
multivariate Student−t process and then derive a new considerable method, MV-
TPR. Both MV-GPR and MV-TPR have closed form expressions for the marginal
likelihoods and predictive distributions. The usefulness of the proposed methods
are illustrated through several numerical examples.
The proposed methods are also applied to stock market modelling and are shown
to make profitable stock investment. Firstly, MV-GPR and MV-TPR are applied
to predict three ”Chinese concept stocks” together as 3-dimensional outputs. The
Buy&Sell strategies based on the proposed models have more satisfactory perfor-
mances compared with Buy&Hold strategies for the corresponding stocks and three
main indices in the US, especially the strategy based on MV-TPR has outstanding
returns for NetEase among three Chinese stocks. Secondly, our proposed methods
are applied to make Buy&Sell strategies from view of industrial sectors in Dow 30
and their results indicate that the strategies based on MV-GPR generally has con-
siderable performances in Industrials, Consumer Goods, Consumer Services, and
Financials sectors while the strategies based on MV-TPR can make maximum profit
in Health Care sector.
Multivariate Gaussian and Student−t process regression for multi-output prediction 23

It is noted that in this paper we assumed that the different outputs were ob-
served at the same covariate values. In practice, the different responses sometimes
are observed at several different locations. This is, however, difficult for the pro-
posed method since all the outputs had to be considered as a matrix rather than a
vector with adjustable length and thus each response had to depend on the same
input. Additionally, the kernel used in our model was squared exponential and was
the same for each output, whilst it may be better to use different kernels for different
outputs [25]. All these problems remain to make a further exploration.

Conflict of interest: The authors declare no conflict of interest.

References

1. Akbilgic, O., Bozdogan, H., Balaban, M.E.: A novel hybrid RBF neural networks model as a
forecaster. Statistics and Computing 24(3), 365–375 (2014)
2. Alvarez, M.A., Rosasco, L., Lawrence, N.D., et al.: Kernels for vector-valued functions: A re-
view. Foundations and Trends R in Machine Learning 4(3), 195–266 (2012)
3. Baca, S.P., Garbe, B.L., Weiss, R.A.: The rise of sector effects in major equity markets. Financial
Analysts Journal 56(5), 34–40 (2000)
4. Boyle, P., Frean, M.: Dependent Gaussian processes. In: Advances in neural information pro-
cessing systems, pp. 217–224 (2005)
5. Brahim-Belhouari, S., Bermak, A.: Gaussian process for nonstationary time series prediction.
Computational Statistics and Data Analysis 47(4), 705–712 (2004)
6. Brahim-Belhouari, S., Vesin, J.M.: Bayesian learning using Gaussian process for time series pre-
diction. In: Statistical Signal Processing, 2001. Proceedings of the 11th IEEE Signal Processing
Workshop on, pp. 433–436. IEEE (2001)
7. Chakrabarty, D., Biswas, M., Bhattacharya, S., et al.: Bayesian nonparametric estimation of
Milky Way parameters using matrix-variate data, in a new gaussian process based method.
Electronic Journal of Statistics 9(1), 1378–1403 (2015)
8. Chen, Z., Wang, B.: How priors of initial hyperparameters affect Gaussian process regression
models. arXiv preprint arXiv:1605.07906 (2016)
9. Dawid, A.P.: Some matrix-variate distribution theory: notational considerations and a bayesian
application. Biometrika 68(1), 265–274 (1981)
10. De Vito, S., Massera, E., Piga, M., Martinotto, L., Di Francia, G.: On field calibration of an elec-
tronic nose for benzene estimation in an urban pollution monitoring scenario. Sensors and
Actuators B: Chemical 129(2), 750–757 (2008)
11. Duvenaud, D., Lloyd, J.R., Grosse, R., Tenenbaum, J.B., Ghahramani, Z.: Structure discovery in
nonparametric regression through compositional kernel search. arXiv preprint arXiv:1302.4922
(2013)
12. Fanaee-T, H., Gama, J.: Event labeling combining ensemble detectors and background knowl-
edge. Progress in AI 2(2-3), 113–127 (2014)
13. Gentle, J.E.: Matrix algebra: theory, computations, and applications in statistics. Springer Sci-
ence & Business Media (2007)
14. Gupta, A.K., Nagar, D.K.: Matrix variate distributions, vol. 104. CRC Press (1999)
15. Liu, Y., Gao, Z.: Real-time property prediction for an industrial rubber-mixing process with
probabilistic ensemble gaussian process regression models. Journal of Applied Polymer Sci-
ence 132(6) (2015)
16. Luo, Y., Fang, F., Esqueda, O.A.: The overseas listing puzzle: Post-IPO performance of Chinese
stocks and adrs in the US market. Journal of multinational financial management 22(5), 193–211
(2012)
17. MacKay, D.J.: Gaussian processes-a replacement for supervised neural networks? (1997)
18. MacKay, D.J.: Introduction to Gaussian processes. NATO ASI Series F Computer and Systems
Sciences 168, 133–166 (1998)
19. Neal, R.M.: Monte carlo implementation of Gaussian process models for bayesian regression
and classification. arXiv preprint physics/9701026 (1997)
20. Neal, R.M.: Bayesian learning for neural networks, vol. 118. Springer Science & Business Media
(2012)
24 Zexun Chen et al.

21. Rasmussen, C.E.: Evaluation of Gaussian processes and other methods for non-linear regres-
sion. University of Toronto (1999)
22. Rasmussen, C.E., Williams, C.K.: Gaussian processes for machine learning, vol. 1. MIT press
Cambridge (2006)
23. Roberts, S., Osborne, M., Ebden, M., Reece, S., Gibson, N., Aigrain, S.: Gaussian processes for
time-series modelling. Phil. Trans. R. Soc. A 371(1984), 20110,550 (2013)
24. Shah, A., Wilson, A.G., Ghahramani, Z.: Student-t processes as alternatives to Gaussian pro-
cesses. In: AISTATS, pp. 877–885 (2014)
25. Wang, B., Chen, T.: Gaussian process regression with multiple response variables. Chemomet-
rics and Intelligent Laboratory Systems 142, 159–165 (2015)
26. Williams, C.K.: Computing with infinite networks. Advances in neural information processing
systems pp. 295–301 (1997)
27. Williams, C.K., Rasmussen, C.E.: Gaussian processes for regression. In: Advances in neural
information processing systems, pp. 514–520 (1996)
28. Wilson, A.G.: Covariance kernels for fast automatic pattern discovery and extrapolation with
Gaussian processes. Ph.D. thesis, PhD thesis, University of Cambridge (2014)
29. Zhang, F.: The Schur complement and its applications, vol. 4. Springer Science & Business
Media (2006)
30. Zhu, S., Yu, K., Gong, Y.: Predictive matrix-variate t models. In: Advances in Neural Informa-
tion Processing Systems, pp. 1721–1728 (2008)

A Negative Log Marginal Likelihood and Gradient Evaluation

A.1 Matrix derivatives

According to the chain rule of derivatives of matrix, there exists [13]: Letting U = f ( X ), the deriva-
tives of the function g(U ) with respect to X are
" T #
∂g(U ) ∂g(U ) ∂U
= tr ,
∂Xij ∂U ∂Xij

where X is an n × m matrix. Additionally, there are another two useful formulas of derivative with
respect to X.
∂ ln det( X ) ∂
= X −T , tr( AX −1 B) = −( X −1 BAX −1 )T ,
∂X ∂X
where X is an n × n matrix, A is a constant m × n matrix and B is a constant n × m matrix.

A.2 Multivariate Gaussian process regression

For a matrix-variate observations Y ∼ MN n,d ( M, Σ, Ω) where M ∈ Rn×d , Σ ∈ Rn×n , Ω ∈ Rd×d ,

the negative log likelihood is

nd d n 1
L= ln(2π ) + ln det(Σ) + ln det(Ω) + tr(Σ−1 (Y − M )Ω−1 (Y − M )T ), (18)
2 2 2 2

where actually Σ = K + σn2 I As we know there are several parameters in the kernel k so that we
can denote K = Kθ . The parameter set denotes Θ = {θ1 , θ2 , . . .}. Besides, we denote the parameter
matrix Ω = ΦΦT since Ω is positive semi-definite, where

φ11 0 · · · 0
 
 φ21 φ22 · · · 0 
Φ= . .. . . . .
 
 .. . . .. 
φd1 φd2 · · · φdd
Multivariate Gaussian and Student−t process regression for multi-output prediction 25

To guarantee the uniqueness of Φ, the diagonal elements are restricted to be positive and denote
ϕii = ln(φii ) for i = 1, 2, · · · , d. Therefore,

∂Σ ∂Σ ∂Kθ0 ∂Ω ∂Ω
= In , = , = Eij ΦT + ΦEij , = Jii ΦT + ΦJii ,
∂σn2 ∂θi ∂θi ∂φij ∂ϕii

where Eij is the d × d elementary matrix having unity in the (i,j)-th element and zeros elsewhere,
and Jii is the same as Eij but with the unity being replaced by e ϕii .
The derivatives of the negative log likelihood with respect to σn2 , θi , φij and ϕii are as follows.
The derivative with respect to θi is

∂L d ∂ ln det(Σ) 1 ∂
= + tr(Σ−1 (Y − M)Ω−1 (Y − M )T )
∂θi 2 ∂θi 2 ∂θi
" # " T #
∂ ln det(Σ) T ∂Σ ∂tr(Σ−1 G )

d 1 ∂Σ
= tr + tr
2 ∂Σ ∂θi 2 ∂Σ ∂θi
 0   0 
d ∂K 1 ∂K
= tr Σ−1 θ − tr Σ−1 GΣ−1 θ
2 ∂θi 2 ∂θi
 0  
∂K 0

d ∂K 1
= tr Σ−1 θ − tr αΣ Ω−1 αTΣ θ , (19)
2 ∂θi 2 ∂θi

where G = (Y − M)Ω−1 (Y − M)T and αΣ = Σ−1 (Y − M ).The fourth equality is due to the sym-
metry of Σ.
Due to ∂Σ/∂σn2 = In , the derivative with respect to σn2 is:

∂L d 1
= tr(Σ−1 ) − tr(αΣ Ω−1 αTΣ ). (20)
∂σn2 2 2

Letting αΩ = Ω−1 (Y − M )T , the derivative with respect to φij is

∂L n ∂ ln det(Ω) 1 ∂
= + tr(Σ−1 (Y − M )Ω−1 (Y − M)T )
∂φij 2 ∂φij 2 ∂φij
! " #
n ∂Ω 1 ∂Ω
= tr Ω−1 − tr ((Ω−1 (Y − M)T Σ−1 (Y − M)Ω−1 )T )T
2 ∂φij 2 ∂φij
! !
n ∂Ω 1 ∂Ω
= tr Ω−1 − tr αΩ Σ−1 αTΩ
2 ∂φij 2 ∂φij
n 1
= tr[Ω−1 (Eij ΦT + ΦEij )] − tr[αΩ Σ−1 αTΩ (Eij ΦT + ΦEij )], (21)
2 2

where the third equation is due to the symmetry of Ω. Similarly, the derivative with respect to ϕii
is

∂L n ∂ ln det(Ω) 1 ∂
= + tr(Σ−1 (Y − M )Ω−1 (Y − M )T )
∂ϕii 2 ∂ϕii 2 ∂ϕii
n 1
= tr[Ω−1 (Jii ΦT + ΦJii )] − tr[αΩ Σ−1 αTΩ (Jii ΦT + ΦJii )]. (22)
2 2
26 Zexun Chen et al.

A.3 Multivariate Student−t process regression

The negative log likelihood of observations Y ∼ MT n,d (ν, M, Σ, Ω) where M ∈ Rn×d , Σ ∈ Rn×n , Ω ∈
Rd×d , is
1
L= (ν + d + n − 1) ln det(In + Σ−1 (Y − M)Ω−1 (Y − M)T )
2
 
d n 1 1
+ ln det(Σ) + ln det(Ω) + ln Γn (ν + n − 1) + dn ln π
2 2 2 2
 
1
− ln Γn ( ν + d + n − 1)
2
1 ν+n−1
= (ν + d + n − 1) ln det(Σ + (Y − M)Ω−1 (Y − M)T ) − ln det(Σ)
2 2
   
1 1
+ ln Γn (ν + n − 1) − ln Γn ( ν + d + n − 1)
2 2
n 1
+ ln det(Ω) + dn ln π.
2 2
Letting U = Σ + (Y − M )Ω−1 (Y − M )T and αΩ = Ω−1 (Y − M )T , the derivative of U with
respect to σn2 , θi ,ν, φij and ϕii are

∂U ∂U ∂Kθ0 ∂U
= In , = , = 0, (23)
∂σn2 ∂θi ∂θi ∂ν

∂U ∂Ω −1 ∂Ω
= −(Y − M)Ω−1 Ω (Y − M)T = −αTΩ αΩ , (24)
∂φij ∂φij ∂φij
∂U ∂Ω −1 ∂Ω
= −(Y − M)Ω−1 Ω (Y − M )T = −αTΩ αΩ . (25)
∂ϕii ∂ϕii ∂ϕii

Therefore, the derivative of negative log marginal likelihood with respect to θi is

∂L (τ + d) ∂ ln det(U ) τ ∂ ln det(Σ)
= −
∂θi 2 ∂θi 2 ∂θi
 0  
∂K 0

(τ + d) ∂K τ
= tr U −1 θ − tr Σ−1 θ , (26)
2 ∂θi 2 ∂θi
where the constant τ = ν + n − 1.
The derivative with respect to σn2 is
∂L (τ + d) τ
= tr(U −1 ) − tr(Σ−1 ). (27)
∂σn2 2 2
The derivative with respect to ν is
∂L 1 1 1 1 1 1
= ln det(U ) − ln det(Σ) + ψn ( τ ) − ψn [ (τ + d)] (28)
∂ν 2 2 2 2 2 2
where ψn (·) is the derivative of the function ln Γn (·) with respect to ν.
The derivative of L with respect to φij is

∂L (τ + d) ∂ ln det(U ) n ∂ ln det(Ω)
= +
∂φij 2 ∂φij 2 ∂φij
(τ + d) n
=− tr[U −1 αTΩ (Eij ΦT + ΦEij )αΩ ] + tr[Ω−1 (Eij ΦT + ΦEij )]. (29)
2 2
Similarly, the derivative with respect to ϕii is
∂L (τ + d) n
=− tr[U −1 αTΩ (Jii ΦT + ΦJii )αΩ ] + tr[Ω−1 (Jii ΦT + ΦJii )]. (30)
∂ϕii 2 2
Multivariate Gaussian and Student−t process regression for multi-output prediction 27

B Three Chinese Stocks Investment Details

Table B1 The movement of invested $100 for 200 days split in to 20 periods (Stock: BIDU)

Buy&Sell decisions by prediction models Buy&Hold stock/index

Forecast terms
MV-GPR MV-TPR GPR TPR BIDU INDU NDX SPX

Beginning ($) 100 100

Period 1 103.53 100.97 103.53 100.97 97.14 101.20 98.70 100.71

Period 2 109.80 106.09 115.60 102.03 94.87 99.55 94.10 98.44
Period 3 108.37 104.71 115.96 100.71 93.89 101.50 96.64 100.62
Period 4 117.17 113.22 125.37 108.89 95.21 101.70 96.94 100.87
Period 5 127.84 123.52 136.79 118.80 102.22 102.22 100.79 102.55
Period 6 137.94 130.14 147.59 128.18 107.39 102.44 101.54 103.09
Period 7 146.20 137.93 156.43 135.86 112.11 103.12 103.25 104.54
Period 8 155.97 147.15 166.88 144.94 113.57 103.72 105.34 105.09
Period 9 177.94 167.88 175.27 152.21 138.22 103.82 106.98 105.67
Period 10 179.83 174.07 177.13 153.83 131.26 101.33 104.90 103.17
Period 11 179.08 173.35 176.05 153.19 130.71 104.07 109.34 106.20
Period 12 190.96 184.85 187.73 163.35 137.58 104.75 110.49 106.91
Period 13 201.08 194.63 204.44 177.89 131.12 105.12 109.57 106.52
Period 14 207.28 200.64 210.75 183.38 132.54 104.01 108.35 104.94
Period 15 210.54 203.80 214.06 186.26 132.19 100.39 104.41 101.70
Period 16 233.92 226.43 237.83 206.95 144.35 106.31 112.48 107.77
Period 17 252.47 233.71 256.69 223.36 148.99 108.03 113.68 109.03
Period 18 250.38 227.57 254.56 221.51 143.25 109.45 116.17 110.38
Period 19 260.24 223.38 264.59 230.23 134.23 104.49 110.33 105.37
Period 20 270.29 233.29 274.81 239.13 139.12 109.10 114.29 109.97

Table B2 The movement of invested $100 for 200 days split in to 20 periods (Stock: CTRP)

Buy&Sell decisions by prediction models Buy&Hold stock/index

Forecast terms
MV-GPR MV-TPR GPR TPR CTRP INDU NDX SPX

Beginning ($) 100 100

Period 1 105.67 105.67 105.67 105.67 100.78 101.20 98.70 100.71

Period 2 102.39 102.52 102.39 102.39 99.65 99.55 94.10 98.44
Period 3 102.77 102.90 102.77 102.77 91.63 101.50 96.64 100.62
Period 4 110.05 110.19 110.05 110.05 100.39 101.70 96.94 100.87
Period 5 121.51 121.66 121.51 121.51 108.33 102.22 100.79 102.55
Period 6 131.02 131.19 131.02 131.02 113.59 102.44 101.54 103.09
Period 7 138.90 139.08 144.25 138.90 120.90 103.12 103.25 104.54
Period 8 140.19 140.37 145.58 140.19 118.02 103.72 105.34 105.09
Period 9 150.29 146.38 151.82 146.20 131.35 103.82 106.98 105.67
Period 10 167.38 163.03 154.49 162.82 128.82 101.33 104.90 103.17
Period 11 166.33 162.01 154.28 161.80 127.37 104.07 109.34 106.20
Period 12 176.07 171.50 163.32 171.28 133.52 104.75 110.49 106.91
Period 13 176.00 171.43 163.25 171.21 117.53 105.12 109.57 106.52
Period 14 170.50 166.08 158.15 165.86 109.88 104.01 108.35 104.94
Period 15 178.68 174.04 165.73 173.82 108.35 100.39 104.41 101.70
Period 16 184.31 187.64 170.96 179.30 114.27 106.31 112.48 107.77
Period 17 183.77 191.85 176.72 180.87 115.96 108.03 113.68 109.03
Period 18 163.88 194.84 158.00 169.76 93.94 109.45 116.17 110.38
Period 19 169.89 201.99 163.80 176.73 80.69 104.49 110.33 105.37
Period 20 183.25 222.82 176.67 190.63 89.20 109.10 114.29 109.97

C Final Investment Details of Stocks in Dow 30

28 Zexun Chen et al.

Table B3 The movement of invested $100 for 200 days split in to 20 periods (Stock: NTES)

Buy&Sell decisions by prediction models Buy&Hold stock/index

Forecast terms
MV-GPR MV-TPR GPR TPR NTES INDU NDX SPX

Beginning ($) 100 100

Period 1 104.51 104.51 104.51 104.51 106.79 101.20 98.70 100.71

Period 2 106.19 106.19 106.19 106.19 106.35 99.55 94.10 98.44
Period 3 106.80 106.80 106.80 109.12 104.14 101.50 96.64 100.62
Period 4 115.90 115.90 115.90 114.28 108.66 101.70 96.94 100.87
Period 5 115.82 115.82 115.82 114.21 109.84 102.22 100.79 102.55
Period 6 120.96 117.65 120.73 115.20 116.55 102.44 101.54 103.09
Period 7 123.27 121.54 124.72 119.01 120.32 103.12 103.25 104.54
Period 8 127.77 125.33 128.62 122.73 117.34 103.72 105.34 105.09
Period 9 133.21 128.81 134.09 127.95 128.53 103.82 106.98 105.67
Period 10 133.36 128.96 134.24 128.09 127.40 101.33 104.90 103.17
Period 11 141.13 136.47 142.80 135.56 134.83 104.07 109.34 106.20
Period 12 141.45 136.78 139.44 135.87 137.23 104.75 110.49 106.91
Period 13 145.98 141.16 143.90 140.22 134.27 105.12 109.57 106.52
Period 14 147.95 144.00 145.84 143.04 129.90 104.01 108.35 104.94
Period 15 151.75 147.70 149.59 146.71 139.19 100.39 104.41 101.70
Period 16 158.59 154.36 156.33 153.33 144.80 106.31 112.48 107.77
Period 17 170.12 165.58 167.70 165.07 156.05 108.03 113.68 109.03
Period 18 177.19 171.24 174.67 171.93 162.04 109.45 116.17 110.38
Period 19 179.84 177.72 177.28 174.50 153.20 104.49 110.33 105.37
Period 20 188.32 176.70 180.31 177.48 153.52 109.10 114.29 109.97

Table B4 The detailed movement of invested $100 for last 10 days period (Period 20, Stock: BIDU)

Buy&Sell decisions by prediction models Buy&Hold stock/index

MV-GPR MV-TPR GPR TPR

Day BIDU INDU NDX SPX
Act Dollar Act Dollar Act Dollar Act Dollar

190 260.24 223.38 264.59 230.23 134.23 104.49 110.33 105.37

191 Buy 263.29 Buy 226.00 Buy 267.69 Buy 232.93 136.60 106.25 112.37 107.51
192 Keep 272.74 Keep 234.11 Keep 277.29 Keep 241.29 141.50 108.83 115.14 110.09
193 Keep 275.50 Keep 236.48 Keep 280.10 Keep 243.73 142.94 108.99 115.52 110.60
194 Keep 275.94 Keep 236.85 Keep 280.55 Keep 244.12 143.16 109.94 115.84 111.02
195 Sell 275.70 Sell 236.65 Sell 280.31 Sell 243.91 142.28 110.33 115.45 111.21
196 Buy 273.26 Buy 234.55 Buy 277.83 Buy 241.75 140.95 110.37 115.55 111.20
197 Keep 277.87 Keep 238.52 Keep 282.52 Keep 245.83 143.33 110.51 116.39 111.56
198 Keep 272.35 Keep 233.77 Keep 276.90 Keep 240.94 140.48 110.42 116.35 111.66
199 Sell 270.29 Keep 233.57 Sell 274.81 Sell 239.13 140.36 110.08 115.53 111.11
200 Keep 270.29 Sell 233.29 Keep 274.81 Keep 239.13 139.12 109.10 114.29 109.97

Table B5 The detailed movement of invested $100 for last 10 days period (Period 20, Stock: CTRP)

Buy&Sell decisions by prediction models Buy&Hold stock/index

MV-GPR MV-TPR GPR TPR

Day CTRP INDU NDX SPX
Act Dollar Act Dollar Act Dollar Act Dollar

190 169.89 201.99 163.80 176.73 80.69 104.49 110.33 105.37

191 Buy 175.02 Buy 208.09 Buy 168.75 Buy 182.07 83.65 106.25 112.37 107.51
192 Keep 180.77 Keep 214.92 Keep 174.28 Keep 188.05 86.39 108.83 115.14 110.09
193 Keep 185.40 Keep 220.43 Keep 178.75 Keep 192.87 88.61 108.99 115.52 110.60
194 Sell 184.91 Sell 220.28 Sell 178.28 Sell 192.36 88.55 109.94 115.84 111.02
195 Keep 184.91 Buy 222.82 Keep 178.28 Keep 192.36 88.45 110.33 115.45 111.21
196 Keep 184.91 Keep 222.82 Keep 178.28 Keep 192.36 88.22 110.37 115.55 111.20
197 Buy 185.16 Keep 222.82 Buy 178.52 Buy 192.62 88.92 110.51 116.39 111.56
198 Keep 184.06 Keep 222.82 Keep 177.46 Keep 191.47 88.39 110.42 116.35 111.66
199 Sell 183.25 Keep 222.82 Sell 176.67 Sell 190.63 88.45 110.08 115.53 111.11
200 Keep 183.25 Keep 222.82 Keep 176.67 Keep 190.63 89.20 109.10 114.29 109.97
Multivariate Gaussian and Student−t process regression for multi-output prediction 29

Table B6 The detailed movement of invested $100 for last 10 days period (Period 20, Stock: NTES)

Buy&Sell decisions by prediction models Buy&Hold stock/index

MV-GPR MV-TPR GPR TPR

Day NTES INDU NDX SPX
Act Dollar Act Dollar Act Dollar Act Dollar

190 179.84 177.72 177.28 174.50 153.20 104.49 110.33 105.37

191 Buy 176.57 Buy 174.49 Buy 174.06 Buy 171.33 151.35 106.25 112.37 107.51
192 Keep 184.84 Keep 182.66 Keep 182.21 Keep 179.36 158.44 108.83 115.14 110.09
193 Keep 184.16 Keep 181.98 Keep 181.54 Keep 178.69 157.86 108.99 115.52 110.60
194 Keep 185.46 Keep 183.27 Keep 182.82 Keep 179.95 158.97 109.94 115.84 111.02
195 Sell 185.33 Keep 179.25 Sell 182.69 Sell 179.83 155.49 110.33 115.45 111.21
196 Buy 187.22 keep 180.86 Buy 184.55 Buy 181.66 156.88 110.37 115.55 111.20
197 Keep 187.75 Keep 181.38 Keep 185.08 Keep 182.18 157.33 110.51 116.39 111.56
198 Sell 188.32 Keep 177.52 Keep 181.15 Keep 178.30 153.98 110.42 116.35 111.66
199 Keep 188.32 Sell 176.70 Sell 180.31 Sell 177.48 153.29 110.08 115.53 111.11
200 Keep 188.32 Keep 176.70 Keep 180.31 Keep 177.48 153.52 109.10 114.29 109.97

Table C1 The detailed stock investment results under different strategies

Buy&Sell Strategy Buy&Hold Stragegy

Ticker Industry
MV-GPR MV-TPR GPR TPR Stock INDU NDX SPX

CVX Oil & Gas 134.97 133.69 143.47 143.81 99.38

XOM Oil & Gas 128.39 132.72 131.31 136.02 99.74

MMM Industrials 166.76 162.96 167.12 162.65 125.96

BA Industrials 160.12 159.98 158.60 157.38 106.39
CAT Industrials 142.58 138.45 146.13 151.75 97.16
GE Industrials 137.51 134.63 135.35 139.72 101.15
UTX Industrials 144.29 139.47 143.29 145.18 101.94

KO Consumer Goods 128.11 128.59 124.88 124.52 112.47

MCD Consumer Goods 120.69 117.09 122.19 119.59 98.81
PG Consumer Goods 126.62 123.32 127.14 127.10 117.04

JNJ Health Care 146.00 146.70 147.42 145.16 113.65

MRK Health Care 129.40 134.48 129.36 135.05 102.45 109.10 114.29 109.97
PFE Health Care 128.60 136.53 130.26 134.48 100.16
UNH Health Care 164.98 164.63 166.14 162.79 131.14

HD Consumer Services 171.46 165.74 169.55 170.18 133.33

NKE Consumer Services 147.17 146.13 142.36 148.26 122.27
WMT Consumer Services 136.50 132.59 133.77 135.67 117.31
DIS Consumer Services 168.19 168.43 168.51 168.12 115.97

AXP Financials 160.39 158.52 160.73 160.12 102.34

GS Financials 170.46 171.29 167.71 165.72 116.16
JPM Financials 174.90 170.07 176.48 172.12 110.09
TRV Financials 149.81 145.88 150.70 145.71 128.18
V Financials 161.50 153.48 157.04 158.70 116.37

AAPL Technology 201.82 206.64 203.45 208.07 147.34

CSCO Technology 159.13 164.88 158.61 155.92 131.34
IBM Technology 116.10 128.79 124.92 123.74 88.06
INTC Technology 183.80 179.45 185.52 188.22 149.24
MSFT Technology 173.61 166.09 176.57 172.76 120.01

Table C2 The detailed industry portfolio investment results under different strategies

Buy&Sell Strategy Buy&Hold Stragegy

Industry Portfolio
MV-GPR MV-TPR GPR TPR Stock INDU NDX SPX

Oil & Gas 131.68 133.20 137.39 139.92 99.56

Industrials 150.25 147.10 150.10 151.34 106.52
Consumer Goods 125.14 123.00 124.73 123.73 109.44
Health Care 142.24 145.59 143.30 144.37 111.85 109.10 114.29 109.97
Consumer Services 155.83 153.22 153.55 155.56 122.22
Financials 163.41 159.85 162.53 160.47 114.63
Technology 166.89 169.17 169.81 169.74 127.20