This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LWC.2017.2778240, IEEE Wireless
Communications Letters

How Many Signals Can be Sent in A Multi-cell

Massive MIMO System
Haiquan Wang, Dakuan Yang, Xiao Li and Peng Pan

Abstract—In this letter, we study how large the signal con- To alleviate pilot contamination, a lot of works on channel
stellation can be used under pilot contamination in a multi-cell, estimation and signal detection were proposed, for example,
massive multiple-input, multiple-output (MIMO) uplink system. see [4]- [11]. In [4], it was shown that, under certain conditions
Specifically, based on an equivalent transmit-receive signal model,
we observe that the transmitted signals by all users in different on channel covariance, the pilot contamination effects can
cells are compacted into a low-dimensional space when they be removed when the number of transmit antennas goes
arrive at the target base station (BS). Then, the equivalent model to infinity. In [5], a new channel estimator was proposed
reveals that the size of constellations used by all users is limited based on subspace projection, and it was shown that the pilot
by a parameter, which is related to the large-scale fading factors contamination effects were mitigated under some conditions
only. We find that when the size of the signal constellation is
smaller than the parameter, the error probability goes to zero. on the coherence time. An eigenvalue decomposition based
Otherwise, an error floor occurs regardless of the number of channel estimation approach was proposed in [6]. More recent-
antennas and the pilot signal to noise ratio. Our simulations ly, the subspace based channel estimations were considered
results validate the proposed analytical results. and analysed in [7] for a finite-dimensional physical channel
Index Terms—Pilot reuse; Pilot contamination; Multi-cell, model, where signals impinge on the base station from a
massive MIMO. finite number of angles of arrival. In [8], pilot optimization
and channel estimation were formulated to an optimization
I. I NTRODUCTION problem and can be solved by using iterative algorithms. In [9],
pilot beam pattern design was investigated and a new algorithm
Massive multiple-input, multiple-output (MIMO) technol-
was proposed for the optimal channel estimation. Recently,
ogy, where the base station (BS) is equipped with a large
partially decoded data were used to aid channel estimation in
number of antennas and serves several single-antenna users
[10]. In [11], a multi-cell zero-forcing (ZF) detector which
at the same frequency band simultaneously, has attracted
exploits and orthogonalizes all available directions to mitigate
significant research interest recently. It has been demonstrated
parts of the inter-cell interference was proposed. Also, in
that the system can significantly increase the total throughput
[12], an iterative detector was considered to reduce decoding
with simple signal processing algorithms, if the channel state
complexity.
information (CSI) is available for the receiver. However, when
In this letter, for a multi-cell massive MIMO uplink system,
the CSI is not available, the performance will be greatly
we establish an equivalent transmit-receive signal transmission
degraded, see [1]- [3]. Thus, the great benefits of the massive
model. A detailed analysis on this model reveals the following
MIMO system are heavily based on the assumption that the
fact: if the size of the signal constellations used by all users
receiver has the CSI.
is smaller than a parameter, which only depends on the large-
Generally, the CSI is acquired from training-based channel
scale fading factors, the error floor does not exist when both
estimation. For an uplink, each user transmits pilot signals
the number of antenna in the BS and the signal-to noise (SNR)
to the BS, and then, the BS estimates the CSI based on the
of the pilots go to infinity. Otherwise, the error floor exists.
received pilots. Obviously, the channel estimation accuracy of
The rest of this letter is organized as follows: in Section
depends on the pilot signals. One of the pilot signal designs
II, system model is introduced. In Section III, an equivalent
is that the pilot signals transmitted by different users are
transmit-receive signal model is given, and an impact of this
mutually orthogonal, but the orthogonality consumes a lot of
model is analyzed. Simulation results are shown in Section IV.
radio resources. Hence, non-orthogonal pilot signals have been
Finally, the last section concludes the paper.
considered, where the users in different cells may use the same
Notations: For a matrix A, At denotes its transpose, AH
pilot signals, and the users in the same cell use orthogonal
denotes its conjugate transpose, det(A) denotes its determi-
pilots, referred to as “pilot reuse” [1]. In the pilot reuse,
nant, rank(A) denotes its rank, and tr(A) denotes its trace.
channel estimation is interfered by pilot signals transmitted
by the users in the neighboring cells, which degrades the II. S YSTEM M ODEL
performance of the estimator drastically. The phenomenon is
We consider uplink of a multi-cell, massive MIMO with L
called as pilot contamination [1].
cells. Assume that the BS at each cell is equipped with M
The authors are with School of Communications Engineering, Hangzhou antennas, and supports K independent users, each with single
Dianzi University, Hangzhou, China. (e-mail: [email protected], antenna. The channel gains consist of two parts: one contains
[email protected], [email protected],
[email protected]) This work is supported by National Natural path loss and shadowing effects (also called as the large-
Science Foundations of China under grant no. 61372093 and no. 61728105. scale fading factors), and the other is instantaneous fading

2162-2337 (c) 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LWC.2017.2778240, IEEE Wireless
Communications Letters

gain. Thus, the channel gain between the k-th user of the l-th III. A N E QUIVALENT M ODEL OF THE S YSTEM
cell√and the m-th antenna in the first BS can be expressed In this section, we derive an equivalent model of the system
as βmkl hmkl , where βmkl denotes the large-scale fading in Section II and then show some impacts on the system
based on the path-loss and shadowing, and hmkl represents the performance.
small-scale fading. Also, we assume that βmkl = βkl for all
m = 1, 2, · · · , M without loss of generality. The fading factor
hmkl is an independent and identically distributed (i.i.d.) zero- A. An equivalent model
mean circularly-symmetric complex Gaussian random variable Assume that Φl = Φ0 for l = 1, 2, · · · , L, where Φ0 is
with unit variance. All fading factors hmkl are assumed to be a K × K unitary matrix. That is, the pilots used by users
unknown to all the users and the BS, while the large-scale in the same cell are orthogonal but are reused in other cells.
fading factors are known to the BSs. Let U0 D0 V0H be the Singular Value Decomposition (SVD) of
The input-output equation of the system can be written as matrix BΦ, that is, BΦ = U0 D0 V0H , where U0 is a KL×K
√ unitary matrix, D0 is a K × K diagonal matrix and V0 is a
Y = ρHBX + W, (1)
K × K unitary matrix. Then, we get
where Y is the received signal vector in the BS, matrix B is  
B1
a diagonal matrix, which can be expressed as  B2  (
  )− 1
B = diag(B1 , B2 , · · · , BL ) U0 =  .  B21 + B22 + · · · + B2L 2 ,
 . .
√ √ √
and Bl = diag( β1l , β2l , · · · , βKl ). The M × KL chan- BL
nel matrix H can be represented as follows: ( 2 )1
D0 = B1 + B22 + · · · + B2L 2 ,
H = (H1 H2 · · · HL ) (2)
V0 = ΦH
0 . (8)
where Hl = (hmkl )M ×K . Finally, ρ is the average received
SNR, X is the transmitted signal vector, and W is the Define
∆
Gaussian noise matrix. P = U0 UH
0 . (9)
Consider that the system adopts the training-based trans- Obviously, P2 = P, PH = P and rank(P) = rank(Φ) = K.
mission scheme, where the signal transmissions are divided Thus, P is a projection matrix. It projects the KL-dimensional
into two phases. At the first phase, the k-th user in the l- complex Euclidean vector space onto the subspace generated
th cell sends τ pilot signals Φkl = (ϕkl,1 , ϕkl,2 , · · · , ϕkl,τ ), by columns of matrix U0 , which is equal to the space
k = 1, 2, · · · , K and l = 1, 2, · · · , L. Denote generated by columns of matrix BΦ. Hence, PBΦ = BΦ.
Φl = (Φt1l , Φt2l , · · · , ΦtKl )t , Φ = (Φt1 , Φt2 , · · · , ΦtL )t Next, we reconsider (1).
(3) √ √
Y = ρHPBX + ρH(IKL − P)BX + W
Thus, matrix Φl of size K × τ and matrix Φ of size LK × τ √
denote the pilot signals transmitted by the users in the l-th = ρHU0 UH 0 BX + Wτ
√ H
cell and by users from all cells, respectively. We normalize = ρHτ U0 BX + Wτ , (10)
the energy of the pilot signals as follows:
where
Φkl ΦH
kl = 1, and hence tr(ΦΦH ) = KL. (4) √
Hτ = HU0 , Wτ = ρH(IKL − P)BX + W. (11)
The minimum mean square error (MMSE) estimator of
channel matrix H, denoted as Ĥ, can be written as [13] By using (8), the above can be written as
√ ( )−1 H H  β11 x11 +β12 x12 +···+β1L x1L 
√
Ĥ = ρ0 Y0 Iτ + ρ0 ΦH B2 Φ Φ B , (5) β11 +β12 +···+β1L
 β21 x√
21 +β22 x22 +···+β2L x2L 
√  
where ρ0 is the average SNR. Y = ρHτ   +Wτ . (12)
β21 +β22 +···+β2L
 .. 
At the second phase, users in different cells transmit their  . 
own data signals simultaneously. Denote the signals transmit- βK1 xK1
√ +βK2 xK2 +···+βKL xKL
ted by users in the l-th cell as Xl = (x1l , x2l , · · · , xKl )t , βK1 +βK2 +···+βKL

where xkl represents the signal transmitted by the k-th user Equation (12) shows that the transmitted signals of all
in the l-th cell. Denote X = (Xt1 , Xt2 , · · · , XtL )t . Then, the users, which can be viewed as vectors in the KL-dimensional
MMSE signal detection is [13] complex space, is compacted into a K-dimensional space.
√
x̂k1 = arg min ∥Gmmse,k Ỹ − ρ̃xk ∥, (6)
xk
B. On the size of constellations used by all users
where Ỹ = Y/σ, and Gmmse,k denotes the k-th row of matrix Assume that each user in the considered system uses the
Gmmse and following signal constellation:
( )−1
IKL QAMkl = {z | z = n1 d + jn2 d, n1 , n2 ∈ N } (13)
Gmmse = + BH ĤH ĤB BH ĤH , (7)
ρ̃ √
where d = 3/(2(4N 2 − 1)) and
and σ 2 = 1 + ρtr(BA−1 B), A = IKL + ρ0 BΦΦH BH ,
ρ̃ = ρ/σ 2 . N = {−(2N −1), −(2N −3), · · · , −1, 1, · · · , 2N −3, 2N −1}.

2162-2337 (c) 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LWC.2017.2778240, IEEE Wireless
Communications Letters

L=K=3, ρ=4dB, N=2, 16−QAM

Thus, QAMkl is the standard 4N 2 -QAM with total energy 10
0

4N 2 . Define −1
10
∆ βk1 1
γk = + . (14)

Symbol Error Probability

2(βk2 + βk3 + · · · + βkL ) 2 10
−2

for k = 1, 2, · · · , K. Then, we have the following main results. −3

10
Theorem 1: Assume that the first BS decodes the signals
with the MMSE decoder and each user in cells uses 4N 2 - 10
−4
The first user
QAM constellation given in (13). If N < γk , then The second user
The third user
−5
10
50 100 150 200 250 300 350 400 450 500
lim lim Pe,k = 0, (15) Number of Antennas
M →+∞ ρ0 →+∞

and if N > γk , then Fig. 2. Performances on large scale factors and antenna numbers (II).

lim lim Pe,k ≥ δ > 0, (16) L=K=3, ρ=4dB, N=4, 64−QAM

M →+∞ ρ0 →+∞ 0
10

where Pe,k is the error probability of user k, and δ is a positive

constant.

Symbol Error Probability

A sketch proof of Theorem 1 is given in Appendix.
Theorem 1 shows that if the k-th user in each cell uses −1
10

4N 2 -QAM and N < γk , the error floor does not exist when
both ρ0 and M go to infinity. Otherwise, the error floor exists.
The first user
The second user
IV. S IMULATION R ESULTS −2
10
The third user
50 100 150 200 250 300 350 400 450 500
In this section, we show some simulation results to verify Number of Antennas

analysis presented in Section III. In all following simulations,

Fig. 3. Performances on large scale factors and antenna numbers (III).
we assume that the system has three cells, and three users in
each cell. That is, L = K = 3. Let Φ = (I3 I3 I3 )T . Matrix
B in (1) is selected as follows:
does not exist for all three users. It is confirmed in Fig. 1.
B = diag(0.9511, 0.7371, 0.6860, When N = 2, condition N < γk is satisfied for user 2 and
0.6171, 0.3100, 0.1537, 0.1457, 0.1379, 0.0833). user 3. Thus, for these two users, the error floors do not exist,
but it does exist for user 1. Fig. 2 shows these phenomena.
Bases on (14), we have Finally, when N is equal to 4, condition N < γk is satisfied
γ1 = 1.6250, γ2 = 2.8599, γ3 = 8.1989. for user 3 only. Thus, the error floor does not exist for user 3
only. This is again confirmed by Fig. 3.
We perform the simulations of three cases. The first case
is that N = 1, that is, the constellation is 4-QAM, and the V. C ONCLUSIONS
SNR ρ is fixed to −10dB. The second case is that N = 2 (the
constellation is 16-QAM), and ρ = 4dB. The last case is that In this paper, we studied the maximum number of signals
N = 4 (the constellation is 64-QAM), and ρ = 4dB. In the that can be supported in a multi-cell massive MIMO system.
three cases, antenna number M varies from 50 to 500, and By deriving an equivalent transmit-receive signal model, we
ρ0 = 40dB. The simulation results are shown in Fig. 1, Fig. found that if the size of the signal constellation is smaller
2 and Fig. 3, respectively. than a parameter, which is related to the large-scale fading
factors only, the error probability goes to zero. Otherwise, an
0
L=K=3, ρ=−10dB, N=1, 4−QAM error floor occurs regardless of the number of antennas and
10
the pilot SNR.
−1
10

A PPENDIX
Symbol Error Probability

−2
10

In this appendix, we give a sketch of proof of Theorem 1.

−3
10 First, in the pilot training phase, from (10), we see that when
X = Φ and ρ0 = ρ, the received pilot signal Y0 can be
−4
10 The first user
written as
The second user √
−5
10
The third user Y0 = ρ0 Hτ D0 Φ0 + W. (17)
50 100 150 200 250 300 350 400 450 500
Number of Antennas
Since the rank of channel matrix Hτ is equal to the rank of the
Fig. 1. Performances on large scale factors and antenna numbers (I). pilot matrix, according to the results given in [13], we have
∑
K
1
In the first case, condition N = 1 < γk is satisfied for EHτ [∥Hτ − Ĥτ ∥] = M , (18)
k = 1, 2, 3. From Theorem 1, we deduce that the error floor 1 + ρ0 λk
k=1

2162-2337 (c) 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/LWC.2017.2778240, IEEE Wireless
Communications Letters

∑L Ĥτ is the MMSE estimation of Hτ , and λk =

where Consequently, we get
l=1 βkl . Thus, if ρ0 goes to infinity, EHτ [∥Hτ − Ĥτ ∥] goes √ βk1 xk1 + βk2 xk2 + · · · + βkL xkL
to zero, and channel Hτ can be perfectly estimated. Hence, in ŷk = ρτ √ + ŵk , (29)
βk1 + βk2 + · · · + βkL
the following, we assume that Hτ is perfectly known to the
G Y
first BS when ρ0 goes to infinity. Since where ŷk = m,k στ , and wˆk is the corresponding noise.
( ) ( ) Hence, the decoder (27) is reduced into
E Hτ HH τ = E HU0 UH 0 H
H
= τ IM , (19)
x̂k1 = argminx∈QAMk1
and √
( ) ( H H ) ρτ βk1 x
E HH
τ Hτ = E U0 H HU0 = M Iτ , (20) ŷk − √ . (30)
βk1 + βk2 + · · · + βkL
we know that components of Hτ are i.i.d. Gaussian random Assume that N > γk . Then, (24) does not hold, and
variables with zero mean and unit variance. By the large Cov(Ck,x )∩Cov(Ck,x′ ) ̸= ∅ if x and x′ are neighboring points
number theorem, we have in QAMk1 . Thus, when the k-th user in the first cell transmits
HH signal x ∈ QAMk1 , there always exist linear combinations
τ Hτ a.e.
lim = Iτ , (21)
M →+∞ M √ βk1 x + βk2 xk2 + · · · + βkL xkL
ρτ √
where a.e. means that the equality holds with probability one. βk1 + βk2 + · · · + βkL
In the signal transmission phase, (12) indicates that the with xk2 ∈ QAMk2 , · · · , xkL ∈ QAMkL , such that these
considered system can be viewed as a “virtual system”, where points lie outside the decision region of x. Thus, decoding
a BS with M receiving antennas receives signals sent from K errors occur. Moreover, these error events are independent of
different users with single antenna, and the k-th user uses the M and ρ0 , we have proved the theorem 1.
following constellation:
∆
∪ R EFERENCES
Ck = Ck,x , (22)
x∈QAMk1
[1] T. L. Marzetta, “Noncooperative cellular wireless with unlimited num-
bers of base station antennas," IEEE Trans. Wireless Commun. , vol. 9,
where no. 11, pp. 3590-3600, Nov. 2010.
{ } [2] F. Rusek, D. Persson, B. K. Lau, E. G. Larsson, T. L. Marzetta, O. Edfors
βk1 x + βk2 xk2 + · · · + βkL xkL
∆ and F. Tufvesson, “Scaling up MIMO: opportunities and challenges with
Ck,x = √ | zkl ∈ QAMkl . very large arrays," IEEE Signal Processing Magazine, vol. 30, no. 1, pp.
βk1 + βk2 + · · · + βkL 40-60, Jan. 2013.
(23) [3] H. Q. Ngo, E. G. Larsson and T. L. Marzetta, “Energy and spectral
Assume that N < γk . Then efficiency of very large multiuser MIMO systems," IEEE Trans. on
Commun. , vol. 61, no. 4, pp. 1436-1449, April 2013.
βk1 − (2N − 1)(βk2 + · · · + βkL ) [4] H. Yin, D. Gesbert, M. Filippou, and Y. Liu, “A coordinated approach
√ > 0. (24)
βk1 + βk2 + · · · + βkL to channel estimation in large-scale multiple-antenna systems," IEEE J.
Sel. Areas Commun., vol. 31, no. 2, pp. 264-273, Feb. 2013.
Thus, set Cov(Ck,x ) and set Cov(Ck,x′ ) are separated with [5] R. Müller, L. Cottatellucci, M. Vehkaperä, “Blind pilot decontamina-
tion," IEEE J. Sel. Topics on Signal Process., vol. 8, no. 5, pp. 773-786,
the minimal distance at least 2γ̄k d if x ̸= x′ , where Cov(A) Oct. 2014.
denotes the convex set generated by points in set A. Thus, [6] Q. N. Hien and E. G. Larsson, “EVD-based channel estimation in
according to results presented in [14], we have multicell multiuser MIMO systems with very large antenna arrays,"
ICASSP, 2012, Kyoto, Japan, 2012, pp. 3249-3252.
[7] M. Teeti, J. Sun, D. Gesbert, and Y. Liu, “The Impact of Physical
lim lim Pe,k = 0. Channel on Performance of Subspace-Based Channel Estimation in
M →+∞ ρ0 →+∞
Massive MIMO Systems”, IEEE Trans. Wireless Commun, vol. 14, no.
Let us consider the converse part of the theorem. From (11), 9, pp: 4743-4756, Sept. 2015.
it is easy to get that the covariance of Wτ is [8] T. E. Bogale and L. B. Le, “Pilot optimization and channel estimation
for multiuser massive MIMO systems,” in Proc. 48th Annu. Conf. Inf.
∆ Sci., Syst. (CISS), Princeton, NJ. USA, Mar. 2014, pp. 1-6.
στ2 = 1 + ρtr(BH (IKL − P)B). (25) [9] S. Noh, M. Zoltowski, Y. Sung and D. J. Love, “Pilot beam pattern
design for channel estimation in massive MIMO systems,” IEEE J. Sel.
Thus, the MMSE decoder, denoted as Gm , is Topics Signal Process., vol. 8, no. 5, pp. 787-801, Oct., 2014.
( )−1 [10] J. Ma and L. Ping, “Data-aided channel estimation in large antenna
Iτ H systems”, IEEE Trans. Signal Process., vol. 62, no. 12, pp. 3111-3124,
Gm = + Hτ Hτ HH τ , (26) Jun. 2014.
ρτ [11] E. Björnson, E. G. Larsson, and M. Debbah, “Massive MIMO for
maximal spectral efficiency: how many users and pilots should be
and the decoder is allocated?” IEEE Trans. Wireless Commun, vol. 15, no. 2, pp: 1293-
1308, Feb. 2016.
x̂k1 = argminx∈QAMk1
√ [12] J. Choi, B. Lin, “MMSE-based random sampling for iterative detection
Gm,k Y ρτ βk1 for large-scale MIMO systems.” Journal of communications and infor-
−√ x , (27) mation networks, vol. 1, no. 2, pp. 29-36, Aug. 2016.
στ βk1 + βk2 + · · · + βkL [13] H. Wang, W. Zhang, Y. Liu, Q. Xu and P. Pan, “On design of non-
orthogonal pilot signals for a multi-cell massive MIMO system", IEEE
where Y is given in (12), and ρτ = ρ/στ2 . Wireless Communications Letters, vol. 4, no. 2, pp. 129-132, April 2015.
When M goes to infinity, using (21), we have [14] H. Wang, M. Zhou, R. Chen and W. Zhang, “On error rate performance
of multi-cell massive MIMO systems with linear receivers", Phisical
a.e. Communication, vol. 20, pp. 123-132, 2016.
lim Gm Hτ = Iτ . (28)
M →+∞

2162-2337 (c) 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.