A NALYSIS OF MARKET EFFICIENCY IN MAIN STOCK MARKETS :

USING K ARMAN -F ILTER AS AN APPROACH

Beier Liu Haiyun Zhu

Stern School of Business Nanyang Business School
arXiv:2404.16449v1 [q-fin.CP] 25 Apr 2024

New York University Nanyang Technology University

New York Singapore
[email protected] [email protected]

A BSTRACT
In this study, we utilize the Kalman-Filter analysis to assess market efficiency in major stock markets.
The Kalman-Filter operates in two stages, assuming that the data contains a consistent trendline
representing the true market value prior to being affected by noise. Unlike traditional methods, it
can forecast stock price movements effectively. Our findings reveal significant portfolio returns in
emerging markets such as Korea, Vietnam, and Malaysia, as well as positive returns in developed
markets like the UK, Europe, Japan, and Hong Kong. This suggests that the Kalman-Filter-based
price reversal indicator yields promising results across various market types.

Keywords Kalman-Filter; Market efficiency

1 Introduction

1.1 A Review of Short-term Reversal and Market Efficiency

Modern financial theory rests on the efficient market hypothesis (EMH), which assumes that investors are rational.
Empirically, EMH has been challenged by some “anomalies”, and with the rise of behavioral finance, the traditional
“market efficiency” hypothesis has been called into question. Although stock trading should be based on stocks’ intrinsic
values, many investors act impulsively and detract from the market’s efficiency. Noise trading and herding behavior are
two examples of common irrational actions. It rejects the “rational economic man” assumed in conventional economic
and financial research in favor of the more nuanced “bounded rationality” assumption and the individualization of
investors.
Mid-to long-term stock price overreaction was first noticed and proposed by DeBondt & Thaler (1985)[1] in the
US stock market. They proposed momentum trading and reversal trading, and show that using the latter during an
overreaction will lead to a positive stock return. Furthermore, for more than 40 years, the short-term return reversal has
been a recognized phenomenon. According to Jegadeesh (1990)[2], a reversal strategy that buys and sells stocks based
on their previous month’s returns and holds them for a month produced profits of roughly 2% per month between 1934
and 1987.
Some research attributed the inefficiency of the stock market due to the irrationality of retail investors. Generally
speaking, institutional investors have more resources, more expertise, and easier access to transaction data than retail
investors do. Due to institutional investors’ higher herding behavior and feedback trading tendencies than individual
investors, Dennis & Strickland (2002)[3] showed that equities having a bigger fraction of fund holdings in the market
tended to have higher returns. And groups of individual investors often create stock price swings, whereas institutional
investors help to stabilize prices. Similar results were found in follow-up studies by Barber, Odean, and Zhu (2009)[4].
Beier Liu and Haiyun Zhu, 2024

1.2 A Brief Overview of Market Efficiency of Emerging Markets

Much less research into EMH has been done in developing markets. Moreover, emerging markets are believed to have
higher friction, lower liquidity, and asymmetric information due to less efficient information transmission and less
reliable institutional information (Bekaert and Harvey, 2002[5];Griffin, Kelly, and Nardari,2010[6]). Because of this
information deficiencies in emerging nations, the newly established stock market is inefficient, and market efficiency
generally follows the norms of dynamic evolution (Cornelius, 1993[7]).
Recently, only a few of studies have put EMH to the test in isolated emerging economies. However, limitations
associated with small sample sizes render these investigations unconvincing. The Indian market was found to be
inefficient by Hiremath and Kumari(2014)[8], although they also found that it was improving with time. For the Czech
market, Arltova (2000)[9] employed variance ratio tests to disprove the random walk hypothesis. When Seth and
Sharma (2015)[10] examined the informational efficiency and integration of several markets in Asia and the US, they
found weak inefficiencies. In an effort to reconcile behavioral finance and market efficiency, Xiong et al. (2019)[11]
demonstrated that while the Chinese stock market is efficient, the "Adaptive Market Hypothesis" provides a more
accurate representation of the market. To date, there has been a lack of specificity in the research on improving market
efficiency.

1.3 The Basic Idea of Testing the Market Efficiency

According to the efficient market hypothesis (weak-form), stock prices fluctuate randomly or that it is impossible to
forecast future prices based on historical data. Tests for independence are used to assess returns in order to assess this
hypothesis. A martingale sequence is frequently used in probability theory to characterize a generalized random walk.
The variance-ratio methodology (Charles and Darné, 2009[12]), the autoregressive time series test (test of randomness
including the runs test), and the unit root tests (Dickey–Fuller, Phillip Perron, Ng–Perron, Elliot–Rothernbert–Stock)
have all been used in previous studies.
However, traditional tests perform very poorly, according to Monte Carlo studies by Elliot, Rothenberg, Stock (ERS),
and NG–Perron, among others. Additionally, several studies addressed this by using a different test called BDS in order
to properly study independently and identically distributed (IID) data, as traditional tests did not take into account the
nonlinearity of the data (Broock et al., 1996[13]).

1.4 Why We Choose Kalman Filter as a Test of Market Efficiency

There are several ongoing debates on how to best measure market efficiency, notwithstanding recent advances in this
area. Beginning in the 1980s, academics began exploring the impact of investor behavior on market fluctuations. Shiller
(1981)[14] noticed that S&P 500 index’s return changed whenever fresh information entered the market. With the
phrase “adaptive market hypothesis” (AMH), Lo provided a novel explanation for efficiency in financial markets in
2004. According to Lo’s theory, the market may take some time to adjust to changing conditions since investor behavior
is not viewed as purely rational but rather as the result of a number of interrelated psychological elements (for examples,
see Barber and Odean (2000)[15] and Shleifer (2000)[16]). Studies of investors’ attitudes are a modern addition to this
field of study.
Following Lo’s development of the idea, methods for examining market efficiency underwent modifications. It appears
that perceptions of the degree of market efficiency have shifted as a result of advancements in machine learning
information extraction algorithms and the accessibility of both quantitative and qualitative data, such as that found on
social media.
As a result, it’s possible that the confirmed markets’ efficiency has since been verified. As a result, the field of finance
has a wealth of literature devoted to modeling and forecasting financial time series using statistics and mathematics.
The techniques employed range from traditional statistical and econometric approaches to more contemporary machine
learning (ML) and evolutionary-based optimization strategies; see e.g. Kanas and Yannopoulos (2001)[17] and Rather
et al. (2015)[18].
Price fluctuations on a daily basis are viewed as a white-noise process, or Brownian motion, in terms of modern financial
engineering principles. Because white noise is a random process in which successive values are independent of each
other, it follows that daily price increases and decreases have an equal chance of occurring. In actuality, it is not unusual
for a market item to experience a stream of consecutive down days or up days over a brief period of time. The prices are
said to be correlated when such event occurs. Our goal is to employ a Kalman Filter on these correlations to foretell
future price changes. We will build upon our prior work here by using a Kalman Filter in place of our previous simple
one-day predictor. As used here, the Kalman Filter is a two-stage technique predicated on the assumption that the data
contains a smooth trendline representing the true value of the market item before being affected by market noise. And

2
Beier Liu and Haiyun Zhu, 2024

this filter be used to forecast stock price movement. A previous paper by the author (Li X. and Li W., 2000)[19] has
explored the evolution of China’s stock market efficiency for the period between May 1992 and March 2000.

1.5 What We Have Found in the Data

In our analysis, we may begin by introducing the data and the fundamental concept of Karman Filter analysis. Next, we
may present the results of evaluations of market efficiency on the major stock exchanges. Finally, for developed and
emerging stock markets, the short-term and long-term adaptation of market efficiency.
Test on Market Efficiency: After constructing portfolio from Kalman Filter model and checking backtest statistics, it is
easy to verify that the Kalman Filter strategy doesn’t work in US but works well in emerging markets such as Korea,
Malaysia and Vietnam. Meanwhile, the strategy performance decays significantly in UK, Europe, Japan, Hong Kong in
recent years, which may indicate improvement of price discovery efficiency in those markets.

2 Data and methodology

We obtain the constituents of each stock exchange’s main board index from WIND and the exchange official website.
And we then obtain the stock price from Yahoo Finance. Exchanges characteristics are shown in 1.

country code exchange type main board index number of stocks data history
US DM SPY 504 2017.01 - 2024.03
UK DM FTSE100 100 2017.01 - 2024.03
EU DM STOXX500 500 2017.01 - 2024.03
JP DM TOPIX2000 2159 2017.01 - 2024.03
HK DM HSCI+HSI 523 2017.01 - 2024.03
KR EM KOSPI1000 1000 2017.01 - 2024.03
VN EM VNX 415 2023.01 - 2024.03
MY EM MYX 95 2017.01 - 2024.03
ESG EM CB 168 2017.01 - 2024.03

Table 1: Key Information of Stocks Exchanges.

2.1 Dataset

We use Yahoo finance api to download stock market data (included open, high, low, close, volume) from 2018.01 to
2024.03. We have collected the market data for most comprehensive universe/main board index constituents for major
developed markets (US, UK, Europe, Japan, Hong Kong), and emerging markets (Korea, Malaysia and Vietnam).

2.2 Karman Filter

The Kalman Filter is a recursive algorithm developed in the 1960s to estimate the future stage of a moving target
by measuring its current position. When applied to financial market data, this technology transforms noisy price
observations into a sequence of intrinsic value hidden states. This implies that the estimated result and the current
observation alone can be used to calculate the estimate for the current state. It does not require any prior estimates or
observations, in contrast to batch estimation techniques.
Two variables are used to represent the filter’s state: 2) Pk|k , the a posteriori estimate covariance matrix (a measure of
the estimated accuracy of the state estimate); 1) Xk|k , the a posteriori state estimate mean at time k given observations
up to and including at time k. We will refer to the state transition matrix Fk , measurement matrix Hk , and observation
variable zk for clarity’s sake.
The alpha-beta filter and the Kalman-Filter share a similar algorithmic structure. Despite the fact that the Kalman-zFilter
can only be expressed as one equation,the two stages that are typically considered to be involved are "Predict" and
"Update." To be more precise, the predict phase creates an estimate of the state at the current timestep using the state
estimate from the preceding timestep.
The equation for (a priori) anticipated state estimation is:
xk|k−1 = Fk xk−1|k−1 + Bk uk

And the equation for (a priori) estimate covariance estimation is:

3
Beier Liu and Haiyun Zhu, 2024

Pk|k−1 = Fk Pk−1|k−1 FkT + Qk

This predicted state estimate is also referred to as the a priori state estimate since it is an estimate of the state at the
current timestep but does not include observational data from the current timestep. During the update phase, the state
estimate is refined by adding the innovation (the pre-fit residual) to the previous estimate and multiplying it by the
optimal Kalman gain. The term "a posteriori state estimate" refers to this improved estimate that was obtained from the
current observation.
Where the estimation of Innovation or measurement pre-fit residual is:
yk = zk − Hk xk|k−1
Innovation (or pre-fit residual) covariance Sk can be estimated as:
Sk = Hk Pk|k−1 HkT + Rk

And the optimal Kalman gain Kk can be calculated as

Kk = Pk|k−1 HkT Sk−1
. In the update phrase, we will update a posteriori state using estimation below:
xk|k = xk|k−1 + Kk yk
And the covariance matrix can be updated as:
Pk|k = (I − Kk Hk )Pk|k−1
Where the post-fit residual can be measured as:
yk|k = zk − Hk xk|k
The two stages usually happen in turn, with the update taking into account the observation and the prediction pushing
the state forward until the following planned observation. In our estimation, the predicted stock price can be obtained
from xk|k−1 , and we assume that the noise coveriance matrix is identity matrix.

2.3 Model and Stock Return Forecast

For each day and each stock, we will feed the latest close price(C) to Karman-Filter model as observations, and we
retrieve the true state value of today’s close (Ckalman ). With the assumption that the next day stock price will mean
revert to true state value, our stock return forecast for next day will hence be (C − Ckalman )/((C + Ckalman ) ∗ 0.5).

2.4 Portfolio Construction

We will construct an equity long-short portfolio based off our return forecast. Firstly, in the portfolio construction stage,
for each country, we will normalized our stock return forecast to guarantee mean zero and unit standard deviation. Then,
we will daily re-balance our portfolio to make sure each stocks’ daily dollar position is proportional to normalized stock
return forecast. Following these steps, we will run a portfolio based on our Kalman-Filter price reversal strategy, of
which we will calculate simulated back-test statistics.

3 Results
We apply the Karman-Filter analysis to both developed and emerging stock markets, including the United States, the
United Kingdom, Europe, Japan, Hong Kong, South Korea, Vietnam and Malaysia. Overall, the US market is the most
efficient compared to other markets. The y-axis, ROG(bps), represents the daily return on gross value in basis points.
T 1 indicates the daily return on next trading day, and Tj means the daily return on T + j trading day. Regardless
long-term or short-term, we are unable to profit from the U.S. market. The United States, the United Kingdom, and
Japan have the lowest returns, indicating that their markets are the most efficient. And Korea’s returns are greater in the
extended term. Surprisingly, Vietnam’s return decreased over the next five trading days, but increased over the extended
term. Despite Hong Kong’s high stamp duty, its return is not as high as other stock markets, such as Vietnam and Korea.
Next, we calculate the stock return for each less efficient stock market to determine if there is a time trend in their daily
return, where execution1 represents the return from the market open to T+1’s market open, execution2 represents the

4
Beier Liu and Haiyun Zhu, 2024

Figure 1: Country Portfolios ROG(return on gross) Plot

return from the market close to T+1’s market close, and execution3 represents the return from the market open to same
day’s market close. Generally execution3 generates the lowest return among all, and execution1 generates the highest
return. We use metrics like return on gross (ROG) and sharpe-ratio (SR) to compare strategy historical performance.
Since the indication is the measurement of mean reversal to real value anticipated by Karman-Filter on noise price
movements, from the images below. Several patterns are discernible.
The three execution strategies all generate positive returns in all markets except for the US consistently after ignoring
the returns from the COVID period, which shows that most markets are inefficient for the majority of the time. The first
universal “reversal collapse” occurs in 2020.03, which is the start of COVID and most international long short hedge
funds had major draw downs in the same period.
Additionally, by observing each market separately, we may compare the effectiveness of the three strategies depending
on their various execution techniques. execution1 approach is typically the most most difficult to execute in real
financial markets because, during this open market period, the market is digesting a lot of overnight information in the
opening auction, and the market capacity is the smallest when auction trading volume is taken into account. However,
due to the lack of intra-day minute-level data, the execution2 approximation is a practical approximation of the real
market because it accounts for 10% to 20% of the daily trading volume and most importantly, it takes all day to execute.
In reality, traders frequently use time-weighted average price (TWAP) or volume average price (VWAP) and trade in
a certain interval to eliminate the price impact. And the execution3 return can be viewed as the difference between
the returns of the execution1 and a hyper-theoretical overnight portfolio. Therefore, the ability to arbitrage should
decline first with execution2 and then execution1 (which is ordered by the market capacity) if the trading signal of
this Kalman Filter strategy decays, which indicates that the market is becoming more efficient.
The various patterns in the worldwide markets shown below show that the US market 2 (ROG -0.12bps, SR -0.04) is the
most effective because even in execution1 strategy, where the market capacity is the smallest, there is a tiny amount of
room for arbitrage.

Figure 2: Strategy Backtest in USA

And for Europe(ROG 1.15bps, SR 0.47) Fig3, the United Kingdom(ROG 2.25bps, SR 0.75) Fig4, Japan(ROG 2.32bps,
SR 1.32) Fig5, Hong Kong(ROG 1.29 SR 0.45) Fig6, and ESG indexes(ROG 1.92, sr 0.69) Fig10, we can see that

5
Beier Liu and Haiyun Zhu, 2024

the execution1 strategy hardly ever makes a profit while the execution2 strategy has a limited amount of room for
arbitrage when the market capacity is smaller. And out of the four markets, Hong Kong and Japan are the most alluring
due to the largest divergence between the execution1 and execution2 strategies, specifically the higher return of the
execution3 strategy (green line), and the highest SR ratio among the four markets.

Figure 3: Strategy Backtest in Europe

Figure 4: Strategy Backtest in United Kingdom

Figure 5: Strategy Backtest in Japan

Aside from that, the least efficient markets include those in Korea (ROG 9.33, SR 3.18) Fig7, Vietnam(ROG 19.01, SR
2.73) Fig8, Malaysia(ROG 12.76, SR 2.51) Fig9 since there is still room for arbitrage in open auction, regular trading,
and intraday trading. Overall, from the standpoint of efficiency, we can see that the market is becoming more efficient,
even in developed countries.

6
Beier Liu and Haiyun Zhu, 2024

Figure 6: Strategy Backtest in Hong Kong

Figure 7: Strategy Backtest in Korea

Figure 8: Strategy Backtest in Vietnam

7
Beier Liu and Haiyun Zhu, 2024

Figure 9: Strategy Backtest in Malaysia

Figure 10: Strategy Backtest in ESG Index

4 Conclusion and Discussion

In this paper, we applied the Kalman-Filter analysis to test the market efficiency of major stock markets across developed
and emerging markets. The key findings are:
First, even in execution techniques with smaller market capacity, the Kalman-Filter approach does not yield appreciable
gains, suggesting that the U.S. market is the most efficient. Although there is some degree of inefficiency in other
established markets, such as the UK, Europe, Japan, and Hong Kong, the execution2 strategy—which is based on
closing the market close to the next day’s close—generates statistically positive returns albeit tiny ones. With greater
return divergences across various execution techniques, Hong Kong and Japan stand out. All three execution strategies
show significant return premiums for emerging countries, such as Korea, Vietnam, and Malaysia, suggesting a higher
degree of market inefficiency in these areas.
Moreover, when returns of the Kalman-Filter approach is decreasing, particularly during times like the COVID-19
outbreak in 2020, there are indications that market efficiency is improving over time, even in the inefficient markets.
Overall, the Kalman-Filter method looks at how asset prices mean-revert from their fundamental values, offering a
fresh perspective on how to measure market efficiency overtime. And our findings suggest that while market efficiency
varies across countries, emerging markets tend to be less efficient than developed ones.

References

[1] D. B. W. F., T. R., Does the stock market overreact?, The Journal of finance 40(3) (1985) 793–805.
[2] J. N, Evidence of predictable behavior of security returns, The Journal of finance 45(3) (1990) 881–898.
[3] D. P. J., S. D., Who blinks in volatile markets, individuals or institutions?., The Journal of Finance 57(5) (2002)
1923–1949.

8
Beier Liu and Haiyun Zhu, 2024

[4] T. Barber B. M Odean, Z. N, Systematic noise, Systematic noise. Journal of Financial Markets 12(4) (2009)
547–569.
[5] B. G., H. C. R., Research in emerging markets finance: looking to the future., Emerging markets review 3(4)
(2002) 429–448.
[6] G. J. M. K. P. J., N. F., Do market efficiency measures yield correct inferences? a comparison of developed and
emerging markets, The Review of Financial Studies 23(8) (2010) 3225–3277.
[7] C. P. L., Statistical tests and retention of terms in the additive main effects and multiplicative interaction model for
cultivar trials., Crop science 33(6) (1993) 1186–1193.
[8] H. G. S., K. J, Stock returns predictability and the adaptive market hypothesis in emerging markets: evidence from
india, SpringerPlus 3 (2014) 1–14.
[9] A. J., A. M., Variance ratios., Socio-Economical Applications of Statistical Methods. (2000).
[10] S. N., S. A. K., International stock market efficiency and integration: evidences from asian and us markets, Journal
of Advances in Management Research 12(2) (2015) 88–106.
[11] X. X. M. Y. L. X., S. D., An empirical analysis of the adaptive market hypothesis with calendar effects: Evidence
from china, Finance Research Letters 31 (2019).
[12] C. A., D. O., Variance-ratio tests of random walk: an overview., Journal of economic surveys 23(3) (2009)
503–527.
[13] S. J. A. D. W. D. Broock W. A., L. B., A test for independence based on the correlation dimension., Econometric
reviews 15(3) (1996) 197–235.
[14] S. R. J., Do stock prices move too much to be justified by subsequent changes in dividends? (1981).
[15] B. B. M., O. T., Trading is hazardous to your wealth: The common stock investment performance of individual
investors., The journal of Finance 55(2) (2000) 773–806.
[16] S. A., Inefficient markets: An introduction to behavioural finance, Oup Oxford (2000).
[17] K. A., Y. A, Comparing linear and nonlinear forecasts for stock returns, International Review of Economics and
Finance 10(4) (2001) 383–398.
[18] R. A. M. A. A., S. V. N., Recurrent neural network and a hybrid model for prediction of stock returns., Expert
Systems with Applications 42(6) (2015) 3234–3241.
[19] L. X, China’s evolving stock market efficiency reconsidered: The kalman filter analysis, Available at SSRN
961108 (2001).