MACHINE LEARNING FOR TRADING

GORDON RITTER∗

Courant Institute of Mathematical Sciences

New York University
251 Mercer St., New York, NY 10012

Abstract. In multi-period trading with realistic market impact, de-

termining the dynamic trading strategy that optimizes expected utility
of final wealth is a hard problem. Gordon Ritter shows that, with an
appropriate choice of the reward function, reinforcement learning tech-
niques (specifically, Q-learning) can successfully handle the risk-averse
case.

1. Introduction

In this note, we show how machine learning can be applied to the problem
of discovering and implementing dynamic trading strategies in the presence
of transaction costs. Modern portfolio theory (which extends to multi-period
portfolio selection, ie. dynamic trading) teaches us that a rational risk-averse
investor seeks to maximize expected utility of final wealth, E[u(wT )]. Here
wT is the wealth random variable, net of all trading costs, sampled at some
future time T , and u is the investor’s (concave, increasing) utility function.
We answer the question of whether it’s possible to train a machine-learning
algorithm to behave as a rational risk-averse investor.

1.1. Notation. Let R denote the real numbers. If x, y ∈ RN are vectors,

then let xy ∈ RN and x/y denote the pointwise product and pointwise
P
quotient (also vectors), while x · y = i xi yi ∈ R denotes the scalar product.
Let δ denote the first-difference operator for time series, so for any time series
{xt : t = 1, . . . , T }, we have

δxt := xt − xt−1 .

Date: December 2, 2017.

Key words and phrases. Finance; Investment analysis; Machine learning; Portfolio opti-
mization.
* Corresponding author. E-mail address: [email protected].
1
2 MACHINE LEARNING FOR TRADING

We will never use the letter δ in any other way. The bold letters E and V
denote the expectation and variance of a random variable.

1.2. Utility theory. The modern theory of risk-bearing owes most of its
seminal developments to Pratt (1964) and Arrow (1971). Under their frame-
work, the rational investor with a finite investment horizon chooses actions
to maximize the expected utility of terminal wealth:
T
X
(1) maximize: E[u(wT )] = E[u(w0 + δwt )]
t=1

where T ≥ 1 is the finite horizon, δwt := wt − wt−1 is the change in wealth,

and u denotes the utility function.
The investor cannot directly control future changes in wealth δwt ; rather,
the investor makes trading decisions or portfolio-selection decisions which af-
fect the probability distribution of future δwt . In effect, the investor chooses
which lottery to play.
The theory surrounding solutions of (1) is called multiperiod portfolio
choice. As already understood by Merton (1969), problems of the sort (1)
fall naturally under the framework of optimal control theory. Unfortunately,
for most realistic trading cost functions, the associated Hamilton-Jacobi-
Bellman equations are too difficult to solve. Recent work has uncovered
explicit solutions for quadratic costs (Gârleanu and Pedersen, 2013), and
efficient methods to deal with realistic (including non-differentiable) trading
costs were discussed by Kolm and Ritter (2015), and Boyd et al. (2017).
In the theory of financial decision-making a lottery is any random variable
with units of wealth. In the generalized meaning of the word ‘lottery” due
to Pratt (1964), any investment is a lottery. Playing the lottery results in
a risk, defined as any random increment to one’s total wealth. The lottery
could have a positive mean, in which case some investors would pay to play
it, whereas if it has a zero mean then any risk-averse investor would pay an
amount called the risk premium to remove the risk from their portfolio.
The utility function u : R → R is a mapping from wealth, in units of
dollars, to a real number with dimensionless units. The numerical value of
u is important only insofar as it induces a preference relation, or ordering,
on the space of lotteries. If the investor is risk-averse, then u is concave; see
Pratt (1964) and Arrow (1971) for more details.
A risk-neutral investor has a linear utility function, which implies they
are concerned only with maximizing expected wealth, and are indifferent
 MACHINE LEARNING FOR TRADING 3

to risk. Most investors are not indifferent to risk, and hence maximizing
expected wealth is only a valid modus operandi in specific scenarios (eg.
high-frequency trading) where the risk is controlled in some other way.
In the risk-neutral case, u is a linear function and (1) takes the much
simpler form
T
hX i
(2) maximize: E δwt
t=1

In this overly-simplistic approximation (which, we emphasize, is not literally

applicable to risk-averse investors), the problem reduces to a reinforcement
learning problem.
In retrospect it seems natural that reinforcement learning applies here.
Reinforcement learning is a set of algorithms for directly learning value func-
tions and hence finding approximate solutions to optimal control problems,
and multiperiod portfolio choice is a particular kind of optimal control.

1.3. Reinforcement learning. In reinforcement learning, agents learn how

to choose actions in order to optimize a multi-period cumulative “reward.”
We refer to the wonderful book by Sutton and Barto (1998) and the survey
article by Kaelbling, Littman, and Moore (1996) for background and a his-
tory of reinforcement learning. If the per-period reward were identified with
marginal wealth, then the problem would have the same mathematical form
as (2), but this is only the correct specification if risk is ignored.
The identification of (2) with the basic problem of reinforcement learn-
ing is the beginning of a good idea, but it needs a scientifically rigorous
development. Questions that need answering include:
(1) Reinforcement learning algorithms refer to an action space, a state
space, a Markov decision process (MDP), value function, etc. To
which variables do these abstract concepts correspond when analyz-
ing a trading strategy?
(2) Realistic investors do not simply trade to maximize expected wealth;
they are not indifferent to risk. Can reinforcement learning algo-
rithms be modified to account for risk?
(3) What are the necessary mathematical assumptions on the random
process driving financial markets? In particular, financial asset re-
turn distributions are widely known to have heavier tails than the
normal distribution, so our theory wouldn’t be much use if it required
normality as an assumption.
4 MACHINE LEARNING FOR TRADING

After making precise assumptions concerning the trading process, and

other assumptions concerning the utility function and the probability distri-
butions of the underlying asset returns, we will eventually show that mul-
tiperiod portfolio choice can be solved by reinforcement learning methods.
That is, we show that machines can learn to trade.

2. The trading process

2.1. Accounting for profit and loss. Suppose that trading in a market
with N assets occurs at discrete times t = 0, 1, 2, . . . , T . Let nt ∈ ZN denote
the holdings vector in shares at time t, so that

ht := nt pt ∈ RN

denotes the vector of holdings in dollars, where pt denotes the vector of

midpoint prices at time t.
Assume for each t, a quantity δnt shares are traded in the instant just
before t, and no further trading occurs until the instant before t + 1. Let

vt := navt + casht where navt := nt · pt

denote the “portfolio value,” which we define to be net asset value in risky
assets, plus cash. The profit and loss (PL) before commissions and financing
over the interval [t, t + 1) is given by the change in portfolio value δvt+1 .
For example, suppose we purchase δnt = 100 shares of stock just before t
at a per-share price of pt = 100 dollars. Then navt increases by 10,000 while
casht decreases by 10,000 leaving vt invariant. Suppose that just before t+1,
no further trades have occurred and pt+1 = 105; then δvt+1 = 500, although
this PL is said to be unrealized until we trade again and move the profit
into the cash term, at which point it is realized.
Now suppose pt = 100 but due to bid-offer spread, temporary impact, or
other related frictions our effective purchase price was p̃t = 101. Suppose
further that we continue to use the midpoint price pt to “mark to market,”
or compute net asset value. Then as a result of the trade, navt increases by
(δnt )pt = 10, 000 while casht decreases by 10,100, which means that vt is
decreased by 100 even though the reference price pt has not changed. This
difference is called slippage; it shows up as a cost term in the cash part of
vt .
Executing the trade list results in a change in cash balance given by

δ(cash)t = −δnt · p̃t

 MACHINE LEARNING FOR TRADING 5

where p̃t is our effective trade price including slippage. If the components
of δnt were all positive then this would represent payment of a positive
amount of cash, whereas if the components of δnt were negative we receive
cash proceeds.
Hence before financing and borrow cost, one has

δvt := vt − vt−1 = δ(nav)t + δ(cash)t

(3) = δnt · (pt − p̃t ) + ht−1 · rt

where the asset returns are rt := pt /pt−1 − 1. Let us define the total cost ct
inclusive of both slippage and borrow/financing cost, as follows:

(4) ct := slipt + fint , where

(5) slipt := δnt · (p̃t − pt )

where fint denotes the commissions and financing costs incurred over the
period; commissions are proportional to δnt and financing costs are convex
functions of the components of nt . The component slipt is called the slippage
cost. Our conventions are such that fint > 0 always, and slipt > 0 with high
probability due to market impact and bid-offer spreads.

2.2. Portfolio value versus wealth. Combining (4)–(5) with (3) we have
finally

(6) δvt = ht−1 · rt − ct

If we could liquidate the portfolio at the midpoint price vector pt , then vt

would represent the total wealth at time t associated to the trading strategy
under consideration. Due to slippage it is unreasonable to expect that a
portfolio can be liquidated at prices pt , which gives rise to costs of the form
(5).
Concretely, vt = navt + casht has a cash portion and a non-cash portion.
The cash portion is already in units of wealth, while the non-cash portion
navt = nt · pt could be converted to cash if a cost were paid; that cost is
known as liquidation slippage:

liqslipt := −nt · (p̃t − pt )

Hence it is the formula for slippage, but with δnt = −nt . Note that liquida-
tion is relevant at most once per episode, meaning the liquidation slippage
should be charged at most once, after the final time T .
6 MACHINE LEARNING FOR TRADING

To summarize, we may identify vt with the wealth process wt as long as

we are willing to add a single term of the form

(7) E[liqslipT ]

to the multi-period objective. If T is large and the strategy is profitable,

or if the portfolio is small compared to the typical daily trading volume,
then liqslipT  vT and (7) can be neglected without much influence on the
resulting policy. In what follows, for simplicity we identify vt with total
wealth wt .

2.3. Mean-variance equivalence. The goal of recent approaches (Gârleanu

and Pedersen, 2013; Kolm and Ritter, 2015; Boyd et al., 2017) to multiperiod
portfolio choice is to determine the optimal deterministic policy, i.e. the pol-
icy which maximizes the expected utility of final wealth, (1), assuming the
policy is followed. For nonlinear functions u, the optimal-policy problem
may be difficult to attack directly. Fortunately there are cases in which the
solution is also the solution to a much easier problem: the mean-variance
problem.
Let r denote the N × T random matrix of asset returns over all future
periods being considered. Each column rt of r denotes a cross-section of
asset returns for a particular time period in the future.
Asset returns r drive wealth increments δvt = δwt by (6), and hence asset
returns are the primary source of randomness in wT . Expressions such as
E[u(wT )] refer to the expectation value with respect to r.

Definition 1. The underlying asset return random variable r is said to

follow a mean-variance equivalent distribution if it has a density p(r), has
first and second moments, and for any increasing utility function u, there
exists a constant κ > 0 such that the policy which maximized E[u(wT )] is
also optimal for the simpler problem

(8) max {E[wT ] − (κ/2)V[wT ]}

π

Mean-variance equivalence presents a vast simplification, because while u

is generally a nonlinear (concave) function, one has
X
E[wT ] = w0 + E[δwt ]
t
 MACHINE LEARNING FOR TRADING 7

where w0 is the initial wealth, a constant. If δwt is statistically independent

from δws whenever t 6= s, then
X
(9) V[wT ] = V[δwt ]
t

Problem (8) then becomes:

X κ 
(10) max E[δwt ] − V[δwt ]
π
t
2
The multivariate normal is mean-variance equivalent, but the converse is
false; many heavy-tailed distributions (such as the multivariate Student-t)
are also mean-variance equivalent. It is not hard to show that any elliptical
distribution (ie. having characteristic function φ(t) = exp(i t0µ )ψ(t0Ω t)
where Ω is positive definite) is mean-variance equivalent.

Assumptions. Throughout the rest of the paper, we assume that the mul-
tivariate distribution p(r) is mean-variance equivalent. This entails, as a
consequence, that we may solve (1) by equivalently solving the (easier) prob-
lem (10).

3. Reinforcement Learning

Many intelligent actions are deemed “intelligent” precisely because they

are optimal interactions with an environment; an algorithm plays a computer
game intelligently if it can optimize the score. A robot navigates intelligently
if it finds a shortest path with no collisions.
In a historical thread largely independent of the utility-based theory of
portfolio selection, researchers in artificial intelligence developed models of
an agent which “learns” to interact with the agent’s environment, with the
eventual goal of optimizing cumulative “reward” or positive reinforcement
over time.
To solve these problems, machine learning researchers developed a deep
and beautiful set of theoretical results and corresponding algorithms and
engineering solutions, under the name reinforcement learning. These devel-
opments are too numerous to list here, but many are covered in the book
by Sutton and Barto (1998) and references therein.
Like the classic work of Merton (1969) on optimal lifetime consumption,
reinforcement learning has its roots in Bellman’s theory of optimal con-
trol. The main difference is that in reinforcement learning, typically the
connection between actions and rewards (ie. the function to be optimized
8 MACHINE LEARNING FOR TRADING

in optimal control!) is unknown, and must be inferred from the agent’s

interactions with the environment.

3.1. States. Formulating an intelligent behavior as a reinforcement learn-

ing problem begins with identification of the state space S. The relation
with what statisticians call “state-space models” is direct and intuitive: un-
derlying most reinforcement learning problems is a Markov Decision Process
(MDP).
In reinforcement learning, the environment is defined to be everything
outside the agent’s direct control. The agent can still have arbitrarily com-
plete knowledge about the environment, and the environment may be in-
directly affected by the agent’s actions. The term state, in reinforcement
learning problems, usually refers to the state of the environment.
In trading problems, the environment should be interpreted to mean all
processes generating observable data that the agent will use to make a trad-
ing decision. Let st denote the state of the environment at time t; the state
is a data structure containing all of the information the agent will need in
order to decide upon the action. This will include the agent’s current posi-
tion, which is clearly an observable that is an important determinant of the
next action.
At time t, the state st must also contain the prices pt , but beyond that,
much more information may be considered. In order to know how to interact
with the market microstructure and what the trading costs will be, the agent
may wish to observe the bid-offer spread and liquidity of the instrument. If
the decision to trade is driven by a signal – something which is supposed to
be predictive of future returns – then that signal is part of the environment;
the state would necessarily contain the signal.
If the process is to be Markov, then the action decision must not depend
on the whole history of states, hence the state itself must be a sufficiently
rich data structure to make the optimal decision.
This prescription means that the state of the environment is a fairly high-
dimensional object. Fortunately, in many cases it can be simplified. With
realistic trading costs, the multiperiod trading problem is already interesting
and useful even for a single asset. In cases where the cross-asset correlation
can be ignored, (eg. if it is being hedged externally) then a multi-asset
problem decouples into a system of single-asset problems. For a single-asset,
the state described above is not such a high-dimensional object.
 MACHINE LEARNING FOR TRADING 9

3.2. Actions. In a general reinforcement-learning setup, the agent observes

the state, and chooses an action according to some policy. This choice influ-
ences both the transition to the next state, as well as the reward the agent
receives. More precisely, there is assumed to be a distribution p(s0 , r | s, a)
for the joint probability of transitioning to state s0 ∈ S and receiving reward
r, conditional on the previous state being s and the agent taking action a.
In portfolio choice problems, the space A of actions available at time t can
be identified with either the space of trades δnt or equivalently the space of
target portfolios ht . In general the space of admissible actions depends on
the current state, but we always denote the action space by A, and when
only a subset of A is admissible, this will be made clear.
The action at leading to ht means that

δvt+1 = ht · rt+1 − ct+1

and hence the joint distribution of all subsequent value updates {δvt+1 , . . . , δvT },
as seen at time t, is determined by the action at .
In cases where the agent’s interaction with the market microstructure is
important then there will typically be more choices to make, and hence a
larger action space. For example, the agent could decide which execution
algorithm to use, whether to cross the spread or be passive, etc.

3.3. Value functions and policies. A policy, π, is a mapping from a

state s to a probability distribution over actions: π(a | s) is the probability
of taking action a when in state s. The policy will be called deterministic
if there is only one action with nonzero probability, in which case π is a
mapping from the current state to the next action.
Following the notation of Sutton and Barto (1998), the sequence of re-
wards received after time step t is denoted Rt+1 , Rt+2 , Rt+3 , . . .. The agent’s
goal is to maximize the expected cumulative reward, denoted by

(11) Gt = Rt+1 + γRt+2 + γ 2 Rt+3 + . . .

The agent then searches for policies which maximize E[Gt ]. The sum in
(11) can be either finite or infinite. The constant γ ∈ [0, 1] is known as
the discount rate, and is especially useful in considering the problem with
T = ∞, in which case γ is needed for convergence.
According to Sutton and Barto (1998), “the key idea of reinforcement
learning generally, is the use of value functions to organize and structure
10 MACHINE LEARNING FOR TRADING

the search for good policies.” The state-value function for policy π is

vπ (s) = Eπ [Gt | St = s]

where Eπ denotes the expectation under the assumption that policy π is

followed. Similarly, the action-value function expresses the value of starting
in state s, taking action a, and then following policy π thereafter:

qπ (s, a) := Eπ [Gt | St = s, At = a]

Policy π is defined to be at least as good as π 0 if vπ (s) ≥ vπ0 (s) for all

states s. An optimal policy is defined to be one which is at least as good as
any other policy. There need not be a unique optimal policy, but all optimal
policies share the same optimal state-value function v∗ (s) = maxπ vπ (s) and
optimal action-value function q∗ (s, a) = maxπ qπ (s, a).
The state-value function and action-value function satisfy Bellman opti-
mality equations
X
v∗ (s) = max p(s0 , r | s, a)[r + γ v∗ (s0 )]
a
s0 ,r
X
q∗ (s, a) = p(s0 , r | s, a)[r + γ max
0
q∗ (s0 , a0 )]
a
s0 ,r

where the sum over s0 , r denotes a sum over all states s0 and all rewards r.
In a continuous formulation, these sums would be replaced by integrals.
If we possess a function q(s, a) which is an estimate of q∗ (s, a), then the
greedy policy is defined as picking at time t the action a∗t which maximizes
q(st , a) over all possible a, where st is the state at time t. To ensure that,
in the limit as the number of steps increases, every action will be sampled
an infinite number of times we use an -greedy policy: with probability 1 − 
follow the greedy policy, while with probability  uniformly sample the action
space.
Given the function q∗ , the greedy policy is optimal. Hence an iterative
method which converges to q∗ constitutes a solution to the original problem
of finding the optimal policy.

3.4. Q-learning. An important breakthrough in reinforcement learning came

when Watkins (1989) suggested an iterative method which converges to the
optimal action-value function q∗ . The algorithm consists of the following
steps. One initializes a matrix Q with one row per state, and one column
per action. This matrix can be initially the zero matrix, or initialized with
 MACHINE LEARNING FOR TRADING 11

some prior information if available. Let S denote the current state. Repeat
the following steps until a pre-selected convergence criterion is obtained:
1. Choose action A ∈ A using a policy derived from Q (for example,
the -greedy policy described above)
2. Take action A, after which the new state of the environment is S 0
and we observe reward R
3. Update the value of Q(S, A):

(12) Q(S, A) ← Q(S, A) + α[R + γ max Q(S 0 , a) − Q(S, A)]

a

where α ∈ (0, 1) is called the step-size parameter, which influences the rate
of learning.

3.5. The reward function for utility maximization. For a reinforce-

ment learning approach to match (10), we need Rt to be an appropriate
function of wealth increments, such that the following relation is satisfied:
κ
E[Rt ] = E[δwt ] − V[δwt ]
2
One such function is,
κ
(13) Rt := δwt − (δwt − µ̂)2
2
where µ̂ is an estimate of a parameter representing the mean wealth incre-
ment over one period, µ := E[δwt ]. Estimation of the parameter µ̂ in this
context is slightly circular: E[δwt ] depends on the optimal policy, which
depends on the reward function, which depends on µ̂.
Fortunately, there is an easy (approximate) resolution to this circularity
problem. We propose that, at least initially, one use the trivial biased es-
timator µ̂ = 0. This will have the effect that we overestimate variance in
the reward function during an initial burn-in period. This over-estimation
will not be too large. Indeed, unless the Sharpe ratio of the strategy is very
high, one has the approximation

(14) E[(δwt − µ̂)2 ] ≈ E[(δwt )2 ]

Once the value function has sufficiently converged using the approximate
reward function
κ
Rt ≈ δwt − (δwt )2 ,
2
one may then begin to estimate µ̂ by the sample average. We emphasize
that accurate estimation of µ̂ is not crucially important to obtaining a good
policy, due to (14).
12 MACHINE LEARNING FOR TRADING

Identifying the reward function as (13), maximizing the expected value

of (11) with γ = 1 is the same as maximizing utility of final wealth:

E[Gt ] = E[Rt+1 + Rt+2 + Rt+3 + · · · + RT ]

T
X κ
= (E[δws ] − V[δws ])
s=t
2
As noted in our discussion of the trading process, δwt can be identified with
δvt where, as in (6),
δwt ≈ δvt = ht−1 · rt − ct
We now discuss how the reward is observed during the trading process.
A microsecond before time t, the agent observes the state pt and decides an
action, which is a trade list δnt in units of shares. The agent submits this
trade list to an execution system, and then can do nothing until just before
t + 1. The agent waits one period and observes the reward
κ
(15) Rt+1 ≈ δvt+1 − (δvt+1 )2 .
2
If the reinforcement algorithm is working, then the agent will learn how to
maximize the cumulative reward, ie. the sum of (15) which must necessarily
approximate the mean-variance form E[δv] − (κ/2)V[δv].

4. A Detailed Example

In this section, we provide a proof of concept in a controlled numerical

simulation which permits an approximate arbitrage, and we verify that the
Q-learning agent finds and exploits this arbitrage. As mentioned above,
multiperiod trading problems are mathematically interesting even when only
a single asset is being considered. We consider one such problem here.
For this example, assume that there exists a tradable security with a
strictly positive price process pt > 0. (This “security” could itself be a
portfolio of other securities, such as an ETF or a hedged relative-value
trade.) Further suppose that there is some “equilibrium price” pe such
that xt = log(pt /pe ) has dynamics

(16) dxt = −λxt + σ ξt

where ξt ∼ N (0, 1) and ξt , ξs are independent when t 6= s. This means that

pt tends to revert to its long-run equilibrium level pe with mean-reversion
rate λ, and is a standard discretization of the Ornstein-Uhlenbeck process.
For this exercise, the parameters of the dynamics (16) were taken to be
 MACHINE LEARNING FOR TRADING 13

λ = log(2)/H;, where H = 5 is the half-life, σ = 0.1, and the equilibrium

price is pe = 50.
All realistic trading systems have limits which bound their behavior. For
this example we use a reduced space of actions, in which the trade size δnt
in a single interval is limited to at most K round lots, where a “round lot” is
usually 100 shares (most institutional equity trades are in integer multiples
of round lots). Also we assume a maximum position size of M round lots.
Consequently, the space of possible trades, and also the action space, is

A = LotSize · {−K, −K + 1, . . . , K}

The action space has cardinality |A| = 2K +1. Letting H denote the possible
values for the holding nt , then similarly H = {−M, −M + 1, . . . , M } with
cardinality |H| = 2M + 1. For the examples below, we take K = 5 and
M = 10.
Another feature of real markets is the tick size, defined as a small price
increment (such as USD 0.01) such that all quoted prices (i.e. all bids and
offers) are integer multiples of the tick size. Tick sizes exist in order to
balance price priority and time priority. This is convenient for us since we
want to construct a discrete model anyway. We use TickSize = 0.1 for our
example.
We choose boundaries of the (finite) space of possible prices so that sample
paths of the process (16) exit the space with vanishingly small probability.
With the parameters as above, the probability that the price path ever exits
the region [0.1, 100] is small enough that no aspect of the problem depends
on these bounds. Concretely, the space of possible prices is:

P = TickSize · {1, 2, . . . , 1000} ⊂ R+

We do not allow the agent, initially, to know anything about the dynamics.
Hence, the agent does not know λ, σ, or even that some dynamics of the form
(16) are valid.
The agent also does not know the trading cost. We charge a spread cost
of one tick size for any trade. If the bid-offer spread were equal to two
ticks, then this fixed cost would correspond to the slippage incurred by an
aggressive fill which crosses the spread to execute. If the spread is only one
tick, then our choice is overly conservative. Hence

(17) SpreadCost(δn) = TickSize · |δn|

14 MACHINE LEARNING FOR TRADING

We also assume that there is permanent price impact which has a linear
functional form: each round lot traded is assumed to move the price one tick,
hence leading to a dollar cost |δnt | × TickSize/LotSize per share traded, for
a total dollar cost for all shares

(18) ImpactCost(δn) = (δn)2 × TickSize/LotSize.

The total cost is the sum of (17) and (18). Our claim is not that these are
the exact cost functions for the world we live in, although the functional
form does make some sense. For simplicity we have purposely ignored the
differences between temporary and permanent impact, modeling the total
effect of all market impact as (18). The question is: can an agent learn to
trade with the simplest realistic interpretation of bid/offer spread and mar-
ket impact? If so, then more intricate effects such as the intraday reversion
of temporary impact should be studied.
As mentioned above, the state of the environment st = (pt , nt−1 ) will
contain the security prices pt , and the agent’s position, in shares, coming
into the period: nt−1 . Therefore the state space is the Cartesian product
S = H × P.
The agent then chooses an action at = δnt ∈ A which changes the position
to nt = nt−1 +δnt and observes a profit/loss equal to δvt = nt (pt+1 −pt )−ct ,
and a reward Rt+1 = δvt+1 − 0.5κ (δvt+1 )2 as in eq. (15).
We train the Q-learner by repeatedly applying the update procedure in-
volving (12). The system has various parameters which control the learning
rate, discount rate, risk-aversion, etc. For completeness, the parameter val-
ues used in the following example were: κ = 10−4 , γ = 0.999, α = 0.001,
 = 0.1. We use ntrain = 107 training steps (each “training step” consists of
one action-value update as per (12)), and then evaluate the system on 5,000
new samples of the stochastic process.
 MACHINE LEARNING FOR TRADING 15

Simulated net P/L over 5000 out−of−sample periods

3e+06

2e+06

P/L
1e+06

0e+00

0 1000 2000 3000 4000 5000

time periods

Figure 4.1. Cumulative simulated out-of-sample P/L of

trained model.

The excellent performance out-of-sample should perhaps be expected; the

assumption of an Ornstein-Uhlenbeck process implies a near arbitrage in the
system. When the price is too far out of equilibrium, a trade betting that
it returns to equilibrium has a very small probability of a loss. With our
parameter settings, even after costs this is true. Hence the existence of an
arbitrage-like trading strategy in this idealized world is not surprising, and
perfect mean-reverting processes such as (16) need not exist in real markets.
Rather, the surprising point is that the Q-learner does not, at least ini-
tially, know that there is mean-reversion in asset prices, nor does it know
anything about the cost of trading. At no point does it compute estimates for
the parameters λ, σ. It learns to maximize expected utility in a model-free
context, ie. directly from rewards rather than indirectly (using a model).
Expected utility maximization achieves a much higher out-of-sample Sharpe
ratio than expected-profit maximization. The distinction lies entirely in the
value of κ used in (15). Choosing κ = 0 and hence ignoring risk is un-
appealing to the intuition, and indeed one may verify that this leads to a
much lower Sharpe ratio. This should not be surprising. Understanding of
this principle dates back at least to 1713 when Bernoulli pointed out that a
wealth-maximizing investor behaves nonsensically when faced with gambling
based on a martingale (see Bernoulli (1954) for a translation).

5. Simulation-Based Approaches

A major drawback of the procedure we have presented here is that it

requires a large number of training steps (a few million, on the problem we
16 MACHINE LEARNING FOR TRADING

presented). There are, of course, financial data sets with millions of time-
steps (e.g. high-frequency data sampled once per second for several years),
but in other cases, a different approach is needed. Even in high-frequency
examples, one may not wish to use several years’ worth of data to train the
model.
Fortunately, a simulation-based approach presents an attractive resolu-
tion to these issues. In other words, we propose a multi-step training proce-
dure: (1) posit a reasonably-parsimonious stochastic process model for asset
returns with relatively few parameters, (2) estimate the parameters of the
model from market data ensuring reasonably small confidence intervals for
the parameter estimates, (3) use the model to simulate a much larger data
set than the real world presents, and (4) train the reinforcement-learning
system on the simulated data.
For the model dxt = −λxt + σ ξt , this amounts to estimating λ, σ from
market data, which meets the criteria of a parsimonious model. Suppose
we also have a realistic simulator of how the market microstructure will
respond to various order-placement strategies. Crucially, in order to be ad-
missible, such a simulator should be able to accurately represent the market
impact caused by trading too aggressively. With these two components: a
random-process model of asset returns, and a good microstructure simula-
tor, one may then run the simulation until the Q-function has converged to
the optimal action-value function q∗ .
The learning procedure is then only partially model-free: it requires a
model for asset returns, but no explicit functional form to model trading
costs. The “trading cost model” in this case is provided by the market
microstructure simulator, which arguably presents a much more detailed
picture than trying to distill trading costs down into a single function.
Is this procedure prone to overfitting? The answer is yes, but only if the
asset-return model itself is overfit. This procedure simply finds the optimal
action-value function q∗ (and hence the optimal policy), in the context of the
model it was given. The problem of overfitting applies to the model-selection
procedure that was used to produce the asset return model, and not directly
to the reinforcement learning procedure. In the procedure above, steps 1-2
are prone to overitting, while steps 3-4 are simply a way to converge to the
optimal policy in the context of the chosen model.
 REFERENCES 17

6. Conclusions

According to neoclassical finance, no investor should hold a portfolio (or

follow a dynamic trading strategy) that does not maximize expected util-
ity of final wealth, E[u(wT )]. The concave utility function u expresses the
investor’s risk aversion, or preference for a less risky portfolio over a more
risky one with the same expected return. Only a risk-neutral investor would
maximize E[wT ]. The main contribution of this paper is that we show how
to handle the risk-averse case using reinforcement learning.
It is instructive to consider how this differs from approaches such as
Gârleanu and Pedersen (2013); to use their approach in practice requires
three models: a model of expected returns, a risk model which forecasts
the variance, and a (pre-trade) transaction cost model. The methods of
Gârleanu and Pedersen (2013) provide an explicit solution only when the
cost model is quadratic. By contrast, our method can, in principle, be ap-
plied without directly estimating any of these three models, or they can be
applied in cases where one has an asset return model, but machine learning
techniques are used to infer the optimal strategy without directly estimating
the cost function.

References

Arrow, Kenneth J (1971). “Essays in the theory of risk-bearing”.

Bernoulli, Daniel (1954). “Exposition of a new theory on the measurement
of risk”. Econometrica: Journal of the Econometric Society, pp. 23–36.
Boyd, Stephen et al. (2017). “Multi-Period Trading via Convex Optimiza-
tion”. arXiv preprint arXiv:1705.00109.
Gârleanu, Nicolae and Lasse Heje Pedersen (2013). “Dynamic trading with
predictable returns and transaction costs”. The Journal of Finance 68.6,
pp. 2309–2340.
Kaelbling, Leslie Pack, Michael L Littman, and Andrew W Moore (1996).
“Reinforcement learning: A survey”. Journal of artificial intelligence re-
search 4, pp. 237–285.
Kolm, Petter N and Gordon Ritter (2015). “Multiperiod Portfolio Selection
and Bayesian Dynamic Models”. Risk.
Merton, Robert C (1969). “Lifetime portfolio selection under uncertainty:
The continuous-time case”. The review of Economics and Statistics, pp. 247–
257.
18 REFERENCES

Pratt, John W (1964). “Risk aversion in the small and in the large”. Econo-
metrica: Journal of the Econometric Society, pp. 122–136.
Sutton, Richard S and Andrew G Barto (1998). Reinforcement learning: An
introduction. Vol. 1. 1. MIT press Cambridge.
Watkins, Christopher JCH (1989). “Q-learning”. PhD Thesis.