Asymptotic Properties of the

Maximum Likelihood Estimator in

Dichotomous Logistic Regression Models

Diploma Thesis

submitted to the

Department of Mathematics
Faculty of Sciences
University of Fribourg Switzerland

in partial fulfilment of the requirements

for the attainment of the

Diploma in Mathematics

by

Michael Beer

???

Academic supervisor: Prof Dr André Antille

Submission date: December 2001

Address of the author: Henzenmoos 41, 3182 Überstorf, Switzerland

E-mail: [email protected]
 Abstract

This diploma thesis investigates the consistency and asymptotic normality of

the maximum likelihood estimator in dichotomous logistic regression mod-
els. The consistency is proved under hypotheses which allow the number of
exogenous variables p to grow along with the sample size n. Conditions on
the interdependence of p and n necessary for the consistency of the estima-
tor are established. The asymptotic normality is proved in the case where
p is constant. Some aspects on the motivation and the origins of the model
are presented at the beginning. Finally, a short example based on a medical
study outlines the application of the logistic regression model.
Contents
1 Introduction 2

2 Logistic Regression Model 4

2.1 Model Specification . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Maximum Likelihood Estimation of the Parameter β . . . . . 5
2.3 Motivation and Interpretation . . . . . . . . . . . . . . . . . . 8
2.4 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Consistency 15
3.1 A Different Approach . . . . . . . . . . . . . . . . . . . . . . 15
3.2 Condition on the Relationship between p and n . . . . . . . . 22
3.3 Reformulation, Assessment, and Comparison . . . . . . . . . 23

4 Asymptotic Normality 27

5 Case Study 32

6 Conclusion 35

Acknowledgements 36

References 37

1
1 Introduction: A Regression Model for Dichoto-
mous Outcome Variables

Among the different regression models, logistic regression plays a particular

role. The base concept, however, is universal. In a statistical investigation,
several variables are defined and their values are determined for a set of
objects. Some of these variables are a priori considered to be dependent
on others within the same framework. Others are thought to be exclusively
dependent on exterior and not quantified (or not quantifiable) aspects, but
not on further variables inside the same model. The main aim of a regression
analysis is to reveal and specify the impact of the so-called independent or
exogenous explanatory variables on the dependent or endogenous response.
The prevalent linear regression model is – under certain hypotheses – in
many circumstances a valuable tool for quantifying the effects of several
exogenous on one dependent continuous variable. For situations where the
dependent variable is qualitative, however, other methods had to be devel-
oped. One of these is the logistic regression model which specifically covers
the case of a binary (dichotomous) response.
In a first section of this thesis, the logistic regression model as well as the
maximum likelihood procedure for the estimation of its parameters are in-
troduced in detail. A notation is fixed and will be used throughout the
remaining parts of this document. Furthermore, several ideas on the moti-
vation of the model and the interpretation of its parameters are outlined.
At the end, a short paragraph illuminates a few historical aspects in relation
to the development of this model.
The second part examines the consistency of the maximum likelihood esti-
mator for the model parameters. It will be shown that, if certain hypotheses
are satisfied, this estimator converges in probability to the true parameter
values as the sample size increases. As a peculiarity of the procedure pre-
sented in this thesis, the number of explanatory variables is allowed to grow
simultaneously with the number of observations. Conditions on the admis-
sible interdependence of both of these characteristics are given in order to
ensure the consistency of the estimator. Finally, these conditions are com-
pared to the assumptions of an existing consistency theorem.
As another important property, the asymptotic normality of the estimator
being discussed is proved in section 4. Picking up the thread of the previous
part, this aspect is established in a similar context. In contrast to the proof
of consistency, however, this section will be based on the “classical” approach
where the number of independent variables is held constant.
To conclude, a short example attempts to illustrate the application of the lo-
gistic regression model. Proceeding from a survey of health enhancing phys-

2
ical activity carried out by the Swiss Federal Offices of Sports and Statistics,
an estimation will be made what effects the linguistic background of a Swiss
inhabitant has on his or her daily or weekly physical activity. The results
are presented along with the Mathematica 1 code used for the calculations.

1
Mathematica is a registered trademark of Wolfram Research, Inc.

3
2 Logistic Regression Model

2.1 Model Specification

We consider a binary random variable y having a Bernoulli distribution,

L

y ∼ B 1, π(x) ,

i.e. the variable y takes either the value 1 or the value 0 with probabilities
π(x) or 1 − π(x) respectively. x ∈ Rp is a vector of p exogenous variables
and π : Rp −→ [0, 1] a real-valued function. In fact, π(x) represents the
conditional probability P (y = 1 | x) of y = 1, given x.
Let r := y − π(x), which allows us to rewrite our model as

y = π(x) + r ,

where r has an expectation of

E(r) = E y − π(x) = E(y) − π(x) = π(x) − π(x) = 0 (2.1a)

and a variance of


Var(r) = Var(y) = π(x) 1 − π(x) . (2.1b)

For the forthcoming analysis we are going to define the so-called logistic
transformation σLR : R −→ [0, 1] by
exp z 1
σLR (z) := =
1 + exp z 1 + exp −z
which allows us to specify the probability function π as

π(x) := σLR (xT β)

with a vector β ∈ Rp of unknown parameters. This specification yields the

logistic regression model with parameter β.
If we denote the inverse function of σLR , which is called logit transformation,
by
π
logit π = ln ,
1−π
we can write our regression model as

logit π(x) = xT β . (2.2)

4
 σLR (z)
1

0.8

0.6

PSfrag replacements 0.4

0.2

z
0
σLR (z) -4 -2 2 4

0
σLR (z)
0.25

0.2

0.15

PSfrag replacements 0.1

0.05
σLR (z)
z
-4 -2 2 4

Figure 1: The logistic transformation σLR and its first derivative.

Furthermore, it will be important to note that

 
0 ∂ exp z (exp z)(1 + exp z) − (exp z)2
σLR (z) = =
∂z 1 + exp z (1 + exp z)2 (2.3)
exp z
= > 0 ∀z ∈ R.
(1 + exp z)2

The shape of σLR and its first derivative σLR 0 are displayed in figure 1.
Possible motivations for this specific model shall be discussed in section 2.3.

2.2 Maximum Likelihood Estimation of the Parameter β

In order to estimate the true value β 0 of the vector β = (β1 , . . . , βp )T ∈
Rp ,

n
we consider a set of n observations {(yi , xi )}i∈{1,...,n} ∈ {0, 1} × Rp of
mutually independent responses yi with respective explanatory variables xi
(i = 1, . . . , n). Assuming
L

yi ∼ B 1, σLR (xT 0
i β ) (i = 1, . . . , n) (2.4)

we consider the model

y = σLR (Xβ 0 ) + r ,

5
where

y := (y1 , y2 , . . . , yn )T ∈ {0, 1}n ,

 T 
x1
 .. 
X :=  .  ∈ Rn×p , and
xT
n
r := y − σLR (Xβ 0 ) .

The function σLR applied to a vector z = (z1 , . . . , zn )T ∈ Rn has to be

interpreted as σLR (z) := (σLR (z1 ), . . . , σLR (zn ))T . By (2.1a) and (2.1b), we
know that

E(ri ) = 0 (2.5a)

and


s2i := Var(ri ) = σLR (xT 0 T 0
i β ) 1 − σLR (xi β )
exp xT
i β
0 1 + exp xT 0 T 0
i β − exp xi β
= · (2.5b)
1 + exp xT
i β
0 1 + exp xT
i β
0

exp xTi β
0
0
= = σLR (xT 0
i β ).
(1 + exp xTi β 0 )2

In this context, we shall analyse the maximum likelihood estimator β̂ of the

true parameter vector β 0 . The likelihood function is defined as
n
Y
L(β) = P(y = yi )
i=1
n  
Y 1
yi
1−yi
= σLR (xT
i β) 1 − σLR (xT
i β)
yi
i=1
n   yi
Y σLR (xT
i β)


= T
1 − σLR (xT i β)
i=1
1 − σLR (xi β)
 exp(xT
 yi
n i ˛)  
Y T
1+exp(xi ˛) 1 + exp(xT T
i β) − exp(xi β)
=  
i=1
1+exp(xT
i ˛)−exp(xi ˛)
T
1 + exp(xTi β)
T
1+exp(xi ˛)
n
yi n
Y exp(xTi β)
Y exp(yi xTi β)
= =
i=1
1 + exp(xi β) i=1 1 + exp(xT
T
i β)

6
and yields the log-likelihood function
n
X n

X

ln L(β) = ln exp(yi xT
i β) − ln 1 + exp(xT
i β)
i=1 i=1
n (2.6)
X

T
= y Xβ − ln 1 + exp(xT
i β) .
i=1

To determine the maximum of this function, we consider the gradient

n
X exp(xTi β)
∇ ln L(β) = X T y − T β)
xi = X T y − X T σLR (Xβ)
1 + exp(x
i=1 | {z i }
=σLR (xT
i ˛)

= X T (y − σLR (Xβ)) .

A necessary condition for β̂ being the maximum likelihood estimator of β 0

is therefore


X T y − σLR X β̂ = 0 . (2.7)

From the second partial derivatives

n n
!
∂2 ∂ X X
ln L(β) = xir yi − xir σLR (xT
i β)
∂βs ∂βr ∂βs
i=1 i=1
n n
X ∂ X
=− xir σLR (xT
i β) = −
0
xir xis σLR (xT
i β)
∂βs
i=1 i=1

we can derive the Hessian matrix Hln L (β) ∈ Rp×p of the log-likelihood
function ln L(β) given by

Hln L (β) = −X T D(β)X ,

where D(β) = (dij ) ∈ Rn×n is a diagonal matrix defined by

(
0 (xT β) if i = j,
σLR i
dij = (2.8)
0 if i 6= j.

Affirmation. Hln L (β) is negative semi-definite for any β ∈ Rp . 

Verification. We have
n
X
uT Hln L (β) u = −uT X T D(β)Xu = − (xT 2 0 T
i u) σLR (xi β) . (2.9)
i=1

As the first derivative of σLR is always positive (see (2.3)), we can see from
equation (2.9) that uT Hln L (β) u ≤ 0 for all u ∈ Rp and all β ∈ Rp . ¨

7
We therefore know that, in any case, the log-likelihood function ln L(β)
is concave. Consequently, any root of ∇ ln L(β) is a global maximum of
ln L(β) and a maximum likelihood estimator of β 0 . The condition (2.7)
therefore is not only necessary but also sufficient for β̂ being a maximum
likelihood estimator of β 0 . However, neither its existence nor its uniqueness
are a priori guaranteed.
Remark. If we assumed p ≤ n and rg X = p, we would obtain that Xu = 0
is equivalent to u = 0, such that Hln L (β) were negative definite for all
β ∈ Rp and thus ln L(β) strictly concave. Any β̂ satisfying condition (2.7)
therefore would be the unique maximum likelihood estimator of β 0 . 

Even though the existence of β̂ cannot be guaranteed for finite sample

spaces, we shall prove in section 3 that the probability of non-existence
approaches zero as n tends to infinity.
Iterative methods for obtaining the maximum likelihood estimator are pre-
sented for example by Amemiya (1985, pp. 274-275).

2.3 Motivation and Interpretation

The logistic regression model is a common tool for handling dichotomous

response variables. Several reasons account for this circumstance. We now
want to focus on two of them: the interpretation of the regression parameters
and the relationship with the logistic distribution. Other motivations, such
as the “formal connection of the logistic model with loglinear model theory”
and “theoretical statistical considerations including the availability of ‘exact’
analyses of individual parameters” mentioned by Santner and Duffy (1989,
p. 206), shall not be investigated within the scope of this thesis.

2.3.1 Odds, Log Odds and the Odds Ratio

In order to discuss binary data and to interpret the regression coefficients,

it is essential to mention the terms of odds and odds ratios as counterparts
to probabilities and probability ratios.
Suppose that an event A has a probability π. We then define the odds of
A as the probability that A occurs divided by the probability it does not
occur, i.e.
P (A) π
Odds(A) := c
= , (2.10)
P (A ) 1−π
where Ac denotes the complementary event of A in the respective sample
space.

8
Such a way of thinking is quite common in practice. If in a sporting compe-
tition, for example, the probability of one team winning is about 80 per-
cent, many people say that the odds on this team winning are four to
one. However, as Christensen (1997, p. 2) mentions, odds are not infre-
quently confused with probabilities. In a document entitled “What Are the
Odds of Dying?” the US National Safety Council (2001) states for instance:
“The odds of dying from an injury in 1998 were 1 in 1,796.” This number
was approximated by dividing the 1998 US population (270,248,000) by the
respective number of deaths (150,445). From a statistical point of view,
the number 1,796 is an estimation of the reciprocal probability, whereas
an estimation of the real odds on dying from an injury would rather be
150,445/(270,248,000 − 150,445), i.e. 1 to 1,795. The expression “1 in” in-
stead of “1 to” and the large numbers involved in this study justify such an
approximation procedure to a certain extent.
This latter example points out the need of careful distinction between odds
and probability. Note however, that there is a one-to-one relationship be-
tween both of them.2 Given the probability π, the respective odds o are
obtained by formula (2.10), while π is calculated from any given o by
π = o/(1 + o). As Christensen (1997, p. 3) emphasises, examining odds
therefore amounts to a rescaling of the measure of uncertainty: Probabili-
ties between 0 and 1/2 correspond to odds between 0 and 1 whereas odds
between 1 and ∞ correspond to probabilities between 1/2 and 1.
The loss of symmetry inherent in the transformation of probabilities to odds
can be offset by a subsequent application of the natural logarithm function.
Looking at the so-called log odds, we observe that they are “symmetric about
zero just as probabilities are symmetric about one half” (Christensen 1997,
p. 3). This is why mathematical analyses often deal with log odds instead
of “simple” odds. It is worth noting that the log odds are usually allowed to
take the values −∞ and +∞ in order to establish a one-to-one relationship
with the respective probabilities 0 and 1, and, moreover, that the trans-
formation of probabilities to log odds is exactly the logit transformation
introduced in section 2.1.
In order to compare the odds of two different events, it may be useful to
examine not only odds but also odds ratios. If the odds of an event A are
Odds(A) and the odds of an event  B are Odds(B), then the odds ratio
of B to A is defined as Odds(A) Odds(B).3 In the above study on the
“Odds of Dying”, for instance, the odds of dying in a railway accident were
1 to 524,752 whereas the odds of an accidential death in a motor-vehicle
2
This holds under the condition that the odds are allowed to take the value +∞ when
the probability π equals 1.
3
Note that this syntax is not used identically throughout the literature. While many
authors just speak of the ratio of the odds of A to the odds of B, Garson (2001, “Log-linear
Models, Logit, and Probit”) explicitly refers to the expression “odds ratio of B to A”.

9
were 1 to 6,211. The odds ratio of dying in a railway accident to dying in
a motor-vehicle in the USA thus was about 84, i.e. the odds of dying in
a motor-vehicle were about 84 times as important as those of dying in a
railway crash.

2.3.2 Interpretation of the Parameter β

For an interpretation of the parameter β, let x ∈ Rp and x̌ := x + ei where

ei denotes the ith canonical base vector of Rp for an arbitrary i ∈ {1, . . . , p}.
A comparison of the two model equations (2.2) for x and x̌ gives

logit π(x) = xT β
logit π(x̌) = x̌T β = (x + ei )T β = xT β + eT
i β

from where we get the difference

logit π(x̌) − logit π(x) = eT

i β = βi .

On the other hand,

 
    π(x̌)
π(x̌) π(x) 1−π(x̌)
logit π(x̌) − logit π(x) = ln − ln = ln  π(x) .
1 − π(x̌) 1 − π(x)
1−π(x)

We thus see that exp βi is the odds ratio of {y = 1} given x to {y = 1}

given x̌. In other words, the odds on the event that y equals 1 increase (or
decrease) by the factor exp βi if xi grows by one unit. If in a certain study,
for example, β2 has an estimated value of about 0.693, a unit increase in
x2 is likely to double the odds on getting a positive response (y = 1), as
exp 0.693 ≈ 2.
According to Hosmer and Lemeshow (1989, p. 41), “this fact concerning the
interpretability of the coefficients is the fundamental reason why logistic re-
gression has proven such a powerful analytic tool for epidemiologic research.”
At least, this argumentation holds whenever the explanatory variables x are
quantitative.
Collett (1991, pp. 242-246) gives an outline of a suitable procedure for the
case of qualitative exogenous variables. Assume, for instance, that the prob-
ability π is to be related to a single exposure factor g that has m levels, ex-
pressed through integer values between 0 and m − 1. Instead of introducing
g into a model like
logit π = β0 + β1 g ,

10
it is preferable to define a set of dummy variables x1 , . . . , xm−1 as
(
1 if g = i,
xi := (i = 1, . . . , m − 1)
0 if g =6 i,
and to analyse the model
logit π = β0 + β1 x1 + · · · + βm−1 xm−1
for instance. This procedure allows to distinguish between the effects on y
of every single factor level of g. An application of this idea will be shown in
section 5.
The possibility of interpreting the coefficients of β as logarithms of odds ra-
tios provides the foundation of a second important motivation of the logistic
model. Both Santner and Duffy (1989, pp. 206-207) and Christensen (1997,
pp. 118-120, p. 387) emphasise on the difference between prospective and
retrospective studies. Consider for instance an experiment in which 250 peo-
ple of an arbitrary population are sampled. A binary response “diseased”
(D) or “non-diseased” (D c ) is observed for each person. Moreover, there is a
single explanatory variable “exposed” (E) or “non-exposed” (E c ) involved.
This kind of study is called prospective. Let ψP denote the (prospective)
ratio of odds of disease for the exposed group to odds of disease for the
non-exposed group as

P (D|E) P (D|E c )
ψP = .
1 − P (D|E) 1 − P (D|E c )
According to the nature of the study, diseased individuals may be very rare
in a random sample of 250 people. So most of the collected data is about
non-diseased persons. It is therefore sometimes useful to fix the sample size
in the rare event category by design. In our example, one could possibly
study separated samples of 100 diseased and 150 non-diseased individuals
while determining for every person whether he or she had been exposed or
not. This procedure is called retrospective and leads directly to information
about the probability of exposure among the diseased and among the healthy
groups. We thus get the (retrospective) odds ratio

P (E|D) P (E|D c )
ψR = .
1 − P (E|D) 1 − P (E|D c )
However, we obtain by Bayes’s rule that
P (D|E) P (E|D)P (D) P (E|D)
P (D c |E) P (E|D c )P (D c ) P (E|D c )
ψP = P (D|E c )
= P (E c |D)P (D)
= 1−P (E|D)
= ψR
P (D c |E c ) P (E c |D c )P (D c ) 1−P (E|D c )

so that we are able to make inferences about ψP even from a retrospective

study. The generalisation of this result motivates the inspection of odds
ratios.

11
2.3.3 Relationship with the Logistic Distribution

Another important issue in connection with the logistic model is outlined

by Amemiya (1985, pp. 269-270) and Cramer (1991, pp. 11-13). Both of
them cite two examples, one a biometric threshold model and the other an
econometric utility model.
In the biometric model, they suppose a dosage xi of an insecticide is given to
the ith insect of a population. Furthermore, it is assumed that every insect
has its own tolerance or threshold level yi∗ against this insecticide. If the
dosage xi is higher than yi∗ , the insect dies, if it is lower, the insect survives.
The binary variable yi expresses whether the ith insect dies (yi = 1) or not
(yi = 0). We thus get

P (yi = 1) = P (yi∗ < xi ) = F (xi ) ,

where F is the cumulative distribution function of the random variable yi∗ .

The econometric model, on the other hand, attributes separate random util-
ities u1 and u0 to the two possible states y = 1 and y = 0 of a certain variable
y. For example, y = 1 could express that a person drives a car whereas y = 0
would mean that this person travels by transit to work. Both utilities u1
and u0 are assumed to depend on certain characteristics, represented by a
vector xi , as
ui = x T
i β + εi (i = 0, 1) ,
where εi is a random error term. If we suppose that our individual maximises
its utility, the probability of his decision in favour of the state y = 1 is
therefore given by

P (y = 1) = P (u0 < u1 ) = P (xT T

0 β + ε0 < x1 β + ε1 )


= P (ε0 − ε1 < xT T T
1 β − x0 β) = F (x1 − x0 ) β ,

where F is the cumulative distribution function of the random variable  :=

ε0 − ε 1 .
In both of these examples, an assumption on the shape of the cumulative
distribution function F or rather on the distribution of the underlying ran-
dom variable has to be made in order to estimate the probability of the
event {y = 1}. With respect to the central limit theorem, the assumption
F = Φ(0, 1), where the so-called probit function Φ(0, 1) is the cumulative
distribution function of the standardised normal distribution, could most
probably be justified, at least after an appropriate standardisation of the
relevant data. This specific choice would lead to what is called the probit
model.

12
However, the logistic transformation σLR itself provides a very similar and
analytically sometimes more convenient alternative. Being viewed as a dis-
0
tribution function, σLR gives rise to the logistic distribution. Its density σLR
2
has mean zero and variance π /3, so that it is appropriate to define the cu-
mulative distribution function of the standardised logistic distribution with
zero mean and unit variance as
exp λx
Λ(x) := σLR (λx) =
1 + exp λx
√
where λ = π 3. In the literature, for example by Cramer (1991), the
function Λ is sometimes called logit function and is therefore not to be
confused with the logit transformation introduced in section 2.1, which is
nearly its inverse.
Cramer (1991, section 2.3) shows in an example that “by judicious adjust-
ment of the linear transformations of the argument x, the logit and probit
probability functions can be made to coincide over a fairly wide range.”
Moreover, he states that “logit and probit functions which have been fitted
to the same data are therefore virtually indistinguishable, and it is impossi-
ble to choose between the two on empirical grounds.”
As a result, it is justifiable in most cases to assume a logistic distribution
instead of a normal distribution for yi∗ or  in examples as those mentioned
above. This, however, guides us again to the logistic regression approach.

2.4 History of the Logistic Regression Model

An overview of the development of the logistic regression model is given by

Cramer (1991, section 2.9). He identifies three sources having lead to this
model as it is known today: applied mathematics, experimental statistics,
and economic theory.
While studying the development of the human population, Thomas Robert
Malthus (1766-1834) described an increase “in geometric progression”. It
can be argued that at the basis of this statement, there was the formula for
exponential growth, N (t) = A exp αt, evolving directly from the differential
equation Ṅ (t) = αN (t) where N (t) denotes the size of a population at a
time t. This model, however, did not take into consideration the possibility
of a saturation in the growth process. For this reason, Pierre François Ver-
hulst (1804-1849) added an upper limit or saturation value W to the equa-
tion mentioned above: Ṅ (t) = βN (t)(W −N (t)). Defining Z(t) := N (t)/W ,
we obtain an equation of the form Ż(t) = γZ(t)(1 − Z(t)) whose solution is
the logistic function
exp(α + γt)
Z(t) = .
1 + exp(α + γt)

13
As Cramer (1991) states, this model of human population growth was in-
troduced independently of Verhulst’s study by Raymond Pearl and Lowell
Reed in an article entitled “On the rate of growth of the population of the
United States since 1790 and its mathematical representation” published in
1920. Cramer continues: “The basic idea that growth is proportional both
to the level already attained and to the remaining room to the saturation
ceiling is simple and effective, and the logistic model is used to this day to
model population growth or, in market research, to describe the diffusion
or market penetration of a new product or of new technologies. For new
commodities that satisfy new needs like television, compact discs or video
cameras, the growth of ownership is naturally proportional both to the pen-
etration rate already achieved and to the size of the remaining potential
market, and similar arguments apply to the diffusion of new products and
techniques in industry.”
The application of probability models to biological experiments in the thir-
ties of the twentieth century represents another foundation of the logistic
regression model. However, it was in the first place the probit model intro-
duced in section 2.3.3 which found its reflection in the literature. According
to Cramer, economists at that time did not seem to take the logit model
seriously. Only after Henri Theil 1969 generalised the bivariate or dichoto-
mous to the multinomial logit model with more than two states of the de-
pendent variable, the logistic regression gained its wide acceptance. In the
seventies, Daniel McFadden, winner of the 2000 Nobel Prize in economics,
and his collaborators finally provided a theoretical framework to the logit
model linking it directly to the mathematical theory of economic choice (see
McFadden 1974).

14
3 Consistency of the Maximum Likelihood Esti-
mator

3.1 A Different Approach

In this section, we are going to investigate the consistency of the maxi-

mum likelihood estimator β̂. More precisely, we are going to show that β̂
converges under certain hypotheses to the real value β 0 if the number n
of observations (yi , xi ) considered in the model tends to infinity. This, of
course, is a purely theoretical viewpoint since it will never be possible to in-
clude an infinity of samples in an empirical statistical analysis. Nonetheless,
the consistency of an estimator is an important aspect of its nature.
Different proofs of consistency can be found in the literature (see e.g. Gou-
rieroux and Monfort (1981) or Amemiya (1985, pp. 270-274)). All of them
include or base upon the fact that the probability of the existence of the
estimator β̂ approaches 1 as n tends to infinity. Furthermore, they proceed
on the assumption that the number p of exogenous variables is fixed once
and for all. In other words, p is compelled to remain constant while the
sample size n grows.
Results presented by Mazza and Antille (1998) shall enable us to release this
last aspect. We are going to assume that p is variable but dependent on n
and to examine what relationship between p and n is necessary in order not
to destroy the consistency of our estimator β̂. Intuition suggests that the
number of observations should be larger than the number of real parameters
to be estimated, i.e. n > p. The requirements on p, however, are more subtle
as we will see on the following pages.
Following Mazza and Antille (1998), we now define the error function

Ey,σ (β) := 1T T
n Hσ (Xβ) − y Xβ , (3.1)

where 1n := (1, . . . , 1)T ∈ Rn and Hσ is a primitive of σ : R −→ R. Consid-

ering the gradient


∇Ey,σ (β) = X T σ(Xβ) − X T y = X T σ(Xβ) − y

we become aware that

∇ ln L(β) = −∇Ey,σLR (β) .

Any estimator β̂ which maximises the log-likelihood function ln L minimises

thus simultaneously the error function Ey,σLR . This, however, is not very
surprising. If we take into consideration that a possible primitive of σLR is

15
HσLR (z) = ln(1 + exp z), we see directly by (2.6) and (3.1) that Ey,σLR (β) =
− ln L(β).
The theorem 1 provided by Mazza and Antille (1998, p. 4) based on the
definition of Ey,σ is therefore directly applicable to our logistic regression
model. On the following pages, we shall recite this theorem and its proof in
a variant slightly adjusted to our problem.4

Assumptions. We consider the following assumptions:

Pp Pn 2
M1: j=1 i=1 xij ≤ Cnp for some positive constant C > 0.

M2: If λ∗ denotes the smallest eigenvalue of the symmetric matrix X T X ∈

Rp×p , there exists a positive constant c > 0 such that λ∗ > c n for all
n. 

Theorem 3.1. Assume M1 and M2, let β ∈ Rp , and consider the random
vector y = σLR (Xβ 0 ) + r, where r = (r1 , r2 , . . . , rn )T has mutually indepen-
dent entries ri such that E(ri ) = 0 and E(ri2 ) = s2i > 0 for all i ∈ {1, . . . , n}.5
Let B(β 0 , δ) ⊂ Rp be the open ball of radius δ centered at β 0 , and let

aδpn (X, β 0 ) := inf 0

min σLR (ζi ) > 0 .
“∈XB(˛ 0 ,δ) i=1,...,n

Moreover, assume that p = p(n) depends on n such that

r
p 1
−−−→ 0 (3.2)
n apn (X, β 0 ) n→∞
δ

for some δ > 0. Then, with probability converging to 1 as n tends to infinity,

the absolute minimum β̂ of Ey,σLR (i.e. the absolute maximum of ln L) is
the unique root of
∇Ey,σLR (β) = −∇ ln L(β)
and converges in probability to the true value β 0 . 

For the proof of theorem 3.1, we need the following lemma which is in fact a
variant of Ortega and Rheinboldt’s (1970, p. 163) lemma 6.3.4 adapted for
our specific purpose.
4
There are two main differences. First, the assumptions on the differentiability of σ and
the positivity of its derivative are omitted because both of them are automatically satisfied
by σLR . Furthermore, the assumption of homoscedasticity of the random variables ri is
released in order to allow mutually different, but positive variances.
5
Note that, by (2.5b), we have s2i = σLR
0
(xT 0
i ˛ ). Following Mazza and Antille’s original,
the shorter notation s2i shall be applied in this context.

16
Lemma 3.2. Let B = B(β 0 , δ) be an open ball in Rp with center β 0 and
radius δ > 0. Assume that G : B̄ ⊂ Rp −→ Rp is continuous and satisfies
(β − β 0 )T G(β) ≤ 0 for all β ∈ ∂B, the border of B. Then G has a root in
B̄. 

Proof (of Lemma 3.2). We consider the ball B0 = B(0, δ) and define
G0 : B̄0 −→ Rn by G0 (γ) := γ + G(γ + β 0 ). Given the continuity of G,
the function G0 is also continuous. Let γ ∈ ∂B0 , i.e. kγk2 = δ 2 . Then
γ + β 0 ∈ ∂B, and thus, for any λ > 1,


γ T (λγ − G0 (γ)) = γ T (λ − 1)γ − G(γ + β 0 )
= (λ − 1)γ T γ − γ T G(γ + β 0 ) > 0 . (3.3)
| {z } | {z }
=(λ−1)k‚k2 >0 ≤0 by assumption

We now want to show, that G0 has a fixed point γ̂ ∈ B̄0 , i.e. G0 (γ̂) = γ̂. This
result would finally mean that G(γ̂+β 0 ) = 0 and, therefore, β̂ := γ̂+β 0 ∈ B̄
would be a root of G.
Assume that
 G0 has no fixed point in B̄0 . Then the mapping Ĝ(γ) :=
δ G0 (γ) − γ G0 (γ) − γ is well-defined and continuous on B̄0 , and
kĜ(γ)k = δ for any γ ∈ B̄0 . According to the Brouwer Fixed-Point Theo-
rem 6 , Ĝ has a fixed point γ ∗ in B̄0 and kγ ∗ k = kĜ(γ ∗ )k = δ. As

G0 (γ ∗ ) − γ ∗
γ ∗ = Ĝ(γ ∗ ) = δ ·
G0 (γ ∗ ) − γ ∗ ,

we have
γ∗

G0 (γ ∗ ) = G0 (γ ∗ ) − γ ∗ + γ ∗ = 1 + 1 G0 (γ ∗ ) − γ ∗ γ ∗ = λ∗ γ ∗ .
δ
δ | {z }
=:λ∗ >1

However, we thus get γ ∗ T (λ∗ γ ∗ − G0 (γ ∗ )) = 0 which contradicts condi-

tion (3.3). ¨

Remark. For the proof of lemma 3.2, the condition (β − β 0 )T G(β) ≤ 0

is not essential. The same result holds when (β − β 0 )T G(β) ≥ 0 for all
β ∈ ∂B. In the present case, however, the previous version suits better our
needs. 

Proof (of Theorem 3.1). Considering the function G(β) := −∇Ey,σLR (β),
we are going to show that there exists a ball B̄(β 0 , δ) ⊂ Rp , which – with
6
The Brouwer Fixed-Point Theorem reads as follows (see Ortega and Rheinboldt 1970,
p. 161): Every continuous mapping G : C̄ −→ C̄, where C̄ is a compact, convex set in Rp ,
has a fixed point in C̄.

17
probability converging to 1 as n tends to infinity – contains a root β̂ of G
for an arbitrary small δ > 0.
Let G(β)j denote the jth component of the vector G(β). We have


G(β)j = X T y − σ(Xβ)
j
n
X

= xij yi − σLR (xT
i β)
i=1
n
X

= xij yi − σLR (xT 0 T
i β + xi γ)
i=1

where γ := β − β 0 , and by defining η = (η1 , . . . , ηn )T := Xβ 0 we get with

the model assumption (2.4) the expression
n
X

G(β)j = xij σLR (ηi ) + ri − σLR (ηi + xT
i γ)
i=1

such that, as a result of the mean value theorem, there exists some ξi =
η i + α i xT
i γ with αi ∈ ]0, 1[ satisfying

n
X
0


G(β)j = xij ri − σLR (ξi ) · xT
i γ .
i=1

Thanks to the previous lemma, we only need to prove that γ T G(β) ≤ 0 for
all γ = β − β 0 with kγk = δ. We thus consider the expression
p p n
!
X X X

T 0 T
γ G(β) = γj G(β)j = γj xij ri − σLR (ξi ) · xi γ
j=1 j=1 i=1
   
n
X p
X n
X Xp
0
= ri  γj xij  −  γj xij  σLR (ξi ) xT
i γ
i=1 j=1 i=1 j=1
n
X n
X
2
= r i xT
i γ− xT
i γ
0
σLR (ξi ) .
|i=1 {z } |i=1 {z }
=:A1 =:A2

Using the Cauchy-Schwarz inequality7 , we get an upper boundary for |A1 |

by
 n   n !T  n
X   X 
 T  X
|A1 | =  r i xi γ  = 
 r i xi γ ≤
 ri xi · kγk . (3.4)
    |{z}
i=1 i=1 i=1
=δ
7
See e.g. Ortega and Rheinboldt (1970, p. 39).

18
 P
If we examine the second moment of k ni=1 ri xi k, we obtain
!  !2 
Xn 2 p
X Xn

E r i xi = E ri xij 
i=1 j=1 i=1
  
p
X n
X n X
X n
  
= E


 ri2 x2ij + ri rk xij xkj 


j=1 i=1 i=1 k=1
k6=i
 
p
X n n
n X
X 2 2
X 
= 
 E(r i ) x ij + E(ri rk ) xij xkj 

j=1 i=1 i=1 k=1
k6=i

and, as the random variables ri are mutually independent and their ex-
pectations are zero, we get E(ri rk ) = E(ri ) E(rk ) = 0 and thus, as s2i =
0 (xT β 0 ) ≤ 1 for all i,
σLR i 4

n
! n p n p
X 2 X X 1 X X 2 M1 Cnp
2 2
E r i xi = E(r ) xij ≤ xij ≤ (3.5)
| {zi } 4 4
i=1 i=1 j=1 i=1 j=1
=s2i

for some positive constant C > 0. We see from (3.4) and (3.5) that E(A 21 ) ≤
δ 2 Cnp/4 . Using the Tchebychev inequality (see Feller 1968, p. 233), we
have
P(|A1 | ≥ t) ≤ t−2 E(A21 ) ≤ t−2 δ 2 Cnp/4 ∀t > 0
−2 2
⇐⇒ P(|A1 | < t) ≥ 1 − t δ Cnp/4 ∀t > 0
| {z }
=:ε
 √ 
δ Cnp
⇐⇒ P |A1 | < √ ≥1−ε ∀ε > 0.
2 ε
√ 
Defining C ∗ := C 2, it is obvious that
 r   √ 
∗ np δ C ∗ np
P A1 ≤ δ C ≥ P |A1 | < √ .
ε ε
If we set nε := n/ε for any given ε > 0, we obtain
√
P(A1 ≤ δ C ∗ nε p) ≥ 1 − ε ,
so for all ε > 0, there exists a number nε ∈ N such that for any n ≥ nε and
any given C ∗ > 0, we have
√
P(A1 ≤ δ C ∗ np) ≥ 1 − ε . (3.6)
Now, let us turn to the examination of A2 . Let Z := {ξ ∈ Rn | ξi = xT 0
i β +
α i xT
i γ, αi ∈ ]0, 1[}.

19
Affirmation. For any vector ξ ∈ Z, we have
0
σLR (ξi ) ≥ aδpn (X, β 0 ) = inf 0
min σLR (ζi ) ∀ i ∈ {1, . . . , n} .
“∈XB(˛ 0 ,δ) i=1,...,n


Verification. Assume that there is some ξ ∗ ∈ Z such that for an i∗ ∈

{1, . . . , n} we have
0
σLR (ξi∗∗ ) < inf 0
min σLR (ζi ) .
“∈XB(˛ 0 ,δ) i=1,...,n

As ξ ∗ ∈ Z, there are numbers αi ∈ ]0, 1[ (i = 1, . . . , n) such that ξi∗ =

xT 0 T ∗ 0 ∗
i β + αi xi γ. We define ζ := Xβ + αi∗ Xγ. Then, ζ := X(β + αi∗ γ)
0
∗ ∗ 0
and, as kαi γk < kγk = δ, we have ζ ∈ XB(β , δ). But
0
min σLR (ζi∗ ) ≤ σLR
0 0
(ζi∗∗ ) = σLR (xT 0 T 0 ∗
i∗ β + αi∗ xi∗ γ) = σLR (ξi∗ ) ,
i=1,...,n

which is a contradiction. 2

We therefore get
n
X
2
A2 = xT
i γ
0
σLR (ξi )
i=1
n (3.7)
X
2
≥ aδpn (X, β 0 ) xT
i γ = aδpn (X, β 0 ) kXγk2 .
i=1

Looking at kXγk2 = (Xγ)T Xγ = γ T X T Xγ, we observe that X T X is

symmetric and – as a result of M2 – positive definite. Therefore, there exists
a set of orthonormal eigenvectors {v1 , . . . , vp } with attributed
P eigenvalues
{λ1 , . . . , λp } forming a base of Rp . If we now write γ = pi=1 γi∗ vi , we obtain
!  
Xp Xp
kXγk2 = γi∗ viT X T X  γj∗ vj 
i=1 j=1
p X
X p p X
X p
= γi∗ γj∗ viT X T Xvj = γi∗ γj∗ λj viT vj
i=1 j=1
| {z }
i=1 j=1 (3.8)
=λj vj
X p
p X p
X p
X
≥ λ∗ γi∗ γj∗ viT vj = λ∗ γi∗ viT γj∗ vj
i=1 j=1 i=1 j=1
T 2 2
= λ∗ γ γ = λ∗ kγk = λ∗ δ ,

where λ∗ denotes the smallest eigenvalue of X T X. We thus see that A2 ≥

aδpn (X, β 0 ) λ∗ δ 2 and, because of M2, there is a positive constant c > 0 such

20
that A2 ≥ aδpn (X, β 0 ) c n δ 2 . When combining this result with (3.6), we get
that, for any ε > 0, there is a number nε such that
√
P(A1 − A2 ≤ δ C ∗ np − aδpn (X, β 0 ) c n δ 2 ) ≥ 1 − ε

for all n ≥ nε . As we are interested in the case where γ T G(β) = A1 −A2 ≤ 0,

we consider the inequality
√
δ C ∗ np − aδpn (X, β 0 ) c n δ 2 ≤ 0
√
⇐⇒ C ∗ np ≤ aδpn (X, β 0 ) c n δ
r
C∗ p 1
⇐⇒ ≤ δ. (3.9)
c n apn (X, β 0 )
δ

As a consequence of the assumption (3.2), there is for any δ > 0 a number

nδ such that (3.9) holds for all n ≥ nδ . We therefore know that, for any
positive δ and ε, we have


P γ T G(β) ≤ 0 ≥ 1 − ε

for all n ≥ max(nδ , nε ) and γ ∈ ∂B(0, δ). Furthermore, we remember from

(2.9) that
n
X
T
u Hln L (β) u = − (xT 2 0 T
i u) σLR (xi β) ,
i=1

from where we obtain in analogy to (3.7) that

n
X
T
u HEy,σLR (β) u = (xT 2 0 T
i u) σLR (xi β)
i=1
≥ aδpn (X, β 0 ) c n kuk2 ≥ inf aδpn (X, β 0 ) c n kuk2 > 0
n∈N

if u 6= 0. The last inequality is a result of the assumption (3.2): If

n aδpn (X, β 0 ) converged to 0 as n tends to infinity, we would have
r √
p 1 pn
= −−−→ ∞
n aδpn (X, β 0 ) n aδpn (X, β 0 ) n→∞

which is a contradiction. The Hessian matrix HEy,σLR (β) is thus strictly

positive definite and Ey,σLR is strictly convex in B̄(β 0 , δ). Therefore, under
the hypotheses M1 and M2, the absolute minimum β̂ of Ey,σLR (the ab-
solute maximum of ln L) is not only unique but converges in probability to
the true value β 0 as n tends to infinity. ¨

21
Remark. If we choose δ such that
r
C∗ p 1
δ = δn := ,
c n apn (X, β 0 )
δ

we notice that, with probability converging to 1 as n −→ ∞,

!
0 2 p 1
kβ − β̂k = Op
,
n aδpn (X, β 0 ) 2

n

2
as β̂ ∈ B̄(β 0 , δn ). In other words, p aδpn (X, β 0 ) kβ 0 − β̂k2 is bounded in
probability. 

3.2 Condition on the Relationship between p and n

As mentioned before, Mazza and Antille (1998) originally developed this

theorem for an arbitrary strictly increasing and differentiable function σ.
The choice of one precise element σLR from this set of functions gives us
the possibility to study more in detail what condition (3.2) really implies on
the relationship between p and n. In other words, we can attempt to give a
condition on p(n) such that (3.2) is a consequence.
For this purpose we assume that there is some positive constant D > 0 such
that the true value β 0 ∈ Rp satisfies the condition

2
sup max βj0 < D ,
p∈N j=1,...,p

i.e. the norm of β 0 is bounded for all p. We suppose furthermore that

kxi k2 ≤ D∗ p for some positive constant D ∗ > 0 and for all i = 1, . . . , n.
This requirement holds for instance when the elements of X are uniformly
bounded.
For every δ > 0 and all β ∈ B(β 0 , δ) we thus obtain by Cauchy-Schwarz
that
 T 

xi β  ≤ kxi k kβk < kxi k β 0 + δ
p p  √ p
≤ D∗ p Dp + δ = p D∗ D + δ D∗ p .
√ √
Hence, while p and n increase, there is a constant D̄ := D ∗ ( D + δ) > 0
 
such that xT 0
i β ≤ D̄p. As a result, |ζi | ≤ D̄p for all ζ ∈ XB(β , δ) and for

all i ∈ {1, . . . , n}. These calculations allow us to state the inequality

aδpn (X, β 0 ) = inf 0

min σLR 0
(ζi ) ≥ inf σLR (z) . (3.10)
“∈XB(˛ 0 ,δ) i=1,...,n |z|≤D̄p

22
 0 (z) = (exp z)/(1 + exp z)2 is an even function having at z = 0 its
As σLR
maximum (see figure 1), we get

0 0 exp D̄p
inf σLR (z) = σLR (D̄p) = .
|z|≤D̄p (1 + exp D̄p)2

Consequently, the inequality (3.10) can be rewritten as

1 1
≥ ∀n∀p
exp D̄p aδpn (X, β 0 )
(1+exp D̄p)2

which gives us the implication

r r
p (1 + exp D̄p)2 p 1
I := −−−→ 0 =⇒ −−−→ 0 .
n exp D̄p n→∞ n apn (X, β 0 ) n→∞
δ

In other words, our requirement (3.2) on p(n) holds when the left side of
this implication is true. Let us examine
r r
p (1 + exp D̄p)2 p 1 + exp D̄p
I= = (1 + exp D̄p)
n exp D̄p n exp D̄p
r r r 
p
p p
= exp(−D̄p) + 1 (1 + exp D̄p) ≤ c1 + exp D̄p .
n | {z } n n
≤exp(−D̄)+1=:c1

Affirmation. Let C be a constant such that C < 1/(2D̄), and suppose that
p(n) ≤ C ln n. Then limn→∞ I = 0. 

Verification. We have
r r r r
p C ln n
C D̄ C ln n C ln n
exp D̄p ≤ exp C D̄ ln n = n =
n n n n1−2C D̄

which converges to 0 as n tends to infinity if and only if 1 − 2C D̄ > 0.

However, this last inequality is equivalent
p to C < 1/(2D̄). Moreover, this
same argument shows that limn→∞ p/n = 0, and thus limn→∞ I = 0. ¨

3.3 Reformulation, Assessment, and Comparison

We shall now resume the results of the previous two sections in the following
theorem.

23
Assumptions. Consider the following assumptions:

B1: There exists a positive constant D ∗ > 0 such that kxi k2 ≤ D∗ p for all
i ∈ {1, . . . , n}.
B2: There exists a positive constant c > 0 such that λ∗ > c n for all n,
where λ∗ denotes the smallest eigenvalue of X T X.
B3: There exists a positive constant D > 0 such that

2
sup max βj0 < D .
p∈N j=1,...,p

 √ √

B4: For an arbitrary δ > 0 there exists a constant C < 1 2 D∗ ( D +δ)
such that p(n) ≤ C ln n. 

Theorem 3.3. Assume B1, B2, B3, and B4. Then the maximum likeli-
hood estimator β̂ exists almost surely as n tends to infinity, and β̂ converges
to the true value β 0 . 

Remark. Note that B1 implies M1 while B2 equals M2. 

As an assessment of this result, we shall finally attempt to compare it to the

theorem on the existence and strong consistency of the maximum likelihood
estimator β̂ formulated by Gourieroux and Monfort (1981). In contrast to
other authors, they were able to prove these properties under comparatively
weak assumptions, as they themselves affirm.
For the purpose of such a comparison, we return to the case where p is
constant. As a consequence of this restriction, the assumptions B3 and B4
are automatically satisfied, at least if n is sufficiently large. Moreover, B1
reduces to


B1: There exists a positive constant M > 0 such that kxi k2 ≤ M for all
i ∈ {1, . . . , n}.

Let us now turn to the article of Gourieroux and Monfort. They make the
following assumptions:

G1: The exogenous variables are uniformly bounded, i.e. there exists a
positive constant M0 such that |xij | ≤ M0 for all i ∈ {1, . . . , n} and
all j ∈ {1, . . . , p}.
G2: Let λ1n and λpn be respectively the smallest and the largest eigenvalue
of −Hln L (β 0 ) = X T D(β 0 )X, the diagonal matrix D(β 0 ) being de-
fined as in (2.8). There exists a constant M1 such that λpn /λ1n < M1
for all n.

24
Theorem 3.4 (Gourieroux and Monfort). If G1 and G2 are satisfied,
the maximum likelihood estimator β̂ of β exists almost surely as n goes to
infinity, and β̂ converges almost surely to the true value β 0 if and only if

lim λ1n = +∞ . 
n→∞

By the norm-equivalence theorem8 with respect to the Euclidean and the

maximum norm, we immediately get the equivalence of B1  and G1. As, fur-
thermore, B2 implies that λ∗ tends to infinity as n increases, it is interesting
to check whether this fact has an impact on the limit of λ1n .

Affirmation. Assume B1. Then

lim λ∗ = +∞ ⇐⇒ lim λ1n = +∞ . 

n→∞ n→∞

Verification. Let Σ = (sij ) ∈ Rn×n denote the diagonal matrix defined by

(q
0 (xT β 0 ) if i = j,
σLR i
sij = (3.11)
0 if i 6= j.
p
Note that Σ2 = D(β 0 ) and that sii = si = Var(ri ). On one hand, we
0 (z) ≤ 1/4 ∀ z and thus s2 ≤ 1/4. On the other hand,
have σLR i

­ √
B1
|xT 0 0
i β | ≤ kxi k kβ k ≤ M kβ 0 k
√
0 ( M kβ 0 k) =: c > 0 for all i. Consequently, the real-
such that s2i ≥ σLR s
valued function k · ks defined as
v
u n
√ uX
T
kwks := w Σ w = t 2 s2i wi2
i=1

is a norm on Rn , and for all w ∈ Rn we have the inequality

kwk2
cs kwk2 ≤ kwk2s ≤ . (3.12)
4
Let S p−1 denote the unit sphere in Rp , i.e. S p−1 := {v ∈ Rp | kvk = 1}. By
the Courant-Fischer-Weyl minimax principle9 we have

λ∗ = min kXvk2 and λ1n = min kΣXvk2 = min kXvk2s .

v∈S p−1 v∈S p−1 v∈S p−1
8
See e.g. Ortega and Rheinboldt (1970, p. 39).
9
See e.g. Bhatia (1997, p. 58).

25
Choose v∗ ∈ S p−1 such that λ∗ = kXv∗ k2 . By (3.12) we have λ∗ =
kXv∗ k2 ≥ 4 kXv∗ k2s ≥ 4 λ1n . Conversely, if we choose v∗∗ ∈ S p−1 such
that λ1n = kXv∗∗ k2s , we get λ1n = kXv∗∗ k2s ≥ cs kXv∗ k2 ≥ cs λ∗ . In
consequence, the inequality
1
cs λ∗ ≤ λ1n ≤ 4 λ∗ (3.13)

holds for all n, from where we get the affirmed equivalence. ¨

This result shows that B2 implies that λ1n also goes to infinity as n increases.
The opposite, however, is not always true: Assume for instance that λ1n =
cl ln n where cl > 0 is an arbitrary positive constant. By (3.13), we get thus
cl
4 cl ln n ≤ λ∗ ≤ ln n .
cs
So while limn→∞ λ∗ = ∞, there is no positive constant c such that λ∗ > c n
for all n.
The assumption limn→∞ λ1n = ∞ of the theorem 3.4 is therefore less re-
strictive than B2. Conversely, Gourieroux and Monfort (1981) need the
supplementary hypothesis G2 to prove the consistency of β̂. On the other
hand, G2 additionally ensures that limn→∞ λ1n = ∞ is not only sufficient
but also necessary for the consistency of β̂.

26
4 Asymptotic Normality of the Maximum Likeli-
hood Estimator

In the previous section, we have studied the consistency of the maximum

likelihood estimator in the logistic regression model. We were able to prove
that, under certain hypotheses, the estimator β̂ of β ∈ Rp converges in
probability to the true value β 0 .
A second important issue to be studied is the asymptotic normality of the
estimator β̂. More precisely, we are going to show that the normalised linear
combinations of the components of β̂−β 0 converge in distribution to random
variables having a normal law with zero expectation.
The base structure of the following theorem is taken from an article by
Antille and Rech (1999). Some extensions had to be made in respect of the
heteroscedasticity of the components of r. On the other hand, the specific
knowledge of σLR gave rise to several simplifications. In contrast to section 3,
we now assume that the number p of exogenous variables is constant and
thus independent of the sample size n.
Assumptions.
A1: The exogenous variables X are uniformly bounded, i.e. there exists a
positive constant M such that |xij | ≤ M for all i ∈ {1, . . . , n} and all
j ∈ {1, . . . , p}.

A2: If λ∗ denotes the smallest eigenvalue of the symmetric matrix X T X ∈

Rp×p , there exists a positive constant c > 0 such that λ∗ > c n for all
n. 

Theorem 4.1. Assume A1 and A2 and let D(β 0 ) and Σ be defined as in

(2.8) and (3.11) respectively. Then

eT X T D(β 0 )X(β̂ − β 0 ) L
−−−→ N (0, 1)
kΣXek n→∞

for all e ∈ Rp with kek = 1. 

Proof. From theorem 3.1 we know that, with a probability approaching 1

as n → ∞, the maximum likelihood estimator β̂ is the unique solution of


−∇ ln L(β) = X T σLR (Xβ) − y = 0 .

To simplify the notation we will thus assume that, for all n,

X T σLR (X β̂) − y = 0 . (4.1)

27
This condition can be reformulated as


X T σLR X(β 0 + β̂ − β 0 ) − y = 0
n
X  


⇐⇒ xij σLR xT β 0
+ x T
( β̂ − β 0
) − y i =0 ∀j
| i{z } | i {z }
i=1
=:γi0 =:ε̂i
n
X

⇐⇒ xij σLR (γi0 + ε̂i ) − yi = 0 ∀j .
i=1

In application of the mean value theorem, we can write σLR (γi0 + ε̂i ) =
0 (γ 0 ) + ε̂ σ
σLR 0
i i LR (ξi ) where ξi = γi + αi ε̂i with αi ∈ ]0, 1[. This gives us the
equivalent condition
n
X
0


⇐⇒ xij ε̂i σLR (ξi ) + σLR (γi0 ) − yi = 0 ∀j .
| {z }
i=1 =−ri

Let e ∈ Rp with kek = 1. We thus have

p P

X ej ni=1 xij ε̂i σLR
0 (ξ ) − r
i i
0=
kΣXek
j=1
p P p Pn

X ej ni=1 xij ri X ej i=1 xij ε̂i σLR
0 (ξ ) − σ 0 (γ 0 )
i LR i
=− +
kΣXek kΣXek
j=1 j=1
| {z } | {z } (4.2)
=:N1 =:N2
p P
X ej ni=1 xij ε̂i σLR
0 (γ 0 )
i
+ .
kΣXek
j=1
| {z }
=:N3

P
Let us first show that N2 −−−→ 0: We have
n→∞

max |ξi − γi0 | = max |γi0 + αi ε̂i − γi0 | = max |αi xT 0

i (β̂ − β )|
i∈{1,...,n} i∈{1,...,n} i∈{1,...,n}

≤ xT
i∗ (β̂ −β ) 0 ∗
for an i ∈ {1, . . . , n}.

By the inequality of Cauchy-Schwarz (C-S),

v
u p
uX
xT 0 0
x2i∗ j 0

i∗ ( β̂ − β ) ≤ kx i ∗ k β̂ − β = t β̂ − β
j=1
A1 √
≤ p M β̂ − β 0 .

28
 P
Moreover, as β̂ is consistent, kβ̂ − β 0 k −−−→ 0 from where it follows that
n→∞

P
max |ξi − γi0 | −−−→ 0 . (4.3)
i∈{1,...,n} n→∞

Therefore,
p Pn 0 (ξ ) − σ 0 (γ 0 )


X ej i=1 xij ε̂i σLR i LR i
|N2 | =
kΣXek
j=1
v
u p  n 2
C-S kek uX X

≤ t xij ε̂i σLR 0 (ξ ) − σ 0 (γ 0 ) 
 i LR i 
kΣXek 
j=1 i=1
r
1

0 (ξ ) − σ 0 (γ 0 ) 2
≤ p n2 M 2 ε̂2 max σLR i LR i
kΣXek i∈{1,...,n}
1 √  0 0

= p n M ε̂ max σLR (ξi ) − σLR (γi0 )
kΣXek i∈{1,...,n}

where ε̂ := maxi∈{1,...,n} |ε̂i |. For the denominator term, we get

v
u n
uX
kΣXek = t 0 (xT β 0 )(Xe)2 ≥ kXek
σLR min si
i i
i∈{1,...,n}
i=1

q
with si = 0 (xT β 0 ). However, on one hand,
σLR i

A1 √
|xT 0 0
i β | ≤ kxi k kβ k ≤ p M kβ 0 k

0 (√p M kβ 0 k) =: c . As, on the other hand, kXek ≥ √cn

such that s2i ≥ σLR 0
(combine (3.8) with A2), it follows that
√
kΣXek ≥ c0 c n (4.4)

and thus
1 √ √  0 0

|N2 | ≤ √ p M n ε̂ max σLR (ξi ) − σLR (γi0 ) (4.5)
c0 c i∈{1,...,n}

0
By the continuity of σLR we get with (4.3) that
 0 0
 P
max σLR (ξi ) − σLR (γi0 ) −−−→ 0 , (4.6)
i∈{1,...,n} n→∞

and, by Cauchy-Schwarz again,

√
ε̂ = max |ε̂i | = max |xT 0
i (β̂ − β )| ≤ p M kβ̂ − β 0 k
i∈{1,...,n} i∈{1,...,n}

29
Since p is fixed and thus XB(β 0 , δ) is bounded, there is a positive constant
C > 0 such that

|aδpn (X, β 0 )| = | inf min 0

σLR (ζi )| ≥ C .
“∈XB(˛ 0 ,δ) i∈{1,...,n}

√
Therefore, as a conclusion of the remark on page 22, n kβ̂ −β 0 k is bounded
√
in probability. The same holds thus for n ε̂ and we get with (4.5) and (4.6)
that
P
N2 −−−→ 0 .
n→∞

This result yields by (4.2) the condition

P
N1 + N3 −−−→ 0 . (4.7)
n→∞

Furthermore, we have
p P p P
X ej ni=1 xij ε̂i σLR
0 (γ 0 )
i
X ej ni=1 xij xT 0 0 T 0
i (β̂ − β ) σLR (xi β )
N3 = =
kΣXek kΣXek
j=1 j=1

eT X T D(β 0 )X(β̂ − β0 )
= .
kΣXek

As a result, if we can show that

p P
X ej ni=1 xij ri eT X T r L
−N1 = = −−−→ N (0, 1) , (4.8)
kΣXek kΣXek n→∞
j=1

L
we have by (4.7) that N3 −−−→ N (0, 1). For this purpose, we write −N1 as
n→∞
Pn
i=1 ρi
−N1 =
kΣXek
where ρi := (Xe)i ri . We note that
n
X n
X n
X n
X
Var(ρi ) = (Xe)2i Var(ri ) = (Xe)2i s2i = (ΣXe)2i = kΣXek2 .
i=1 i=1 i=1 i=1

Therefore, as a result of Lindeberg’s variant of the central limit theorem (see

Feller 1971, p. 262), (4.8) holds if the variances Var(ρi ) satisfy the Lindeberg
condition, i.e. for all t > 0,
n 
X E(ρ2i  |ρi | ≥ t kΣXek)
−−−→ 0 . (4.9)
kΣXek2 n→∞
i=1

30
However, we have

Var(ρi ) s2i (xT

i e)
2 s2i kxi k2 kek2 p M2
= ≤ ≤ −−−→ 0 , (4.10)
kΣXek2 kΣXek2 kΣXek2 4 c0 c n n→∞

the last inequality being a consequence of A1, (4.4), and the fact that
0 (xT β 0 ) ≤ 1 . Finally, (4.10) implies (4.9), which completes the
s2i = σLR i 4
proof. ¨

31
5 Case Study: Health Enhancing Physical Activ-
ity in the Swiss Population

In this last section, we shall study an example of applied logistic regression.

In 1999, the Swiss Federal Offices of Sports and Statistics carried out a
nationwide survey of attitudes, knowledge and behaviour related to health
enhancing physical activity (HEPA)10 . Based on recommendations11 of the
Swiss Federal Offices of Sports and Public Health as well as the Network
HEPA Switzerland they questioned 1,529 randomly chosen Swiss inhabitants
of at least 15 years of age. The samples for each of the German, French and
Italian speaking parts of Switzerland were designed to have an equal size in
order to obtain results of comparable precision.
Two criteria were used to determine whether a person was physically ac-
tive or not. First, people were asked whether they usually practised half an
hour of moderately intense physical activity each day, which is considered
to be the base criterion for HEPA. An activity of moderate intensity is char-
acterised by getting somewhat out of breath without necessarily sweating.
Brisk walking, hiking, dancing, gardening and sports were stated as exam-
ples for such an activity. In weighting the results accordingly to the language
region, the age, the gender and the household size of the questioned persons,
it was estimated that 50.6% of the Swiss population met the HEPA base
recommendations.
As a second criterion, people were questioned whether they practised regu-
larly at least three times a week during twenty minutes or more a sporting
activity of vigorous intensity. This was the case for 37.1% of the Swiss
population.
A person who met at least one of these criteria was designated as “physically
active”, a person who did not meet both of them was considered “physically
inactive”. This information can now be represented for each person by the
binary variable yi . Let H = {hi }i∈{1,...,n} denote the set of the questioned
individuals. We define
(
1 if hi is “physically inactive”,
yi :=
0 if hi is “physically active”.

It has to be noted that, among the 1,529 questioned Swiss inhabitants,

65 reported not to be able to walk a distance of more than 200 meters
without outside help. These and five others who did not give answers to the
criteria mentioned above will not be taken into consideration. Therefore, n
is assigned the value of 1,459.
10
First results have been published by Martin, Mäder and Calmonte (1999).
11
See Network HEPA Switzerland (1999).

32
In order to illustrate the application of the logistic regression model, we
shall now analyse the impact of the linguistic regions of Switzerland on
the physical activity of their inhabitants. We divide H in three mutually
exclusive sets HG , HF and HI containing the questioned inhabitants of the
German, French and Italian speaking parts of Switzerland respectively. For
every individual hi we define two indicator variables xF i and xIi by

xF i := 1HF (hi ) and xIi := 1HI (hi ) ,

where 1S is the indicator function respective to the set S. Let us now

estimate the coefficients β0 , β1 , and β2 of the logistic regression model

logit π = β0 + β1 xF + β2 xI .

For this purpose, we are going to minimise by the use of Mathematica the
error function Ey,σLR defined as in (3.1) with respect to the given data set.
Note that this procedure is absolutely equivalent to maximising the log-
likelihood function, as has been shown in section 3.1. While the vector y
contains the variables yi defined above, the matrix X is constructed by the
row vectors xT i := (1 xF i xIi ). The explicit values of y and X are not
going to be displayed at this place. We assume that they are stored in the
Mathematica variables y and X. The vector β = (β0 , β1 , β2 )T is represented
by the variable b:

b = {b0, b1, b2};

First of all, we assign the values of n and p:

{n, p} = Dimensions[X]
{1459, 3}

In addition, the vector 1n , the primitive function HσLR , and the error func-
tion Ey,σLR are defined and stored in ev, H, and Err respectively:

ev = Table[1, {e, 1, n}];

H[z ] := Log[Exp[z] + 1]
Err[b ] := ev.H[X.b] − y.(X.b)

Having specified y and X, it is now possible to print out the precise form
of Ey,σLR (β):

Err[b]
 
−721 b0 − 299
 b1 − 261
 b2 + 509  Log 1 + eb0 +
485 Log 1 + eb0+b1 + 465 Log 1 + eb0+b2

33
Finally, starting from β = 0, the function FindMinimum searches for a
local minimum in Ey,σLR :

betahat = FindMinimum[Err[{b0, b1, b2}],

{b0, 0}, {b1, 0}, {b2, 0} , WorkingPrecision− > 20]
{959.3452655931,
{b0 → −0.770799, b1 → 1.2455, b2 → 1.0172}}

The estimated coefficients are thus β̂ T = (−0.770799, 1.2455, 1.0172), and

we get the model equation

logit π = −0.770799 + 1.2455 xF + 1.0172 xI .

For the interpretation of these coefficients, we take into consideration the

comments given in section 2.3.2. From there we know that the odds on the
event that y equals 1 increase (or decrease) by the factor exp βi if xi grows
by one unit. For people living in the German speaking part of Switzerland,
both xF and xI are zero. Consequently, HG can be viewed as a “control
group” in the present investigation. Living in the French or Italian speaking
part of our country means from this viewpoint an increase of xF or xG by
one unit respectively. We shall thus calculate the exponentials of β̂1 and β̂2 :

{Exp[b1], Exp[b2]} /. betahat[[2]]

{3.47466, 2.76544}

As a result, the odds on an inhabitant of the French part of Switzerland

being physically inactive are almost 3.5 times as high as those of someone
living in the German part. For the Italian in comparison to the German
parts, the respective odds ratio is about 2.75.
As Martin and Mäder (2001), both members of the Swiss Federal Office of
Sports, state, these huge differences between the distinct linguistic regions
of Switzerland are astonishing. They assume that the perception of moder-
ately intensive physical activity is influenced by each indiviual’s social and
cultural background. The desirability of physical fitness is considered to
be particularly developed in the German part of Switzerland which could
have lead to a considerable overreporting (overclaiming) among this group
of questioned people.

34
6 Conclusion

The example provided in section 5 gives a very rudimentary impression

of the power of applied logistic regression. The analysis of this specific
approach for modelling binary data within the scope of this thesis, however,
has illustrated its versatility.
In the case of a fixed number of explanatory variables, both the consistency
and asymptotic normality of the maximum likelihood estimator are estab-
lished under two comparably weak hypotheses. Both of them are technical
conditions with respect to the “layout” of the independent input variables.
If, on the other hand, the number of exogenous variables is allowed to grow
along with the dimension of the sample space, it has been shown that there
had to be essentially a logarithmic relation between the former and the latter
in order to maintain the consistency of the maximum likelihood estimator.
Even without an explicit quantification of this relation, this result provides
an interesting insight into the interdependence of these two characteristics.

35
Acknowledgements

I would like to thank Prof Dr André Antille for the proposal of this interest-
ing and many-sided topic as well as for his academic assistance during the
development process of this diploma thesis.
Secondly, I want to express my gratitude to Prof Dr Bernard Marti, head of
the Institute of Sports Sciences within the Swiss Federal Office of Sports in
Magglingen, and to his collaborator Dr Brian W. Martin for placing at my
disposal the complete data set of the HEPA study used and referred to in
section 5. Thanks also to PD Dr Hans Howald for arranging the necessary
contacts.
Finally, my appreciation goes to Manrico Glauser and Ralf Lutz for proof-
reading this text and giving me precious feedback.

36
References

Amemiya, T. (1985). Advanced Econometrics, Harvard University Press,

Cambridge.

Antille, A. and Rech, M. (1999). Learning Single Layer Neural Networks:

Asymptotic Normality of Flat Spot Elimination Techniques, preprint.

Bhatia, R. (1997). Matrix Analysis, Graduate Texts in Mathematics,

Springer, New York.

Christensen, R. (1997). Log-Linear Models and Logistic Regression, Springer

Texts in Statistics, 2nd edn, Springer, New York.

Collett, D. (1991). Modelling Binary Data, Chapman & Hall, London.

Cramer, J. S. (1991). The LOGIT model: an introduction for economists,

Edward Arnold, London.

Feller, W. (1968). An Introduction to Probability Theory and Its Applica-

tions, Vol. I, 3rd edn, John Wiley & Sons, New York.

Feller, W. (1971). An Introduction to Probability Theory and Its Applica-

tions, Vol. II, 2nd edn, John Wiley & Sons, New York.

Garson, G. D. (2001). Statnotes: An Online Textbook [online], available

from: http://www2.chass.ncsu.edu/garson/pa765/statnote.htm.
[Accessed 23 September 2001].

Gourieroux, C. and Monfort, A. (1981). Asymptotic Properties of the Max-

imum Likelihood Estimator in Dichotomous Logit Models, Journal of
Econometrics 17: 83–97.

Hosmer, D. W. and Lemeshow, S. (1989). Applied Logistic Regression, John

Wiley & Sons, New York.

Kleinbaum, D. G. (1994). Logistic Regression: A Self-Learning Text, Statis-

tics in the Health Sciences, Springer-Verlag, New York.

Martin, B. W. and Mäder, U. (2001). Körperliches Aktivitätsverhalten in

der Schweiz, preprint.

Martin, B. W., Mäder, U. and Calmonte, R. (1999). Einstellung, Wissen und

Verhalten der Schweizer Bevölkerung bezüglich körperlicher Aktivität:
Resultate aus dem Bewegungssurvey 1999, Schweizerische Zeitschrift
für Sportmedizin und Sporttraumatologie 47: 165–169.

Mazza, C. and Antille, A. (1998). Learning Single Layer Neural Networks:

Consistency of Flat Spots Elimination Techniques, preprint.

37
McFadden, D. (1974). Conditional logit analysis of qualitative choice be-
haviour, in P. Zarembka (ed.), Frontiers in Econometrics, Academic
Press, New York, pp. 105–142.

National Safety Council (2001). What Are the Odds of Dying? [online],
available from: http://www.nsc.org/lrs/statinfo/odds.htm. [Ac-
cessed 15 October 2001].

Network HEPA Switzerland (1999). Health Enhancing Physical Activ-

ity HEPA (Recommendations) [online], Available from: http://
www.hepa.ch/english/pdf-hepa/Empf_e.pdf. [Accessed 10 Novem-
ber 2001].

Ortega, J. M. and Rheinboldt, W. C. (1970). Iterative Solutions of Nonlinear

Equations in Several Variables, Academic Press, New York.

Santner, T. J. and Duffy, D. E. (1989). The Statistical Analysis of Discrete

Data, Springer Texts in Statistics, Springer, New York.

38