Maximum Likelihood Estimation: Numerical Solution for Weibull

Distribution1

The Weibull distribution has the probability density function:

λy λ−1
   
y λ
f (y|θ, λ) = exp − (1)
θλ θ

where: y > 0, θ is the scale parameter and λ is the shape parameter (nuisance parameter).

Maximum Likelihood Estimation

Let y1 , . . . , yn , denote the data. Assume, ∀i : yi independente random variables, share the same
parameters from a Weibull distribution described in (1).

λyi λ−1
   
yi λ
f (yi |θ, λ) = exp − (2)
θλ θ
then, their join probability distribution is:
n
Y
f (y1 , . . . , yn |θ, λ) = f (yi |θ, λ)
i=1
n
λyi λ−1
   
Y yi λ
= exp − (3)
i=1
θλ θ

The likelihood function is:

L (θ|y1 , . . . , yn , λ) = f (y1 , . . . , yn |θ, λ) (4)

The log-likelihood function is:

` (θ|y1 , . . . , yn , λ) = log (L (θ|y1 , . . . , yn , λ))

n
λyi λ−1
   
Y yi λ
= log exp −
i=1
θλ θ
n
λyi λ−1
X    y λ 
i
= log λ
exp −
i=1
θ θ
X n   y λ 
i
` (θ|y1 , . . . , yn , λ) = log λ + (λ − 1) log yi − λ log θ − (5)
i=1
θ

The maximum likelihood estimator (MLE), denoted by θ̂, is such that:

θ̂ = argmax {` (θ|y1 , . . . , yn , λ)} (6)

1
To refer this document and the implemented code, please cite as: Alvarado, M. (2020, August 8). Maximum
Likelihood Estimation: Numerical Solution for Weibull Distribution (Version v1.0.0). Zenodo. http://doi.org/10.
5281/zenodo.3986931. Also at GitHub: https://github.com/miguel-alvarado-stats/MLE_Weibull.

1
 To maximize the log-likelihood function (5), requires the derivative with respect to θ. The resulted
function is called the Score function, denoted by U (θ|y1 , . . . , yn , λ).
∂` (θ|y1 , . . . , yn , λ)
U (θ|y1 , . . . , yn , λ) =
∂θ
n 
λ λyi λ
X 
= − + λ+1
i=1
θ θ
n
λn λ X λ
= − + λ+1 yi (7)
θ θ i=1

Then, the MLE θ̂, is the solution of:

 
U θ = θ̂|y1 , . . . , yn , λ = 0 (8)

Maximum Likelihood Estimation: Newton-Raphson Method

Just for notation, let write equation (8) as:

U (θ∗ ) = 0 (9)

The equation (9), generally, is a nonlinear equation, that can be aproximate by Taylor Series:

U (θ∗ ) ≈ U θ(t) + U 0 θ(t) θ∗ − θ(t)

(10)

Then, using (10) into (9), and solving for θ∗ :

U θ(t) + U 0 θ(t) θ∗ − θ(t) = 0

(t)


U θ
θ∗ = θ(t) − 0 (t) (11)
U (θ )

where U 0 is the derivative of the Score function (7) respect of θ.

∂U (θ|y1 , . . . , yn , λ)
U 0 (θ|y1 , . . . , yn , λ) =
∂θ
n
λn λ (λ + 1) X λ
= − yi (12)
θ2 θλ+2 i=1

Then, with the Newton-Raphson method: starting with an initial guess θ(1) successive approxima-
tions are obtained using (13), until the iterative process converges.


(t+1) (t) U θ(t)
θ = θ − 0 (t) (13)
U (θ )

In order to example the use of Newton-Rapshon method, we use the data of lifetimes (times to
failure in hours) of Kevlar epoxy strand pressure vessels at 70% stress level2 .

2
This data was taken from Dobson, A. Barnett, A. (2018) An Introduction to Generalized Linear Models. Texts
in Statistical Science. Chapman Hall/CRC.

2
# load data
Y <- as.matrix(read.delim(file = "data.txt", header = FALSE))

We load the code developed into our R function MLE_NR_Weibull, stored in the R object with the
same name.

# load the function to solve by Newton-Raphson

load("MLE_NR_Weibull.RData")

The function MLE_NR_Weibull takes the sample mean θ = ȳ as a first guess for the iterative process
and, besides some other default parameters that can be modified such as λ = 2, only needs the data
vector Y.

# MLE by Newton-Raphson (NR) for Weibull distribution

MLE_NR_Weibull(Y)

## ML Estimator Likelihood Log-Likelihood

## [1,] "8805.693878" "3.51556268514669e-210" "-482.285669935176"
## [2,] "9633.777408" "1.37775513478331e-209" "-480.919828975047"
## [3,] "9875.898292" "1.47709267871969e-209" "-480.850208686227"
## [4,] "9892.110004" "1.47748579671734e-209" "-480.849942578555"
## [5,] "9892.176818" "1.47748580332318e-209" "-480.849942574083"
## [6,] "9892.176819" "1.47748580332318e-209" "-480.849942574084"

Then, the MLE by Newton-Raphson method: θ̂ = 9892.176819.

Maximum Likelihood Estimation: Fisher-Scoring Method

A distribution belongs to the exponential family if it can be written in the form:

 
a (y) b (θ) − c (θ)
f (y|θ) = exp + d (y, φ) (14)
φ

Since (1) can be written as a member of exponential family as in (14):

  λ−1    
λy y λ
f (y|θ, λ) = exp log exp −
θλ θ
= exp y −θ−λ − (λ log θ − log λ) + (λ − 1) log y
 λ

(15)

where, a (y) = y λ , b (θ) = −θ−λ , c (θ) = λ log θ − log λ, φ = 1, and d (y, φ) = (λ − 1) log y.

Then, since the Weibull distribution belongs to the exponential family, it can be show that the
variance of U , denoted by J , is:

J = Var {U } = −E {U 0 } (16)

where:
c0 (θ)
 
0 1 00
E {U } = − b (θ) 0 − c00 (θ) (17)
φ b (θ)

3
 For MLE, it is common to approximate U 0 by its expected value E {U 0 }. In this case:

J = −E {U 0 }
= E {−U 0 }
( n )
X
= E − Ui0
i=1
n
X
= −E {Ui0 }
i=1
n
c0 (θ)
 
X 1 00 00
= − b (θ) 0 − c (θ) (18)
i=1
φ b (θ)

where, using (1), the previous derivaties:

b0 (θ) = λθ−(λ+1)
b00 (θ) = −λ (λ + 1) θ−(λ+2)
c0 (θ) = λθ−1
c00 (θ) = −λθ−2
1
= 1
φ
Then, replacing them into (18):
n  −1

−(λ+2) λθ
X
−2
J = − −λ (λ + 1) θ + λθ
i=1
λθ−(λ+1)
n
X λ2
=
θ2
i=1
 2
λ
= n (19)
θ
Finally:
 2
0 λ
J = −E {U } = n
θ
 2
λ
−J = E {U 0 } = n (20)
θ

Then, approximating U 0 by its expected value E {U 0 }, the equation (13) results into:
(t)


U θ
θ(t+1) = θ(t) + (21)
J (θ(t) )
In order to example the use of Fisher-Scoring method, we use the same data used in the Newton-
Rapshon method. We load the code developed into our R function MLE_FS_Weibull, stored in the R
object with the same name.

4
# load the function to solve by Fisher-Scoring
load("MLE_FS_Weibull.RData")

The function MLE_FS_Weibull takes the sample mean θ = ȳ as a first guess for the iterative process
and besides some other default parameters that can be modified such as λ = 2, only needs the data
vector Y.

# MLE by Fisher-Scoring (FS) for Weibull distribution

MLE_FS_Weibull(Y)

## ML Estimator Likelihood Log-Likelihood

## [1,] "8805.693878" "3.51556268514669e-210" "-482.285669935176"
## [2,] "9959.204199" "1.47092711455647e-209" "-480.854391543592"
## [3,] "9892.402373" "1.47748572804852e-209" "-480.849942625031"
## [4,] "9892.176822" "1.47748580332318e-209" "-480.849942574083"
## [5,] "9892.176819" "1.47748580332319e-209" "-480.849942574084"

Then, the MLE by Fisher-Scoring method: θ̂ = 9892.176819.

In summary, the same estimate for the MLE is achieved by both approaches: the Newton-Raphson
and the Fisher-Scoring method.

Naive Approach

The main idea behind the Maximum Likelihood (ML) method is to choose those estimates for the
unknown parameters that maximize the join probability of our observed data (our sample). Keeping in
mind this idea, if we want to get the MLE and avoiding to implement a numerical solution, a naive
approach is to set a large range of possible values for unknown parameters, evaluate the log-likelihood
function (also, the likelihood function) at each point and the point for which the log-likelihood function
(also, the likelihood function) reaches its maximum value, will be our MLE looking for.

We set a set of values por the parameter, from θ̂ = 4000 to θ̂ = 13.000, spaced by 1; and evaluate
(4) and (5) at each point from the set of values.

# package dplyr is required to use the pipe operator

library(dplyr)

# set a large range of values for the parameter

theta <- seq(4000, 13000, 1)
tabla <- matrix(c(NA,NA,NA), nrow = length(theta), ncol = 3)
tabla[,1] <- theta

# evaluate the likelihood and the log-likelihood at each point

for (i in 1:length(theta)){
tabla[i,2] <- prod(dweibull(Y, shape = 2, scale = theta[i], log = FALSE))
tabla[i,3] <- sum(dweibull(Y, shape = 2, scale = theta[i], log = TRUE))
}
colnames(tabla) <- c("theta", "Likelihood", "Log-Likelihood")
df_tabla <- as.data.frame(tabla)

5
 Then, the plot of likelihood function evaluated at each point:

Figure 1: Likelihood Function

1.5e−209

1.0e−209
Likelihood

5.0e−210

0.0e+00

4000 6000 8000 10000 12000

parameter

The point for which the likelihood function is maximum, is our MLE from a naive approach:

mxl <- max(tabla[,2])

df_tabla %>% filter(`Likelihood` == mxl)

## theta Likelihood Log-Likelihood

## 1 9892 1.477486e-209 -480.8499
Also, the plot of log-likelihood function evaluated at each point:

Figure 2: Log−Likelihood Function

−500
Log−Likelihood

−550

−600

4000 6000 8000 10000 12000

parameter

6
 The point for which the log-likelihood function is maximum, which is the same point at the likelihood
function reaches its maximum value, is our MLE from a naive approach:

mxllog <- max(tabla[,3])

df_tabla %>% filter(`Log-Likelihood` == mxllog)

## theta Likelihood Log-Likelihood

## 1 9892 1.477486e-209 -480.8499

As we can see, using this naive approach, we reach a value that is close enough to that which is
reached using the numerical solution: the Newton-Raphson and the Fisher-Scoring method.