1

A Simple Example of Finding MLE’s Using SAS PROC NLP

With the Nelder-Mead Simplex Method, Followed by Newton’s Method

We want to obtain MLE’s for a simple example in which a relatively small (n = 5) data set has been
sampled from a normal distribution with mean µ and finite variance σ 2. The data values are: x 1=1,
x 2=3, x 3=4, x 4 =5, and x 5=7 .

We will use SAS PROC NLP to obtain simultaneous MLE’s for both parameters using the Nelder-Mead
Simplex algorithm, followed by refinement of the parameter estimates using Newton’s method.

The log-likelihood function to be maximized is:

n
−n 1
ln ( 2 π )−nln ( σ )− 2 ∑ ( x i−μ ) .
2
(1) l ( μ ,σ ⃗
; x )=
2 2 σ i=1
In PROC NLP, we enter this function for a single observation; SAS then recognizes that the data
represent n observations from the distribution.

The Nelder-Mead method (without modification) does not necessarily converge to a minimum point for
general non-convex functions. However, it does tend to produce “a rapid initial decrease in function
values…” 1 regardless of the form of the function.

In this example, we are finding the extreme point of a convex function; hence the Nelder-Mead method
should work well. The SAS program for implementing the first algorithm is shown below, followed by
an explanation of the options and statements in the PROC, and the output.

data Ex1;
input x @@;
datalines;
1 3 4 5 7
;
proc nlp data=Ex1 tech=nmsimp vardef=df covariance=h pcov phes;
max loglik;
parms mean=0., sigma=1.;
bounds sigma > 1e-12;
loglik=-0.5*((x-mean)/sigma)**2-log(sigma);
;
run;

Options in the PROC NLP statement:

a) “data=” tells the procedure where to find the data set.

b) “tech” specifies the method for numeric optimization; in this case “nmsimp” tells SAS to use the
Nelder-Mead simplex method. “newrap” would specify the use of the Newton-Raphson method.
If no method is specified, the default method is Newton-Raphson.
2

c) “vardef” specifies the divisor to be used in the calculation of the covariance matrix and standard
errors. The default value is N, the sample size. Here, “df” specifies that the degrees of freedom
should be used as the divisor. (See p. 371 of the manual.)
d) “covariance” specifies that an approximate covariance matrix for the parameter estimates is to be
computed. There are several ways to do this (See p. 370 of the manual.) The option “h” used
here specifies that, since the “max” statement is used in the procedure, the covariance matrix
n −1
should be computed as G , where G is the Hessian matrix (the matrix of second
max { 1 ,n−df }
partial derivatives of the objective function, (1), with respect to the parameters.)
e) “pcov” tells SAS to display the covariance matrix.
f) “phes” tells SAS to display the Hessian matrix.

Other statements:

a) The “max” statement specifies that the maximum point for the objective function (1) is to be
found. The “loglik” variable denotes the objective function.
b) The “parms” statement specifies initial values of the parameters, to be used in this case to
generate the initial simplex for starting the Nelder-Mead algorithm. (See p. 374 of the manual.)
c) The “bounds” statement specifies constraints on the values of the parameters. Here, the only
constraint is that σ must be positive. (See p. 325 of the manual.)
d) The objective function to be maximized is defined by a statement in the procedure. Here, we

( )
2
( ) −1 x−μ
define loglik μ , σ = −ln ( σ ).
2 σ

The output of the program is listed below:

The SAS System
PROC NLP: Nonlinear Maximization

Gradient is computed using analytic formulas.

Hessian is computed using finite difference approximations (2) based on analytic gradient.

The SAS System

PROC NLP: Nonlinear Maximization
Optimization Start
Parameter Estimates
Gradient Lower Upper
Objective Bound Bound
N Parameter Estimate Function Constraint Constraint
1 mean 0 20.000000 . .
2 sigma 1.000000 95.000000 1E-12 .

Value of Objective Function = -50

Hessian Matrix
mean sigma
mean -5 -40.00000001
sigma -40.00000001 -295.0000001

Determinant = -124.9999997
Matrix has 1 Positive Eigenvalue(s)
3

WARNING: Second-order optimality condition violated.

The SAS System

PROC NLP: Nonlinear Maximization
Nelder-Mead Simplex Optimization
Parameter Estimates 2
Functions (Observations) 5
Lower Bounds 1
Upper Bounds 0

Optimization Start
Active Constraints 0
Objective Function -50

Objective Std Dev Restart

Function Active Objective Function of Simplex Vertex Simplex
Iter Restarts Calls Constraints Function Change Values Length Size

1 0 11 0 -6.40098 1.7805 0.7758 1.000 2.750

2 0 20 0 -6.08749 0.3135 0.1291 1.000 1.438
3 0 29 0 -5.97918 0.0496 0.0218 1.000 0.262
4 0 38 0 -5.96671 0.00693 0.00312 1.000 0.101
5 0 48 0 -5.96581 0.000095 0.000039 1.000 0.0213
6 0 58 0 -5.96574 1.57E-6 6.954E-7 1.000 0.00459

Optimization Results
Iterations 6 Function Calls 60
Restarts 0 Active Constraints 0
Objective Function -5.965738194 Std Dev of Simplex Values 6.9541052E-7
Deltax 1 Size 0.0045923843

FCONV2 convergence criterion satisfied.

NOTE: At least one element of the (projected) gradient is greater than 1e-3.

The SAS System

PROC NLP: Nonlinear Maximization
Optimization Results
Parameter Estimates
Gradient
Approx Approx Objective
N Parameter Estimate Std Err t Value Pr > |t| Function
1 mean 3.998087 0.894406 4.470100 0.006579 0.002392
2 sigma 1.999952 0.632418 3.162391 0.025028 0.000123
Value of Objective Function = -5.965738194

Hessian Matrix
mean sigma
mean -1.250060277 -0.002392044
sigma -0.002392044 -2.500304826

Determinant = 3.1255260207
Matrix has Only Negative Eigenvalues

Covariance Matrix 2: H = (NOBS/d) inv(G)

mean sigma
mean 0.7999628893 -0.000765325
sigma -0.000765325 0.399951966

Factor sigm = 1
Determinant = 0.3199461445

Matrix has 2 Positive Eigenvalue(s)

Approximate Correlation Matrix

of Parameter Estimates
mean sigma
mean 1 -0.001353029
sigma -0.001353029 1
4

Determinant = 0.9999981693
Matrix has 2 Positive Eigenvalue(s)

The final parameter estimates obtained using the Nelder-Mead method were ^μ=3.998087 and
σ^ =1.999952. We will use these as our initial guesses of the parameters, and refine these values by
using Newton’s method.

The program for implementing Newton’s method is listed below, followed by the output. The final
MLE’s for the parameters (after 1 iteration) were found to be ^μ=4.000000 and σ^ =1.9999 99.

data Ex1;
input x @@;
datalines;
1 3 4 5 7
;
proc nlp data=Ex1 tech=newrap vardef=n covariance=h pcov phes;
max loglik;
parms mean=3.998087, sigma=1.999952;
bounds sigma > 1e-12;
loglik=-0.5*((x-mean)/sigma)**2-log(sigma);
;
run;
The SAS System
PROC NLP: Nonlinear Maximization
Gradient is computed using analytic formulas.
Hessian is computed using analytic formulas.
The SAS System
PROC NLP: Nonlinear Maximization
Optimization Start
Parameter Estimates
Gradient Lower Upper
Objective Bound Bound
N Parameter Estimate Function Constraint Constraint
1 mean 3.998087 0.002391 . .
2 sigma 1.999952 0.000122 1E-12 .

Value of Objective Function = -5.965738193

Hessian Matrix
mean sigma
mean -1.250060002 -0.002391422
sigma -0.002391422 -2.500303451
Determinant = 3.125523618

Matrix has Only Negative Eigenvalues

The SAS System

PROC NLP: Nonlinear Maximization
Newton-Raphson Optimization with Line Search
Without Parameter Scaling
Parameter Estimates 2
Functions (Observations) 5
Lower Bounds 1
Upper Bounds 0
Optimization Start
Active Constraints 0
Objective Function -5.965738193
5

Max Abs Gradient Element 0.0023913648

Objective Max Abs Slope of

Function Active Objective Function Gradient Step Search
Iter Restarts Calls Constraints Function Change Element Size Direction

1 0 2 0 -5.96574 2.29E-6 2.294E-6 1.000 -458E-8

Optimization Results
Iterations 1 Function Calls 3
Hessian Calls 2 Active Constraints 0
Objective Function -5.965735903 Max Abs Gradient Element 2.2942694E-6
Slope of Search Direction -4.580223E-6 Ridge 0

ABSGCONV convergence criterion satisfied.

The SAS System

PROC NLP: Nonlinear Maximization
Optimization Results
Parameter Estimates
Gradient
Approx Approx Objective
N Parameter Estimate Std Err t Value Pr > |t| Function
1 mean 4.000000 0.894427 4.472138 0.006566 0.000000113
2 sigma 1.999999 0.632455 3.162280 0.025031 0.000002294
Value of Objective Function = -5.965735903

Hessian Matrix
mean sigma
mean -1.250001147 -1.125884E-7
sigma -1.125884E-7 -2.500005736

Determinant = 3.1250100374
Matrix has Only Negative Eigenvalues

Covariance Matrix 2: H = (NOBS/d) inv(G)

mean sigma
mean 0.7999992658 -3.602817E-8
sigma -3.602817E-8 0.3999990823

Factor sigm = 1

Determinant = 0.3199989722
Matrix has 2 Positive Eigenvalue(s)

Approximate Correlation Matrix

of Parameter Estimates
mean sigma
mean 1 -6.368951E-8
sigma -6.368951E-8 1

Determinant = 1

Matrix has 2 Positive Eigenvalue(s)

6

1
Lagarias, J. C.; Reeds, J. A.; Wright, M. H.; and Wright, P. E. (1998). “Convergence Properties of the Nelder-
Mead simplex method in low dimensions,” SIAM Journal on Optimization, 9, 1, pp. 112-147.