Journal of Interdisciplinary Mathematics

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/tjim20

A new class of quasi-Newton updating formulas

for unconstrained optimization

Basim A. Hassan & Mohammed Abdulrazaq Kahya

To cite this article: Basim A. Hassan & Mohammed Abdulrazaq Kahya (2021) A new class
of quasi-Newton updating formulas for unconstrained optimization, Journal of Interdisciplinary
Mathematics, 24:8, 2355-2366, DOI: 10.1080/09720502.2021.1961980

To link to this article: https://doi.org/10.1080/09720502.2021.1961980

Published online: 11 Jan 2022.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=tjim20
Journal of Interdisciplinary Mathematics
ISSN: 0972-0502 (Print), ISSN: 2169-012X (Online)
Vol. 24 (2021), No. 8, pp. 2355–2366
DOI : 10.1080/09720502.2021.1961980

A new class of quasi-Newton updating formulas for unconstrained

optimization

Basim A. Hassan †
Department of Mathematics
College of Computers Sciences and Mathematics
University of Mosul
Mosul
Iraq

Mohammed Abdulrazaq Kahya *

Department of Computer Sciences
College of Education of Pure Sciences
University of Mosul
Mosul
Iraq

Abstract
Problems of sciences concerned with minimizing an objective function that depends
on real values without restrictions on there is called unconstrained optimization problems.
Quasi-Newton methods are one of the most common approaches to solve unconstrained
optimization problems. Mainly, the quasi-Newton equation is the focus of quasi-Newton
methods. In this paper, we extended the quasi-Newton equation introduced by Razieh et al.
[1] and some new quasi-Newton methods are presented. The convergence behaviors of the
proposed methods were debated. It is worth mentioning that the proposed method is proved
better in performance over other competitive methods and it’s able to solve unconstrained
optimization problems.

Subject Classification: (2010) Primary: 49M15, Secondary: 90C53.

Keywords: Quasi-Newton methods, Extend quasi-Newton equations, Convergence properties.

†
E-mail: [email protected]
* E-mail: [email protected] (Corresponding Author)

©
2356 B. A. HASSAN AND M. A. KAHYA

1. Introduction
Many iterative methods have been proposed to find a minimum of a
problem of the form, f : Rn ® R, f Î C 2 , for more details see [2]. Quasi-
Newton methods are an important type of iterative methods for solving
optimization problems. As it is known, the goal of quasi-Newton creates
a perfect approximation to the Hessian matrix to construct a sequence of
symmetric positive definite matrix Bk +1 . The quasi-Newton methods take
the form:
xk +1 = xk + a k dk (1)

where dk is a search direction and a k is the parameter determined by

exact line-search (step length) as :
gkT dk
ak = - (2)
dkT Qdk

and satisfy the Wolfe line-search conditions:

f (xk + a k dk ) £ f (xk ) + da k gkT dk (3)

dkT g(xk + a k dk ) ³ s dkT gk (4)

where 0 < d < s < 1 [3]. To obtain the search direction, we do the following
steps:
Bk dk + gk = 0 (5)

where matrices {Bk } approximate Hessian matrix Ñ 2 f (xk ) and they satisfy
the quasi-Newton equation :
Bk +1 sk = y k (6)

where gk = Ñf (xk ) is the gradient of f (xk ), sk = xk +1 - xk = a k dk and

y k = gk +1 - gk , [4].
Yuan G. and et al [5] presented a successful famous type of the quasi-
Newton methods, BFGS’s method which the Hessian updates are defined
by:
Bk sk skT Bk T y k y kT
BkBFGS
+1
= Bk - + (7)
skT Bk sk skT y k

where they assumed that H k be is the inverse of Bk . The BFGS method

updated to:
QUASI-NEWTON UPDATING FORMULAS FOR UNCONSTRAINED OPTIMIZATION2357

H k y k skT + sk y kT H k é y T H y ù s sT
H kBFGS
+1
= Hk - + ê1 + k T k k ú Tk k . (8)
skT y k êë sk y k úû sk y k

The BFGS method has accurate numerical results, but it may fail in
proving convergence for non-convex functions with an inexact line search
[6]. To overcome convergence problem, different approaches have been
proposed to make modifications to the quasi-Newton equations in order
to obtain different updating formulas that represented approximation
for the Hessian matrix (Wei et al. [7], Biglari et al. [8], Chen et al. [9]
and Hassan [10]). Many other modified methods were presented in [3].
[11]–[14]. An important point to be addressed in the papers of the BFGS
methods (or quasi-Newton methods) is global convergence property when
solving nonconvex minimization problems. It has been regarded as one of
the most fundamental problems in the field of unconstrained optimization
[13]. In particular, many authors have proved the global convergence of
the quasi-Newton methods via line search techniques enhancement [12],
[14] . On the other hand, there is another way to get a global convergence
via improved approximations of the Hessian matrix. In order to overcome
the BFGS hurdles and to get a more accurate approximation of the second
curvature of the objective function, we have presented an alternative
quasi-Newton equation as below:

2. Deriving the new updating of the quasi-Newton equation.

Razieh et al. [1] proposed a modification notion of the quasi-Newton
equation to improve it as follows:
12( fk - fk +1 ) + 5 gkT+1 sk + 7 gkT sk - a k gkT sk
Bk +1 sk = y k + rk . (9)
skT rk

In order to get a new modified version of the quasi-Newton equation,

we get Equation (10) via multiply both sides of Equation (9) by skT :
skT Bk +1 sk = skT y k + 12( fk - fk +1 ) + 5 gkT+1 sk + 7 gkT sk - a k gkT sk (10)

We get Equation (11) via dividing both sides of (10) by gamma:

1 1 12 5 7 ak T
skT Bk +1 sk = skT y k + ( fk - fk +1 ) + gkT+1 sk + gkT sk - g s (11)
g g g g g g k k
Since Bk +1 approximate to Hessian matrix Q, then the relation
Equation (11), together with Equation (2), results in :
2358 B. A. HASSAN AND M. A. KAHYA

1 12 5 æ 8 -ak -g ö T
skT Bk +1 sk = skT y k + ( fk - fk +1 ) + gkT+1 sk + ç ÷ gk sk (12)
g g g è g ø

Now, based on Equation (12), we obtained the following equation :

12 5 æ 8 -ak -g ö T
( fk - fk +1 ) + gkT+1 sk + ç ÷ø gk sk
1 g g è g
Bk +1 sk = yˆ k = yk + rk (13)
g skT rk

where rk is any vector such that skT rk ¹ 0.

It’s very clear that choose different values of the gamma in Equation
(13) give us new quasi-Newton equations. Some of the recommended
gamma values are 12, 5, and (8 - a k )/ 2. Now we use these values of
gamma to get three equations of the quasi-Newton:

(1) Putting g = 12 in Equation (13), thus we get:

5 T æ -4 - a k ö T
( fk - fk +1 ) + gk +1 sk + ç gk sk
1 12 è 12 ÷ø
yˆ k = y + rk
12 k skT rk
.

(2) Putting g = 5 in Equation (13), thus we get:

12 æ 3 -ak ö T
( fk - fk +1 ) + gkT+1 sk + ç gk sk
1 5 è 5 ÷ø
yˆ k = y + rk
5 k skT rk
.

(3) Putting g = (8 - a k )/ 2 in Equation (13), thus we get:

24 10
( fk - fk +1 ) + g T s + gkT sk
2 (8 - a k ) (8 - a k ) k +1 k
yˆ k = y + rk
(8 - a k ) k skT rk

On the other hand, we introduced a more reasonable rk , that leads to

a new quasi-Newton equation where rk = gk +1 . The good property of the
new equations in the next theorem can be seen.

Theorem 1 : Let yˆ k be defined by Equation (13). Then the following statements

skT yˆ k ³ 0 hold.

Proof : Multiplying definition yˆ k by skT , thus we get :

QUASI-NEWTON UPDATING FORMULAS FOR UNCONSTRAINED OPTIMIZATION2359

1 12 5 æ 8 -ak -g ö T
skT yˆ k = skT y k + ( fk - fk +1 ) + gkT+1 sk + ç ÷ gk sk (14)
g g g è g ø
12 6 æ 7 -ak -g ö T
= ( fk - fk +1 ) + skT gk +1 + ç ÷ gk sk
g g è g ø

Using the Wolfe condition with the above equation thus we get:
æ 12 6 7 -ak -g ö
skT yˆ k ³ gkT dk ç - da k + sa k + a k ÷ (15)
è g g g ø

Since d < s then from the above inequality, thus we get:

æ6 æ 7 -ak -g öö
skT yˆ k ³ - gkT sk ç d - ç ÷÷ (16)
èg è g øø

Make a note of the skT gk = a k dkT gk < 0, we know that there exists a
constant w < 0 such that:
skT yˆ k ³ w dkT gk > 0. (17)

The proof has finished.

For general functions, if skT yˆ k > 0 maybe not satisfied, then {Bk + 1 } it
may not be positive definite[15]. To overcome the problem, we simulate
the approach used by Yuan Y. and et al.[11], therefore, we can define our
modification as follows:
ìï 12 5 æ 8 -ak -g ö T üï
Max í0 , ( fk - fk +1 ) + gkT+1 sk + ç ÷ø gk sk ý
1 ï
î g g è g þï
yˆ k = y + rk .
g k skT rk

which guarantees the positive definiteness of the matrix. Depending

on the gamma values, we get the special cases of the quasi-Newton
equations. According to what was presented, we can present a new
algorithm.

Algorithm 1

Stage 1. Give e , an initial point x0 Î Rn . Set k = 0.

Stage 2. If possible, test satisfies then stop.

Stage 3. Compute a search direction: Solve Bk dk = - gk .

2360 B. A. HASSAN AND M. A. KAHYA

Stage 4. Find a k by using the following Wolfe conditions.

Stage 5. Set xk +1 = xk + a k dk . Compute yˆ k by (17). If skT yˆ k > 0, update H k +1

by Equation , otherwise, let H k +1 = H k . Set k = k + 1 and go to Step 2.

3. Global convergence analysis

Global convergence is considered the success criterion for the
optimization algorithms. Therefore, we need some of the assumptions to
prove the global convergence of the algorithm:
Let f is bounded below in R n and that f is smooth in an open set N
containing the level set G = {x Î Rn : f (x ) £ f (x0 )}, where x0 is the initial
point of the method. Set Ncontaining G , there exists a constant L > 0 and
g such that :
g(x ) - g( y ) £ L x - y , " x , y Î N (18)

and
g(x ) £ g , " x , y Î N (19)

Since { fk } is a non-increasing, it is clear that the {xk } Ì N and their

existence a constant f * such that :
lim fk = f * . (20)
k ®¥

More specifics has presented in [16], [17].

Theorem 2 : Let {xk } be generated by the new algorithm and there exist constants
a1 and a2 such that they satisfy the relation :
2
Bk sk £ a1 sk and skT Bk s2 ³ a2 sk (21)

for infinitely k, then we get:

lim inf gk = 0 (22)
k ®¥

Proof : similar to the proof done in Hassan [10].

Theorem 3 : Assume that f satisfies assumption and {xk } generated by

Algorithm 1. Then, we get:
lim inf gk = 0 (23)
k ®¥
QUASI-NEWTON UPDATING FORMULAS FOR UNCONSTRAINED OPTIMIZATION2361

Proof : Depending on the Theorem 2, it suffices to verify Equation when

ïì skT yˆ k ïü
K is infinite. Suppose K = ík : ³ d ý . Assuming that K is a finite set,
îï sk 2 þï
depending on the definition of K, Equation holds. Letting K is an infinite
set, yields:
2
skT yˆ k ³ d sk (24)

Based on that f is bounded and Equation , then there exists a positive

number M0 such that (see [12]):
2
yk
£ M0 . (25)
skT y k

Using the definitions of yˆ k , we get :

1 ìï 12 5 æ 8 -ak -g ö T üï
skT yˆ k = skT y k + Max í0 , ( fk - fk +1 ) + gkT+1 sk + ç ÷ gk sk ý
g îï g g è g ø þï
1
³ skT y k . (26)
g

and

ìï 12 5 æ 8 -ak -g ö T ïü
Max í0 , ( fk - fk +1 ) + gkT+1 sk + ç ÷ø gk sk ý
1 ïî g g è g ïþ
yˆ k = y + r
g k skT rk
k
k

1
£ yk + yk (27)
g

We add Equation to Equation , thus we obtain:

2
é1 ù 2

yˆ k
2
êg + 1ú y k
£ë û £ M. (28)
skT yˆ k 1 T
sk y k
g

We get the conclusion of this Theorem.

Applying Theorm 1 to the {Bk }; $ a1 , a2 > 0 be given constants such
that Equation holds for infinitely many k. Thus, the proof has finished.
2362 B. A. HASSAN AND M. A. KAHYA

4. Numerical Experiments
For a fair test of the method performance, the proposed method
(extension of BFGS) was compared with standard BFGS and BBFGS
proposed by Hassan[10].
The following tables contain the number of iterations and the
number of function evaluations. The methods were terminated if
“| f (xk ) | > 10 -5 , let stop 1 = | f (xk ) - f (xk +1 ) | / | f (xk ) | ; Otherwise, let
stop 1 = | f (xk ) - f (xk +1 ) | . For every problem, if  gk  < e or stop 1 < 10 -5
is satisfied, the program will be stopped”. Stop rule used as the Himmeblau
stop rule used by Yuan Y. and et al.[18]. In all algorithms, the H 0 = I where
I is an identity matrix, and the step length a k was computed satisfying
the Wolfe conditions, with d = 0.1, s = 0.9 and e = 10 -5. All programs
were written in Matlab, using 30 test problems. Tested problems used in
this paper are the same benchmark problems were used by More J. and et
al. [19].

Table 1
Comparison between the influence of gamma values over the proposed
method and rival methods in terms of #iterations and #functions evaluation.

g = (8-ak)/2
BFGS with

BFGS with
gamma=12

BFGS with
algorithm

algorithm

gamma=5
BBFGS
BFGS

Problems n NI NF NI NF NI NF NI NF NI NF
ROSE 2 35 140 13 51 8 35 5 17 21 90
FROTH 2 9 26 9 26 7 22 9 30 8 47
BADSCB 2 3 30 3 30 3 30 3 30 3 30
BEALE 2 15 50 15 51 10 38 7 24 6 46
JENSAM 2 2 27 2 27 2 27 2 27 2 27
BARD 3 16 54 9 28 17 56 12 38 11 38
GAUSS 3 2 4 2 4 2 4 2 4 2 4
BOX 3 2 27 2 27 2 27 2 27 2 27
SING 4 20 60 34 105 26 104 25 90 28 98
WOOD 4 19 61 9 33 9 32 8 29 8 28
KOWOSB 4 21 65 26 94 11 32 8 24 11 33
BD 4 17 54 13 44 11 40 6 21 11 39
Contd...
QUASI-NEWTON UPDATING FORMULAS FOR UNCONSTRAINED OPTIMIZATION2363

OSB1 5 2 27 2 27 2 27 2 27 2 27
BIGGS 6 25 72 4 12 3 9 3 9 3 9
OSB2 11 3 31 3 31 3 31 3 31 3 31
WATSON 20 31 102 6 20 6 43 5 17 6 21
ROSEX 100 231 806 10 41 8 35 5 17 10 50
SINGX 400 64 209 31 116 40 143 29 105 37 135
PEN1 400 2 27 2 27 2 27 2 27 2 27
PEN2 200 2 5 2 5 2 5 2 5 2 5
VARDIM 100 2 27 2 27 2 27 2 27 2 27
TRIG 500 9 33 8 33 9 32 5 18 10 36
BV 500 2 4 2 4 2 4 2 4 2 4
IE 500 6 16 6 16 8 22 12 34 7 19
TRID 500 53 170 12 45 11 41 9 33 11 42
BAND 500 57 281 9 54 5 17 5 17 5 17
LIN 500 2 4 2 4 2 4 2 4 2 4
LIN1 500 3 7 3 7 3 7 3 7 3 7
LIN0 500 3 7 3 7 3 7 3 7 3 7
Total 658 2426 244 996 219 928 183 750 223 975

Depending on the numerical results in Table 1, the proposed method

improved the rate of (67-73) % of the total numbers of the iterations and
(60-70) % of the total numbers of the evaluation of the functions compared
with the BFGS-method. On the other hand, the numerical results proved
that the proposed method more effective compared to BBFGS-method.
The proposed method improved the rate of (9-25) % of the total numbers
of the iterations and (3-25) % of the total numbers of the evaluation of the
functions compared with the BBFGS-method.

Table 2
Relative efficiency of the proposed method compared to BFGS

BFGS with BFGS with BFGS with

BFGS
g =5 g = 12 g = (8 - a k ) / 2
NI 100 % 33.28 % 27.81 % 33.89 %

NF 100 % 38.25 % 30.91 % 40.18 %

2364 B. A. HASSAN AND M. A. KAHYA

Table 3
Relative efficiency of the proposed method compared to BBFGS

BFGS with BFGS with BFGS with

BBFGS
g =5 g = 12 g = (8 - a k ) / 2
NI 100 % 89.75 % 75.00 % 91.39 %

NF 100 % 93.17 % 75.30 % 97.89 %

Table 2 and Table 3 show the relative efficiency of the proposed

method over rival methods. It is very clear to see that the proposed method
is more efficient from others.
Depending on the results tables, the reader can observe the efficiency
of the extension of the quasi-Newton equation on the method performance
in reducing the numbers of iterations and functions evaluation.
Specifically, the sum of the iterations and functions evaluation of BFGS
for all recommended gamma values on all test problems is the smallest
among other approaches. Concerning the indicator of the iteration
numbers, we obtained 183 when used BFGS with gamma=12 while we
achieved 219, 223, 244 and 658 when used other approaches. Moreover,
BFGS with gamma=12 had achieved perfect results in a reduction of the
function evaluation numbers where it reached 750 while it achieved 928,
975, 996 and 2426 via other approaches.

5. Conclusion
In this paper, a new class of quasi-Newton updating formulas
was investigated to show the efficiency and effectiveness of updating
techniques over the quasi-Newton equation. The proposed approach was
presented as an extension to the quasi-Newton equation which is inspired
by the technique presented by Razieh et al. [1]. Numerical experiments
showed that the proposed approach is effective to solve test problems.
Insight into Table 1, we can be concluded that BFGS with recommended
values of gamma in this paper outweighed over BFGS and BBFGS. In
particular, the proposed method with gamma=12 was the best of all, it has
the ability to reduce the numbers of iterations and functions evaluation,
and it gives rise to the high numerical accuracy with more favorable in
the convergence. Based on the above talk, we can be recommended to
the proposed method as a successful approach for solving unconstrained
optimization problems.
QUASI-NEWTON UPDATING FORMULAS FOR UNCONSTRAINED OPTIMIZATION2365

References

[1] R. Dehghani, M. M. Hosseini, and N. Bidabadi, “The modified BFGS

method with new secant relation for unconstrained optimization
problems,” Comput. Methods Differ. Equations, vol. 7, no. 1, pp. 28–41,
2019.
[2] D. P. O’Leary, “A discrete Newton algorithm for minimizing a
function of many variables,” Math. Program., vol. 23, no. 1, pp. 20–33,
1982.
[3] A. H. Basim and K. K. Hussein, “A new class of BFGS updating
formula based on the new quasi-newton equation,” Indones. J. Electr.
Eng. Comput. Sci., vol. 3, pp. 945–953, 2019.
[4] G. Yuan, Z. Wei, and X. Lu, “Global convergence of BFGS and PRP
methods under a modified weak Wolfe--Powell line search,” Appl.
Math. Model., vol. 47, pp. 811–825, 2017.
[5] G. Yuan, Z. Sheng, B. Wang, W. Hu, and C. Li, “The global convergence
of a modified BFGS method for nonconvex functions,” J. Comput.
Appl. Math., vol. 327, pp. 274–294, 2018.
[6] Y.-H. Dai, “Convergence properties of the BFGS algoritm,” SIAM J.
Optim., vol. 13, no. 3, pp. 693–701, 2002.
[7] Z. Wei, G. Li, and L. Qi, “New quasi-Newton methods for
unconstrained optimization problems,” Appl. Math. Comput., vol. 175,
no. 2, pp. 1156–1188, 2006.
[8] F. Biglari, M. A. Hassan, and W. J. Leong, “New quasi-Newton
methods via higher order tensor models,” J. Comput. Appl. Math., vol.
235, no. 8, pp. 2412–2422, 2011.
[9] L. H. Chen, N. Y. Deng, and J. Z. Zhang, “A modified quasi-Newton
method for structured optimization with partial information on the
Hessian,” Comput. Optim. Appl., vol. 35, no. 1, pp. 5–18, 2006.
[10] B. A. Hassan, “A new type of quasi-Newton updating formulas
based on the new quasi-Newton equation,” Numer. Algebr. Control \&
Optim., vol. 10, no. 2, p. 227, 2020.
[11] G. Yuan and Z. Wei, “Convergence analysis of a modified BFGS
method on convex minimizations,” Comput. Optim. Appl., vol. 47, no.
2, pp. 237–255, 2010.
2366 B. A. HASSAN AND M. A. KAHYA

[12] G. Yuan, Z. Wei, and Y. Wu, “Modified limited memory BFGS method
with nonmonotone line search for unconstrained optimization,” J.
Korean Math. Soc., vol. 47, no. 4, pp. 767–788, 2010.
[13] Z. Wan, K. L. Teo, X. Shen, and C. Hu, “New BFGS method for
unconstrained optimization problem based on modified Armijo line
search,” Optimization, vol. 63, no. 2, pp. 285–304, 2014.
[14] P. Mtagulwa and P. Kaelo, “A convergent modified HS-DY hybrid
conjugate gradient method for unconstrained optimization
problems,” J. Inf. Optim. Sci., vol. 40, no. 1, pp. 97–113, 2019.
[15] B. A. Hassan and M. W. Taha, “A new variants of quasi-newton
equation based on the quadratic function for unconstrained
optimization,” Indones. J. Electr. Eng. Comput. Sci., vol. 19, no. 2, pp.
701–708, 2020.
[16] X. Fang, Q. Ni, and M. Zeng, “A modified quasi-Newton method for
nonlinear equations,” J. Comput. Appl. Math., vol. 328, pp. 44–58, 2018.
[17] Y. H. Xiao, Z. X. Wei, and L. Zhang, “A modified BFGS method
without line searches for nonconvex unconstrained optimization,”
Adv. Theor. Appl. Math, vol. 1, no. 2, pp. 149–162, 2006.
[18] W. Findeisen, J. Szymanowski, and A. Wierzbicki, “Theory and
Methods of Optimization,” Warsaw Polish Sci. Publ., 1977.
[19] J. J. Moré, B. S. Garbow, and K. E. Hillstrom, “Testing unconstrained
optimization software,” ACM Trans. Math. Softw., vol. 7, no. 1, pp.
17–41, 1981.

Received December, 2020

Revised May, 2021