Anderson acceleration with adaptive relaxation for convergent fixed-point iterations
Abstract
Two adaptive relaxation strategies are proposed for Anderson acceleration. They are specifically designed for applications in which mappings converge to a fixed point. Their superiority over alternative Anderson acceleration is demonstrated for linear contraction mappings. Both strategies perform well in three nonlinear fixed-point applications that include partial differential equations and the EM algorithm. One strategy surpasses all other Anderson acceleration implementations tested in terms of computation time across various specifications, including composite Anderson acceleration.
Keywords: Anderson acceleration, Fixed-point iteration, Optimal relaxation
1 Introduction
Anderson acceleration (AA), also referred to as Anderson mixing, was formulated by Donald Anderson in 1965 [2] to solve integration problems numerically. Since then, its use has expanded to a wide array of problems in physics, mathematics and other disciplines. It is very close to methods developed in other contexts such as Pulay mixing (direct inversion in the iterative subspace) in chemistry [18] or the generalized minimal residual method (GMRES) and its predecessors like the generalized conjugate residual method (GCR) in the linear case (see [20], [23] and [16] for comparisons). There are two versions of the methods, related to type I and type II Broyden methods. This paper focuses on the latter.
Consider the mapping with at least one fixed point: . In what follows, can be interpreted as a residual of , is the EuclideanΒ Β norm of a vector , and is the inner product of the vectors and .
The AA algorithm can be described as follows.
Algorithm 1.
Input: a mapping , a starting point , and an integer
Β
1 | Set |
---|---|
2 | for until convergence |
3 | Β Β ββCompute |
4 | Β Β ββCompute that solve |
Β Β βββ s.t. | |
5 | Β Β ββCompute |
6 | Β Β ββSet |
7 | end for |
Β
The number of lags is usually set as high as possible to accelerate convergence while providing an acceptable conditioning of the linear problem for numerical stability.
The scalar is the relaxation (damping, mixing) parameter. In most AA applications, it is usually stationary and is often set to 1. As discussed by Anderson in [3], the choice of is natural under the implicit assumption that converges to a fixed point . In this case, should be closer to than and, correspondingly, should be closer to than , provided that is a good approximation of . This assumption is of course not always valid. Smaller have sometimes been shown to help stability and improve global convergence for challenging applications (see, for instance, [14] and [25]). But for fixed-point iterations which do converge to a fixed point, this property can be exploited to compute adaptive relaxation parameters which may significantly speed up the convergence of AA.
The idea of AA withΒ dynamically adjusted relaxation is a relatively recent one. In 2019, Anderson [3] outlined an algorithm for adjusting by monitoring the convergence of the Picard iteration for over multiple iterations. Focusing on linearly-converging fixed-point methods, Evans et al. [9] used the optimization gain from an AA step to set . They show that this choice of relaxation parameter improved robustness and efficiency over constant relaxation with and in various applications. Recently, Jin et al. [12] incorporated a dynamically adjusted in their Anderson acceleration of the derivative-free projection method.
In 2013, Potra and Engler [16] extended the results of [23] by characterizing the behavior of AA for linear systems with arbitrary non-zero relaxation parameters. While doing so, they also suggested a simple procedure to compute an optimal minimizing the residual of . Chen and Vuik [6] extended their method to nonlinear problems with a locally optimal that minimizes . Using the approximation , this optimal is
(1.1) |
and the optimal AA update is
(1.2) |
This scheme will be noted AAopt0. In various numerical applications, Chen and Vuik show that AAopt0 needs fewer iterations to converge compared to regular AA. Since AAopt0 needs two extra maps per iteration, and , they suggest using their method for applications for which mappings are inexpensive to compute. Otherwise, the potential efficiency gains from a smaller number of iterations may be offset by the extra computation per iteration.
1.1 Locally optimal relaxation for convergent mapping applications
Two modifications to AAopt0 are now proposed to improve its performance for convergent mapping iterations. First, note that (1.1) and (1.2) constitute by themselves a mini order-1 AA iteration. Using and to compute means implicitly choosing full relaxation () for this mini AA iteration. But for convergent mappings, we can hope that and are closer to than and , respectively. To take full advantage of the work of computing them, a more natural choice is thus the non-relaxed mini AA
(1.3) |
This modified scheme will be labeled AAopt1. Section 2.1 presents a proof that AAopt1 improves convergence compared to AAopt0 for contractive linear mappings.
Second, the empirical tests shown below suggest that the optimal relaxation parameters tend to be correlated between iterations. This opens up the possibility of the same as approximation for , as long as the lower precision due to the approximation does not deteriorate convergence too much. An AAopt1 which only updates the relaxation every iterations will be noted AAopt1_T, with AAopt1_1 AAopt1. Of course, the same modification could be applied to AAopt0. Details of the algorithm are presented in Section 2.3.
1.2 Costless adaptive relaxation for convergent mappings
A new adaptive relaxation parameter for AA is now proposed. It requires only two inner products and no extra maps per iteration.
Assume that has been computed for some . Since the mapping is assumed to converge to , the next map used in the AA algorithm, , should provide useful information on the location of . An improved choice of relaxation parameter in iteration would be the that minimizes the distance between and :
(1.4) |
Section 3.1 includes a demonstration that can improve convergence for a contractive linear operators under an appropriate norm.
In practice, since is needed to compute , using it at iteration to recompute an improved implies computing an extra map. However, as stated before, if is correlated with , it can be used as an approximation for . Computing would then require only two inner products, a negligible cost within the AA algorithm.
1.3 Regularizing
Essentially all the AA literature assumes . One exception could in some way be SqS1-SQUAREMΒ [22], as discussed in [21]. It corresponds to a 1-iteration AA where is also used as the relaxation parameter. Varadhan and Roland show that this choice can greatly speed up convergence. The same was pointed out by Raydan and Svaiter [19] for a linear version of the algorithm. Since the resulting values for depend on the eigenvalues of the Jacobian of , these can take values well above one.
A striking result from the numerical simulations presented in Section 5.4 is that for both suggested schemes, and especially for AAmd, the calculated relaxation parameters are regularly above one. When this happens, Chen and Vuik [6] recommend falling back to a default relaxation parameter below 1. For AAopt1_T and AAmd, empirical tests suggest that relaxation parameters above 1 can provide good results, as long as excessively high values are avoided. To do so, a practical option is simply to cap at some maximum value .Β For all the numerical tests presented below, was used.
1.4 Testing framework
The performances of AAopt1_T and AAmd will be tested in three fixed-point application: a Poisson partial differential equation (PDE) and two expectation-maximization (EM) algorithm [7]. Since AAopt1_T and AAmd vary in terms of mappings per iteration, the number of iterations or maps until convergence are not the most informative criteria for comparison. For practitioners and software library designers, the important benchmark is ultimately computation speed. In practice however, comparing computation times meaningfully is challenging since they can be affected by software implementation, hardware choices, and programming languages. To get the most complete picture, iterations, mappings, and computation times will be shown. To make the speed comparisons as informative as possible, all problems were programmed as efficiently as possible in a compiled programming language and executed on the same hardware.
Another matter of concern when comparing multiple AA implementations is the choice of parameters, especially the number of lags . With a sufficiently large , AA with constant relaxation can often converge in a similar number of iterations as AAopt0 or AAopt1. Of course, a larger entails more work solving the linear optimization problem and potentially less precision due to worse conditioning. To properly compare the various versions of AA, each of them should be implemented with an optimal for each problem, i.e., the one which minimizes computation times.
Finally, initial starting points can play a major and unpredictable role in convergence rates, especially for non-linear applications. To make the comparisons more robust, numerical experiments with many different starting points and different synthetic data will be performed.
2 AAopt1_T
This section demonstrates the convergence improvement of AAopt1 compared to AAopt0 for a contractive linear operator. This advantage is then illustrated by solving numerically a linear system of equations. Finally, the full AAopt1_T algorithm is laid out in detail.
2.1 Local improvements from AAopt1 for a linear contraction mapping
Consider solving where , and is symmetric with eigenvalues for all . To solve this system of equations, define the linear mapping .
At iteration of the AA algorithm, assume that and (as in step 5 of Algorithm 1) have been computed.
Theorem 2.1.
Proof.
It is straightforward to rewrite in terms of :
Expressing in terms of :
The squared 2-norms of and are
and
Since has eigenvalues for , the product is symmetric with eigenvalues for . The ratio
is a Rayleigh quotient that can only take values in . Therefore, , which completes the proof. β
2.2 A linear example with AAopt1
To apply the proof in a concrete example, consider using the mapping to find the solution to where diag,and the initial guess is . To highlight the difference between AAopt0 and AAopt1, no bounds on is imposed on either of them. AA with constant is also shown for comparison. All are implemented with .
Figure 2 shows the Euclidean norm of the residual for each algorithm and Figure 2 shows the relaxation parameter at each iteration. AAopt1 converges almost 3 times faster than AAopt0 and AA with . This small number of iterations for AAopt1βs looks promising. But since each iteration requires three maps per iteration instead of one for stationary AA, comparing computation speeds will be a true test of its promises as an optimization algorithm.
2.3 The full AAopt1_T algorithm
The full AAopt1_T algorithm takes as extra parameters , which determines at which interval the optimal relaxation is recomputed, , and as described in 1.3.
Algorithm 2.
Input: a mapping , a starting point , , and
Β
1 | Set |
2 | for until convergence |
3 | Β Β ββCompute |
4 | Β Β ββCompute that solve |
Β Β βββ s.t. |
5 | Β Β ββCompute |
6 | Β Β ββIf or |
7 | Β Β ββββCompute and |
8 | Β Β ββββCompute |
9 | Β Β ββββIf |
10 | Β Β ββββββSet |
11 | Β Β ββββelse |
12 | Β Β ββββββSet |
13 | Β Β ββββend if |
14 | Β Β ββββSet |
15 | Β Β ββelse |
16 | Β Β ββββSet |
17 | Β Β ββββSet |
18 | Β Β ββend if |
19 | end for |
Β
Note that possible adjustments to such as restarts and composite AA will be described in Section 4.2. Also, , in all experiments.
3 AAmd
This section justifies the use of as a relaxation parameter and shows its properties in the same linear example as before. Then, it details the implementation of the AAmd algorithm.
3.1 Local improvements from AAmd in a linear contraction mapping
Consider the same mapping as the one for Proof 2.1. Contrary to AAopt1, AAmd may improve convergence at each iteration in the familiar Euclidean norm. Nevertheless, we will show that it does in the elliptic norm
induced by the inner product
Assume that at iteration of AA, and (as in step 5 of Algorithm 1) have been computed. To lighten the notation, define the distance . For a given , the next iteration of AA is and the map of is . Using this information, we can compute the improved relaxation parameter as defined in (1.4). Using , we can compute an improved next iterate . The following theorem proves that would have been at least as good a choice as or better for any , in the sense that for all .
Theorem 3.1.
Let with and symmetric with eigenvalues for . For any
where , , and .
Proof.
Substituting in the definition of :
Note that β i.e., is orthogonal to β implies . In other words, is not an improvement over .
The next iterate is
where is a projection matrix. Computing and expressing it as function of :
If (the AA algorithm has converged), and the theorem trivially verified. If not, we may compute and :
Similarly,
By the properties of projection operators and matrix multiplications, the minimum eigenvalue of is zero. Label and are the smallest and largest eigenvalue of , respectively. Since has no negative eigenvalues, the minimum eigenvalue of for any is and its maximum eigenvalue is . Therefore, the maximum eigenvalue of must be .
The ratio
is a Rayleigh quotient. By the property of Rayleigh quotients, its maximum value is the maximum eigenvalue of . Therefore , which completes the proof. β
As mentioned in introduction, with already computed, using at step instead of involves computing , a second map in the same iteration . However, if the optimal relaxation parameters tend to be correlated, using in step only adds two inner products, a negligible computation cost within the AA algorithm.
3.2 A linear example with AAmd
Consider the same example as in Section 2.2. Four different AA implementations will be compared to study the impact of the initial choice of on and the impact of using as approximation for . The first implementation is a stationary relaxation parameter for reference. The second is AA where, at each iteration, a default relaxation parameter is used to compute and and and are used to compute to obtain . This second specification will be labeled βAAmd, (from 1)β. The third implementation is AA where, at each iteration, the previous relaxation parameter is used to compute and , which are used to recompute to obtain . This third specification will be labeled βAAmd, (from )β. The fourth specification is AA where, at each iteration, and are used to compute , and is used directly at iteration to compute . It will be labeled βAAmd, β. Again, is used in all algorithms.
Figure 4 shows the -norm of the residual for each algorithm. AAmd, (from ) and AAmd, show very modest convergence improvements. Note that with , all algorithms would essentially converge in the same number of iterations.
More interestingly, Figure 4 shows the relaxation parameter for each AA implementations. Interestingly, most are above 1, and very often above 2. As argued before, they are also obviously correlated from one iteration to the next. Hence, it makes sense to avoid computing an extra map by using past information to compute . Finally, the default relaxation parameter used to calculate matters, as attested by the relatively poor performance of AAmd, (from ) compared to AAmd, (from ) and AAmd, .
3.3 Extra regularization for AAmd
From now on, we only consider AAmd, β where is used as an approximation for to compute . There is no guarantee that , but a heuristic way of detecting whether it may be the case is by verifying how much varies between iterations. If it varies too much, a safer strategy is to fall back on the default relaxation parameter.
Another concern is the fact that a high can lead to an even higher , , etc. In many numerical experiments shown in Section 5.4, is set to a default relaxation parameter (always 1), but sometimes diverge far away from the unit interval, causing worse convergence than with the default . A very effective solution to this problem is to reset to (or some other default value between 0 and 1) for one iteration if has been above 1 for too many consecutive iterations.
3.4 The full AAmd algorithm
The entire AAmd algorithm is as follows. It takes as input two additional parameters: , the discrepancy allowed between and and , the maximum number of consecutive iterations where is allowed to be greater than 1 before is reset to .
Algorithm 3.
Input: a mapping , a starting point , , ,and .
Β
1 | Set |
2 | Set |
3 | for until convergence |
4 | Β Β ββCompute |
5 | Β Β ββCompute that solve |
Β Β βββ s.t. |
6 | Β Β ββCompute |
7 | Β Β ββIf : Compute |
8 | Β Β ββIf and and and |
9 | Β Β ββββSet |
10 | Β Β ββelse |
11 | Β Β ββββSet |
12 | Β Β ββend if |
13 | Β Β ββIf |
14 | Β Β ββββSet |
---|---|
15 | Β Β ββelse |
16 | Β Β ββββSet |
17 | Β Β ββend if |
18 | Β Β ββSet |
19 | end for |
Β
The parameters chosen for the experiments below are , , .
4 Implementation details for Anderson acceleration
4.1 Solving the linear system
In addition to computing , AA can spend a substantial amount of time solving the linear system. The problem is customarily formulated as an unconstrained optimization
where , and . As suggested in [23], it can be solved more quickly by QR decomposition. New columns can be added from the right of the Q and R matrices at each iteration and efficiently dropped from the left using Givens rotation. In the AA implementation used in the numerical section, the QR decomposition is recomputed anew after 10 rotations to limit the accumulation of numerical inaccuracies.
A central concern of all Anderson-type acceleration methods is the conditioning of the linear system. As the algorithm converges, new columns can be orders of magnitude smaller than old ones and can sometimes be close to linearly dependent. As in [23], this will be addressed by dropping left-most columns of until the R matrix has a reasonable condition number. In the PDE estimation, the upper limit for the condition number was set to while for the EM algorithm application, it was .
4.2 Restarts and composite AA
To limit ill-conditioning and the size of the linear system to be solved, Fang and Saad [10] suggested restarting the algorithm from the last iterate and ignoring past directions. Pratapa and Suryanarayana [17] and Henderson and Varadhan [11] made similar points in the context of Pulay mixing and AA.
A cousin of this idea is the composite AA, explored by Chen and Vuik in 2022 [5]. After one AA iteration, instead of using the iterate directly to compute the map , they propose using as the starting point for a second AA (lasting only one or two iterations), and feeding the result of this second (inner-loop) AA back in the original (outer-loop) AA.
Both ideas were tried on all problems. Whilst periodic restarts did not reliably improve performances, composite AAmd with a one-iteration AA for the inner loop showed very good results for the PDE problem. Hence, composite AA with a one-iteration AA with in the inner loop will be included in the set of specifications to test.
5 Applications
A Poisson PDE applications and two EM algorithm applications were used as benchmarks. They are sufficiently challenging to estimate and require enough iterations to create visible differences in performance between different AA implementations. They also offer a good variety of number of parameters and mapping computation time.
The EM algorithm is commonly used to fit statistical models with missing data or latent variables to estimate the parameters of underlying unobserved distributions. It consists of two steps. An expectation step takes the model parameters as given and updates the parameters of the unobserved data via Bayesβ rule. Then, the likelihood of the observed data is maximized, taking the unobserved distributions as given. The EM algorithm is usually very stable and always converges, although it can sometimes be to a saddle point instead of a maximum. It is also notoriously slow, making it a prime candidate for acceleration. Both EM application were adapted from the R code used in [11].
5.1 The Bratu problem
The standard Liouville-Bratu-Gelfand equation is a nonlinear version of the Poisson equation, described as
where is a function and is a constant physical parameter. It is a popular application for benchmarking new fixed-point acceleration methods (see [23] and [10], for example). Dirichlet boundary conditions are applied such that on the boundary of . It is solved using the inverse of the discrete Laplace operator as preconditioner as in [6]. The mapping is
where is the Laplace operator, is its row and column βs entry, and is the entire row . In the experiments, was set to and a centered-difference discretization on a grid was used.
5.2 The EM algorithm for a proportional hazard model with interval censoring
Proportional hazard models are commonly used in medical and social studies, and censored data is a frequent occurrence which complicates their estimation. Wang et al. [24] proposed using the EM algorithm to estimate a semiparametric proportional hazard model with interval censoring. Their estimation is a two-stage data augmentation with latent Poisson random variables and a monotone spline to represent the baseline hazard function. The algorithm is light and simple to implement (see [24] for details), yet may benefit from acceleration.
The likelihood of an individual observation is
where , , and represent right-, interval-, or left-censoring, respectively.
Synthetic test data is produced as follows. The baseline hazard function is modeled as a six-parameter I-spline and generated as . Failure times are generated from a distribution . The covariates are and Bernoulli, for a total of 10 parameters to estimate. For each subject, censoring is simulated by generating a Exponential distribution and setting if or if . The sample size was 2000 individuals.
During the estimations, monitoring the value of the likelihood was not necessary for convergence.
5.3 EM algorithm for admixed populations
When associating health outcomes with specific genes, a recurring confounding factor is population stratification, the clustering of genes within population subgroups. In [1], Alexander, Novembre and Lange put forward a new algorithm called ADMIXTURE to identify latent subpopulations from genomics data using the EM algorithm.
To test the algorithm, datasets are simulated as follows. Individual are assumed to be part of distinct ancestral groups in various proportions. Each has a pair of alleles () at the marker with major or minor frequencies recorded in a variable . For individual and marker , if both minor alleles are minor, if one is minor and one is major, and if both are major. The probability of observing is determined by ancestry-specific parameters , the frequency of minor alleles at the marker in the ancestral population , and the parameter representing the unobserved proportion of ancestry of individual from group . The log-likelihood function is
(5.1) |
where and are matrices with entries and ,respectively. To restrict probabilities to the unit interval during the estimation, they are modeled as transformations from unbounded parameters : and . New random values for and new starting values are generated for each draw with parameters , , and , for a total of parameters to estimate.
For AA to converge, it was necessary to monitor the likelihood value (5.1) and fall back to the last EM iteration in case an AA iteration lead to a worse likelihood value. Also, since convergence was slow, the stopping criterion was set to .
5.4 Example Bratu problem
5.4.1 Non-composite AA
This section compares AAopt1 and AAmd to AAopt0 and AA with stationary relaxation parameters of and for the Bratu problem with a starting point. Hereafter, AAmd refers to Algorithm 3 with capped at and resets to . To show the impact of these regularizations, the examples also show the performance of AAmd without bounds on or resets, labeled βAAmd (no reg.)β. All algorithms used a maximum of lags. AAopt0 was implemented as in [6], with set to whenever the optimal relaxation parameter was zero or fell outside the unit interval.
Figure 6 shows the residuals for the Bratu problem for non-composite versions of AAopt0, AAopt1, AAmd, and AAmd (no reg.). Like in the linear model of Section 2.2, AAopt1 needed the fewest iterations to converge. However, AAmdβs results are promising knowing that requires a single mapping per iteration.
Figure 6 shows the corresponding relaxation parameters. Looking at AAmd (no reg.), it is striking how without constraints or restarts, takes extremely large values and converges as slowly as AA with stationary relaxation. With regularization, AAmd performs much better than AA with constant relaxation.
5.4.2 Composite AA
The Bratu problem is estimated with the same specifications as in Section 5.4.1, except that AA is now composite with an AA1 inner loop (identified with a c, see Section 4.2).
Figures 6 show the residuals for the Bratu problem with composite versions of AA and Figure 6 shows the corresponding relaxation parameters. The horizontal axis refers to the number of outer loop iterations. AAopt1, c converges in the fewest iterations, although the difference with other AA algorithms is relatively small, especially considering the fact that it requires 6 maps per iteration. AA implementations that require only 3 mappings per iteration such as AAmd, c and AA, c with constant also converged reasonably quickly.
5.5 Experiments
Based on the previous example, the most promising AA implementations to investigate further are AAopt1, AAmd and AAmd, c. Additionally, AAopt1_4 and AAopt1_16, which update every 4 or every 16 iterations, are also studied. These implementations are again compared with stationary AA with and , as well as their composite versions.
To select the optimal for each AA algorithm in each experiment, each algorithm was implemented with for 500 draws. The fastest in terms of computation speed (at the 0.75 quantile to favor robustness) was selected. For the EM algorithm for the proportional hazard model with interval censoring, the maximum was clearly the best choice for all algorithms.
A total of 5000 draws were generated for the Bratu problem and the EM for the proportional hazard model with interval censoring. To reduce simulation time, only 1000 draws were calculated for the EM algorithm for admixed populations. The stopping criterion was for all applications, except the EM algorithm for admixed populations which used . All algorithms that did not converge were stopped after 10 000 mappings.
Computation times are presented using the performance profiles of Dolan and MorΓ© [8]. They show at which frequency each algorithmβs time was within a certain factor of the fastest algorithm for each draw. The 99%Β confidence intervals for the median number of iterations (outer-loop iterations for composite AA), mappings, and computation times are also reported in tables, along with convergence rates. All computations were single-threaded, performed on Julia 1.10.4 [4], with a 13th Gen Intel(R) Core(TM) i9-13900HX 2.20 GHz CPU running the Windows subsystem for Linux.
For the Bratu problem, the starting values were drawn from U(0,1) distributions. The results are reported in Figure 9 and Table 1. The composite AAmd was the fastest more than 60% of the time, requiring a median of 67 iterations and 199 maps to converge. The runner-up was the composite AA with constant . Interestingly, non-composite AAmd was also faster than both non-composite AA with constant relaxation.
Algorithm | Iterations | Β Β Β Maps | Β Β Time (ms) | Converged |
AAopt1, | (89, 89) | (263, 263) | (148.22, 149.19) | 1 |
AAopt1_4, | (161, 162) | (241, 242) | (136.73, 137.67) | 1 |
AAopt1_16, | (203, 204) | (229, 230) | (134.28, 135.47) | 1 |
AAmd, | (218, 219) | (218, 219) | (152.02, 153.4) | 1 |
AAmd, c, | (67, 67) | (199, 199) | (112.12, 112.82) | 1 |
AA, | (223, 224) | (223, 224) | (212.83, 214.67) | 1 |
AA, | (273, 278) | (273, 278) | (194.53, 197.93) | 1 |
AA, , c | (87, 87) | (259, 259) | (157.77, 158.96) | 1 |
AA, , c | (70, 71) | (208, 211) | (116.79, 117.9) | 1 |
Note: 99% conf. interval for the median. 2500 parameters. |
The results of the numerical experiments with the EM algorithm for a proportional hazard model with interval censoring are shown in Figure 11 and Table 2. AAmd was the fastest to converge, with a median number of iterations and mapping evaluations of approximately 99. AAopt1_16 was a close second. Contrary to the Bratu problem, composite AA clearly did not benefit from the AA1 inner loop compared to non-composite AA. Still among composite AA, AAmd did slightly outperformed those with constant relaxation.
Algorithm | Iterations | Maps | Time (ms) | Converged |
---|---|---|---|---|
AAopt1, | (87, 93) | (257, 275) | (71.59, 76.41) | 0.962 |
AAopt1_4, | (102, 110) | (152, 164) | (43.71, 47.56) | 0.894 |
AAopt1_16, | (99, 107) | (113, 121) | (32.8, 35.14) | 0.96 |
AAmd, | (96, 102) | (96, 102) | (28.43, 30.28) | 0.958 |
AAmd, c, | (68, 72) | (202, 214) | (56.84, 60.03) | 0.97 |
AA, , | (129, 139) | (129, 139) | (37.98, 40.41) | 0.952 |
AA, , | (169, 184) | (169, 184) | (48.79, 52.8) | 0.937 |
AA, , c, | (75, 79) | (223, 235) | (61.79, 65.68) | 0.962 |
AA, , c, | (76, 80) | (226, 239) | (63.13, 67.07) | 0.964 |
Note: 99% conf. interval for the median. 10 parameters. |
Algorithm | Iterations | Maps | Time (s) | Converged |
AAopt1, | (245, 253) | (731, 755) | (4.14, 4.28) | 1 |
AAopt1_4, | (355, 374) | (533, 560) | (3.4, 3.68) | 1 |
AAopt1_16, | (434, 454) | (488, 512) | (3.38, 3.55) | 1 |
AAmd, | (363, 379) | (363, 379) | (2.55, 2.65) | 1 |
AAmd, c, | (161, 166) | (481, 496) | (2.76, 2.86) | 0.999 |
AA, , | (377, 389) | (377, 389) | (2.67, 2.77) | 1 |
AA, , | (449, 461) | (449, 461) | (3.21, 3.35) | 1 |
AA, , c, | (182, 189) | (544, 565) | (3.16, 3.3) | 1 |
AA, , c, | (207, 214) | (619, 640) | (3.57, 3.71) | 1 |
Note: 99% conf. interval for the median. 750 parameters. Tolerance set to . 1000 draws. |
Finally, Figure 11 and Table 3 summarize the results of the numerical experiments with the EM algorithm for admixed populations. AAmd was again the fastest, although its edge over stationary AA with was small. Composite AAmd was in third place with a clear edge over both composite AA with constant relaxation.
5.6 Discussion
The results shown in Sections 5.4 and 5.5 show that adaptive relaxation parameters can clearly reduce the number of iterations needed for AA to converge. Since maps are often computationally expensive, a relaxation strategy like AAmd which do not require additional maps per iteration offers the most predictable benefits. AA with a well-chosen constant relaxation parameters can approach its performances, but the results can vary greatly with the choice of . A clear advantage of adaptive relaxation is that they do not require tuning.
Additional numerical experiments (not presented here) were conducted to explore the impact of each parameter on AAmd. Setting higher than 3 generally made little difference. A smaller (the allowed discrepancy between and ) made AAmd slightly slower by falling back on more frequently. Conversly, a larger may not be advisable for highly nonlinear applications for which optimal relaxation parameters could vary considerably between iterations. A key parameter is , the number of consecutive values above before a restart with . A reasonable range is . Without periodic resets of , AAmdβs convergence was often slower than that of stationary AA, as seen in Section 5.4.
Composite AA was successful in accelerating the estimation of the Bratu problem but performed poorly for the EM algorithm. Among non-composite AA implementations, AAopt1 often converged in fewer iterations, though it was not competitive in terms of computation time. By not recomputing at each iteration, AAopt1_4 and AAopt1_16 were often faster than AA with constant mappings with or , but did not match the speed of AAmd.
Since AA can be made efficient by using QR decomposition and Givens rotations to solve the internal linear optimization problem, the fastest algorithms were clearly those requiring the the fewest mappings to converge. This could change for applications with truly negligible mapping computation times. However, for such applications, lighter algorithms that do not require solving any linear systems, such as SQUAREM [22] or ACX [13], would likely be faster than AA.
6 Conclusion
Two adaptive relaxation schemes for Anderson acceleration have been proposed for convergent fixed-point applications. Both are demonstrated to improve Anderson accelerationβs convergence for a linear contraction mapping.
The first scheme, AAopt1, uses two extra maps to compute a locally optimal relaxation parameter. Convergence is accelerated by reusing the same maps a second time to compute the next iterate. Furthermore, by reusing the same relaxation parameter over multiple iterations, AAopt1_T always outperforms AAopt1 in terms of speed, though AA with a constant, well-chosen relaxation parameter can still be faster.
The second proposed scheme, AAmd, requires fewer iterations to converge while needing minimal extra calculation. As a result, it outperformed all other AA specifications across all tests in terms of computation time. Interestingly, AAmdβs adaptive relaxation parameters are frequently above one, an unexpected result in the context of the AA literature that warrants further investigation.
References
- Alexander et al., [2009] David H. Alexander, John Novembre, and Kenneth Lange. Fast model-based estimation of ancestry in unrelated individuals. Genome Research, 19:1655β1664, July 2009.
- Anderson, [1965] Donald G. M. Anderson. Iterative procedures for nonlinear integral equations. Journal of the Association for Computing Machinery, 12(4):547β560, October 1965.
- Anderson, [2019] Donald G. M. Anderson. Comments on βAnderson acceleration, mixing and extrapolationβ. Numerical Algorithms, 80:135β234, 2019.
- Bezanson et al., [2017] Jeff Bezanson, Alan Edelman, Stefan Karpinski, and ViralΒ B. Shah. Julia: A fresh approach to numerical computing. SIAM Review, 59(1):65β98, 2017.
- Chen and Vuik, [2022] Kewang Chen and Cornelis Vuik. Composite Anderson acceleration method with two window sizes and optimized damping. International Journal for Numerical Methods in Engineering, 123(23):5964β5985, August 2022.
- Chen and Vuik, [2024] Kewang Chen and Cornelis Vuik. Non-stationary Anderson acceleration with optimized damping. Journal of Computational and Applied Mathematics, 451, June 2024.
- Dempster et al., [1977] ArthurΒ P. Dempster, NanΒ N. Laird, and DonaldΒ B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B (Statistical Methodology), 39(1):1β38, 1977.
- Dolan and MorΓ©, [2002] ElizabethΒ D. Dolan and JorgeΒ J. MorΓ©. Benchmarking optimization software with performance profiles. Mathematical Programming, 91:201β213, 2002.
- Evans et al., [2020] Claire Evans, Sara Pollock, Leo G. Rebholz, and Mengying Xiao. A proof that Anderson acceleration improves the convergence rate in linearly converging fixed-point methods (but not in those converging quadratically). SIAM Journal on Numerical Analysis, 58(1):788β810, 2020.
- Fang and Saad, [2009] Haw-ren Fang and Yousef Saad. Two classes of multisecant methods for nonlinear acceleration. Numerical Linear Algebra with Applications, 16:197β221, 2009.
- Henderson and Varadhan, [2019] Nicholas C. Henderson and Ravi Varadhan. Damped Anderson acceleration with restarts and monotonicity control for accelerating EM and EM-like algorithms. Journal of Computational and Graphical Statistics, May 2019.
- Jin et al., [2024] Jiachen Jin, Hongxia Wang, and Kangkang Deng. Anderson acceleration of derivative-free projection methods for constrained monotone nonlinear equations, 2024.
- Lepage-Saucier, [2024] Nicolas Lepage-Saucier. Alternating cyclic vector extrapolation technique for accelerating nonlinear optimization algorithms and fixed-point mapping applications. Journal of Computational and Applied Mathematics, 439, March 2024.
- Pollock and Rebholz, [2021] Sara Pollock and Leo G. Rebholz. Anderson acceleration for contractive and noncontractive operators. IMA Journal of Numerical Analysis, 41:2841β2872, January 2021.
- Pollock and Rebholz, [2023] Sara Pollock and Leo G. Rebholz. Filtering for Anderson acceleration. SIAM Journal on Scientific Computing, 45(4):A1571βA1590, 2023.
- Potra and Engler, [2013] Florian A. Potra and Hans Engler. A characterization of the behavior of the Anderson acceleration on linear problems. Linear Algebra and its Applications, 438:1002β1011, November 2013.
- Pratapa and Suryanarayana, [2015] Phanisri P. Pratapa and Phanish Suryanarayana. Restarted Pulay mixing for efficient and robust acceleration of fixed-point iterations. Chemical Physics Letters, 635:69β74, 2015.
- Pulay, [1980] Peter Pulay. Convergence acceleration of iterative sequences. the case of SCF iteration. Chemical Physics Letters, 73(2):393β398, July 1980.
- Raydan and Svaiter, [2002] Marcos Raydan and Benar F. Svaiter. Relaxed steepest descent and Cauchy-Barzilai-Borwein method. Computational Optimization and Applications, 21:155β167, 2002.
- Saad and Schultz, [1986] Youcef Saad and Martin H. Schultz. GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on Scientific and Statistical Computing, 7(3):856β869, July 1986.
- Tang et al., [2023] Bohao Tang, Nicholas C. Henderson, and Ravi Varadhan. Accelerating fixed-point algorithms in statistics and datascience: A state-of-art review. Journal of Data Science, 21(1):1β26, July 2023.
- Varadhan and Roland, [2008] Ravi Varadhan and Christophe Roland. Simple and globally convergent methods for accelerating the convergence of any EM algorithm. Scandinavian Journal of Statistics, 35:335β353, 2008.
- Walker and Ni, [2011] Homer F. Walker and Peng Ni. Anderson acceleration for fixed-point iterations. SIAM Journal on Numerical Analysis, 49(4):1715β1735, 2011.
- Wang et al., [2016] Lianming Wang, Christopher S. McMahan, Michael G. Hudgens, and Zaina P. Qureshi. A flexible, computationally efficient method for fitting the proportional hazards model to interval-censored data. Biometrics, 72:222β231, March 2016.
- Warnock, [2021] Robert Warnock. Equilibrium of an arbitrary bunch train with cavity resonators and short range wake: Enhanced iterative solution with Anderson acceleration. Physical Review Accelerators and Beams, 2021.