allow setting both absolute and relative tolerance of sparse iterative solvers

[rc]

Currently a single tolerance (tol argument) is used, which is impractical, when using an iterative solver as a preconditioner. In that case it is desirable to set a relative tolerance to some not-so-small value (e.g. 1e-2), without caring about the absolute tolerance. The current behaviour prevents that, as something like setting tol=1e-2 often leads to absolute convergence in iteration zero of the preconditioner, hence no preconditioning done at all.

This behaviour breaks the IMHO very good lgmres solver. For example, the following happens when using lgmres with cg (tol=1e-2) as a preconditioner for symmetric positive definite diagonal blocks:

<snip>
sfepy: cg-s: iteration 1
sfepy: cg-s: iteration 2
sfepy: cg-s: iteration 3
sfepy: cg-s: iteration 4
sfepy: cg-s: iteration 5
sfepy: cg-s: iteration 6
sfepy: cg-s: cg convergence: 0 (successful exit)
rhs norm: 8.751133e-02, solution norm: 4.484927e+00
sfepy: iterative-s: iteration 4: |Ax-b| = 2.323006e-03
sfepy: cg-s: iteration 1
sfepy: cg-s: cg convergence: 0 (successful exit)
rhs norm: 4.101975e-04, solution norm: 0.000000e+00
sfepy: cg-s: iteration 1
sfepy: cg-s: cg convergence: 0 (successful exit)
rhs norm: 2.286502e-03, solution norm: 0.000000e+00
Traceback (most recent call last):
<snip>
  File "....local/lib/python2.7/site-packages/scipy/sparse/linalg/isolve/lgmres.py", line 145, in lgmres
    "|v| ~ %.1g, |M v| = 0" % rnorm)
RuntimeError: Preconditioner returned a zero vector; |v| ~ 0.002, |M v| = 0

This happened because the rhs norm was 2.286502e-03 < 1e-2 (see the line above the traceback), so no solution was attempted.

So I propose having both rtol and atol arguments (or allow tol to be a tuple), and possibly adding a tol_mode argument - either 'and' or 'or', defaulting to 'or', that tell whether both tolerances have to be met ('and' mode) or a single one is enough.

I can make a PR implementing this for lgmres, as it is written in Python. I am not sure if I am brave enough to dive into the legacy fortran of the other solvers.

[person142]

When a similar issue came up with differential_evolution we kept tol for relative tolerance and added an atol for absolute tolerance:

#6449 (comment)

Seems like doing the same thing here would be appropriate.

[pv]

This has been on the todo list forever, so it's good if you want to fix it finally... . For backward compat: I would add new arguments rtol and atol. The `tol` argument sets both to the same value, and an error is raised if the user gives both `tol` and `rtol` or `atol`. We might also want to issue a DeprecationWarning if the user sets the `tol` argument. . The second question is what to do with left-preconditioning. I'm not sure applying an absolute tolerance to preconditioned residuals is very sensible --- this is just a surprise waiting to happen. . Note that there's some refactoring of LGMRES here #5910, plus some minor fixes.

[pv]

(Btw, review of gh-5910 would be appreciated in case you're interested in the topic ;)

[rc]

Yes, leaving tol and adding rtol, atol was another option I considered :). I will give it a shot after I resolve my compilation issues with the scipy master.

[rc]

The fortran-based solvers seem all to use the relative-only test (contrary to what the docstrings say), as defined in STOPTEST2.f.src, which is equivalent to r_norm <= tol * b_norm.

The python-based solvers (lgmres, and gcrotmk) use both the relative and absolute test.

So the PR should

• modify STOPTEST2.f.src for RTOL, ATOL
• update iterative.py for rtol, atol
• update lgmres (and gcrotmk) for rtol, atol

Now the condition above measures |r_k| / |r_0| only for the zero initial guess. Does it make sense to use |b - A x0| instead of |b|? I guess no - I have just checked that petsc uses also just |b|, so it is what people probably expect, but what do you think?

I would also like to introduce the tol_mode argument (described above) - do you agree?

[pv]

The fortran solvers also do absolute termination checks, iirc, but this is elsewhere in the code than in STOPTEST2. The extra check may also in some cases be done only for the first step.

[rc]

Yes, I can see the absolute test for the first step, e.g. in CG. Thanks!

[rc]

Heretical thought: having the fortran solvers rewritten in Python, following the examples of lgmres and gcrotmk, would be much more maintainable.

[person142]

Heretical thought: having the fortran solvers rewritten in Python, following the examples of lgmres > and gcrotmk, would be much more maintainable.

Doesn't seem heretical to me (I've wished for the same thing in fact), though Cython might be even better. I'd guess that the issue here is just someone having time to do it.

[person142]

As a fun side-effect it would also make the solvers reentrant.

[rc]

Would Cython help here? IMHO the solvers spend most of the time for non-tiny problems in blas/matvec calls, that are compiled anyway. But I did not measure that :)

[person142]

Would Cython help here?

I doubt it would speed things up much. IMO the benefit would come from possibly exposing the Cython interface directly in addition to a Python interface.

[rc]

Well, there is [1] - a ready python implementation with MIT license and compatible function arguments.

[1] https://github.com/pyamg/pyamg/tree/master/pyamg/krylov