Request/Proposal: Greatly improve scipy.interpolate.Rbf

[Jwink3101]

Request (Issues with rbf)

The scipy.interpolate.Rbf has a lot of issues and can stand some real improvement.

First it is really an outlier from the rest of the scattered data interpolation methods for its call signature. It would be much better to take an (N,ndim) array of points and a (N,) array of values like the rest of them.

Second, it is technically missing some major features. The most glaring is the lack of a augmenting polynomial to be solved along side the RBF weights. If this is included, you also introduce the ability to make it a polyharmonic spline (using r**k * (log(r) if np.mod(k,2) else 1) (that's more pseudo-code than real code. In reality, you would use xlogy).

Implementing that isn't too hard though it would require a multi-dimensional Vandermond builder.

Proposal

I have working code based roughly on the existing that adds a lot of this. It allows for arbitrary polynomial degree and also can accept integer basis functions representing the splines. Of course, it also can use the positive definite kernels.

Mine also does some additional things like implement an analytical leave-one-out error estimator and a brute-force optimizer to find the ideal shape parameter. I think those are outside the scope for scipy.

So my proposal is to improve the RBF function. Since the call signature will change (for the better), I think it should have a new name and the old one should be deprecated.

However, I would need some help with working it into SciPy including adding appropriate tests.

Is there interest? Any volunteers?

[rgommers]

Sounds like a good idea to me - Rbf could definitely use some improvements. Your thoughts on what's in and out of scope for SciPy make sense to me.

Changing the signature is always tricky, so yes a new class can be considered. Would be good to have a rough outline here (signatures of __init__ and main methods). Note that in the end we'll need to run this by the scipy-dev mailing list for approval.

It would be good if you could browse through the open issues for Rbf as well. If we introduce a new class, some of those could be addressed. And there may be useful content (things to test, code to reuse). Most important ones seem to be:

• RBF fixup programme #4790, ideas for new features (partially done in RBF module fix-up, issue 4790 #5070, but that wasn't completed)
• Rbf interpolation class is not pickleable thus prevents multiprocessing (Trac #1784) #2303, make Rbf picklable
• Rbf interpolation - use only K nearest neighbors #5180, use K nearest neighbors only for interpolation

[Jwink3101]

Thanks for the info and tips. 4790 is interesting and I am working through if I think it is worth it. Scaling by the eigenvalue feels very implicit to me and you lose the connection of RBFs to Kriging surfaces (which doesn't exist for the splines but that is another story).

My code seems to be pickleable at the moment. Not having *args seems to go a long way

5180 feels out of scope. If you're making a nearest neighbor RBF, then that should be its own class since it is, at its heart, a different animal.

I will do more reading and browsing to better develop the idea but right now, its public method will basically be those of the current implementation. The biggest difference is how it would assume the input. And it would solve with polynomial basis augmentation as well. That requires being able to make a polynomial multi-index and a vandermond matrix. None are too hard.

Below is the actual code I have planned though it needs to be converted into SciPy's documentation format and probably some other cleanup. The references are also not all needed since I do not include some of the fancy features from them.

import sys
if sys.version_info[0] >2:
    unicode = str

import numpy as np
import scipy.spatial
from scipy.special import xlogy
import scipy.linalg


class RadialBasisInterpolation:
    """
    Augmented Radial Basis function interpolation with (optional) polynomial 
    augmentation. The construction follows [1] with code borrowed from [2] 
    for the kernel (which they call the "function")
    
    Inputs
    ------
    X (array like)
        (N,ndim) array or (N,) vector of build points
        
    f (array like)
        (N,) vector of function values
    
    Options: 
    --------
    degree [0]
        Use an augmented RBF. Specify the *total* degree fit if given as an 
        integer. If given as a matrix, must be (P+1,ndim) shape. This is the 
        degree of the augmenting polynomial.
    
    kernel : ['gaussian'] 
        The radial basis kernel, based on the radius, r
        If string:
            'multiquadric': sqrt((r/epsilon)**2 + 1)
            'inverse': 1.0/sqrt((r/epsilon)**2 + 1)
            'gaussian': exp(-(r/epsilon)**2)
            'linear': r
            'cubic': r**3
            'quintic': r**5
            'thin_plate': r**2 * log(r)
        If callable
            must take the arguments (r,epsilon) and ignore epsilon if 
            not needed. Tip: If epsilon is not needed, set it to
            0 to avoid trying to optimize it
        If integer:
            Spline of the form: r**kernel if kernel is odd else r**kernel*log(r)

    epsilon [None]
        Adjustable parameter for some kernels.
        
    smooth : float, optional
        Values greater than zero increase the smoothness of the
        approximation.  0 is for interpolation (default). Actually set to
        max(smooth,1e-10 * ||f||) for numerical precision

    Theory:
    ------
    The RBF is based on a radial distance 
    
        r_j = ||x - x_j||                                               (1)
    
    with the kernel phi(r) and an optional "Agumenting" polynomial of set 
    degree. The interpolant is of the form: [1]
    
        s(x) = sum_{j=1}^N c_j * phi(r_j) + sum_{k \in P} g_k * p_k     (2)
    
    where `P` is a polynomial index and `p_k` is the corresponding
    polynomial (e.g. k = {1,2,0} then p_k = x*y^2*z^0).
    
    In order to solve for c_j and g_k we enforce 
        (a) s(x_j) = f_j for all j (i.e. exact interpolation)
        (b) sum_{j=1}^N c_j*p_k(x_j) = 0 for all k=1,...,N 
    
    That results in a matrix system:
        ____________________________________________________   ___     ___
       |                            |                       | |   |   |   |
       |   phi(||x_i - x_j||) (N,N) |   p_k(x_j) (N,len(K)) | | c |   | f |
       |                            |                       | |   | = |   | (3)
       |----------------------------|-----------------------| |---|   |---|
       |       p_k(x_j).T           |           0           | | g |   | 0 |
        ----------------------------------------------------   ---     ---
      
    which is then solved. Given c and g, (2) can be used to solve for
    any given x
    
    References:
    ----------  
    [1] G. B. Wright. Radial Basis Function Interpolation: Numerical and 
        Analytical Developments. PhD thesis, University of Colorado, 2003.

    [2] SciPy version 1.1.0
        https://github.com/scipy/scipy/blob/v1.1.0/scipy/interpolate/rbf.py
    
    [3] S. Rippa. An algorithm for selecting a good value for the parameter 
        c in radial basis function interpolation. Advances in Computational 
        Mathematics, 11(2-3):193–210, 1999.
        
    [4] J. D. Martin and T. W. Simpson. Use of Kriging Models to Approximate 
        Deterministic Computer Models. AIAA journal, 43(4):853–863, 2005.

    """
    def __init__(self,X,f,degree=0,
                 epsilon=1,smooth=0,
                 kernel='gaussian',_solve=True):
        self.X = X = np.atleast_2d(X)
        self.N,self.ndim = X.shape
        self.f = f = np.ravel(f)
        self.degree = degree
        self.kernel = kernel
        self.epsilon = epsilon
        
        
        r = scipy.spatial.distance.cdist(X,X)

        self.smooth = max(float(smooth),1e-10*scipy.linalg.norm(f)/np.sqrt(len(f))) # Scale by ||f||
        
        K = self._kernel(r) + np.eye(self.N)*self.smooth
        P = RadialBasisInterpolation.vandermond(X,degree=self.degree)
        
        # Build the matrix
        Z = np.zeros([P.shape[1]]*2)
        z = np.zeros(P.shape[1])
        KP = np.block([[K   , P],
                       [P.T , Z]])
        b = np.hstack([f,z])
        coef = scipy.linalg.solve(KP,b)
        self.rbf_coef = coef[:self.N]
        self.poly_coef = coef[self.N:]
        

    def __call__(self,X):
        X = np.atleast_2d(X)
        K = self._kernel( scipy.spatial.distance.cdist(X,self.X) )
        P = RadialBasisInterpolation.vandermond(X,degree=self.degree)
        return K.dot(self.rbf_coef) + P.dot(self.poly_coef)

    def _kernel(self,r):
        r = np.asarray(r)
        if callable(self.kernel):
            return self.kernel(r,self.epsilon)
        elif isinstance(self.kernel,(int,np.integer)):
            if np.mod(self.kernel,2) == 0:
                return xlogy(r**self.kernel,r)
            else:
                return r**self.kernel
        elif not isinstance(self.kernel,(str,unicode)):
            raise ValueError('Kernel must be a callable with signature (r,epsilon), an integer (spline) or a valid string')
        
        kernel = self.kernel.lower().replace(' ','_').replace('-','_')
        
        if kernel in ['multiquadric']:
            return np.sqrt((r/self.epsilon)**2 + 1)
        elif kernel in ['inverse_multiquadric','inverse']:
            return 1.0/np.sqrt((1.0/self.epsilon*r)**2 + 1)
        elif kernel in ['gaussian','squared_exponential']:
            return np.exp(-(1.0/self.epsilon*r)**2)
        elif kernel in ['linear']:
            return r
        elif kernel in ['cubic']:
            return r**3
        elif kernel in ['quintic']:
            return r**5
        elif kernel in ['thin_plate']:
            return xlogy(r**2, r)
        else:
            raise ValueError('Not valid kernel name')
    
    @staticmethod
    def vandermond(X,degree):
        """
        Return a Vandermond matrix of X up to the
        *total* order degree
        """
        X = np.atleast_2d(X)
        ndim = X.shape[1]
        index = np.asarray(RadialBasisInterpolation.total_index(degree,ndim),dtype=int)
        V = np.ones([X.shape[0],len(index)])
        for d in range(ndim):
            v = np.fliplr(np.vander(X[:,d],N=degree+1))
            V *= v[:,index[:,d]]
        return V

    
    @staticmethod
    def total_index(P,ndim,sort=True):
        P = tuple([P]*ndim)
        curr = [0]*ndim
        ind = []
        ind.append(tuple(curr))
        while True:
            for d in range(ndim):
                if sum(curr) < P[0]:
                    curr[d] += 1
                    break
                else:
                    curr[d] = 0
            else:
                break
            ind.append(tuple(curr))
        if sort:
            ind.sort(key=lambda a:(sum(a),(np.array(a)**2).sum(),a[::-1]))
        return ind

And a RadialPolyharmonSpline is basically the same thing with an integer kernel param and degree=1.

[Jwink3101]

Other things not included in this but I think worth considering are scaling all inputs to a unit hypercube. That fixes some issues in practice but also may be the kind of thing that should be left up to the users

[rgommers]

My code seems to be pickleable at the moment. Not having *args seems to go a long way

nice

5180 feels out of scope. If you're making a nearest neighbor RBF, then that should be its own class since it is, at its heart, a different animal.

yes, makes sense.

[rgommers]

@Jwink3101 thanks for the sketch of your new class.

@ev-br could you have a quick peak at the proposed API?

[fellhorn]

I tried out the implementation of @Jwink3101 and had quite positive results for my problem.
Are there still plans to integrate it in scipy or ways how to help integrating it?

@Jwink3101 would you give your code snippet under scipy's BSD-3 so we can use it for now?

[Jwink3101]

I am happy to release it under whatever. Though if possible, I would like to make the pull request though that’s not a huge deal.

It still needs minor work such as to remove the reference to fitting epsilon.

There is also a way to estimate the leave one out error based on inverting the Gram matrix. (This is how I optimize epsilon in mine). I removed it but i think it is actually a really useful feature. I’d be interested in thoughts on putting it back. I can copy the code when I am not on mobile.

The issue is, in the main code, you solve the equations but don’t invert the matrix (since it’s less stable to do so). For the leave-one-our error, you do need to invert it. When I do this with Gaussian Processes, I use the cholesky to do both but many kernels are not positive definite so that doesn’t work.

This isn’t a big deal but still worth considering.

[ev-br]

I would like to make the pull request

Please do!

[fabienlefloch]

I have some doubts about the validity of the smoothing approach implemented here, as well as specified in various references such as Wendland (2006) Computational Aspects of Radial Basis
Function Approximation or Myers (1999) Smoothing and interpolation with radial basis
functions.
In particular, the classic Tikhonov regularization
min ||A*x - y||^2 +lambda*||x||^2,
does not reduce to the least squares solution of
(A+lambda*I)*x = y as suggested in Wendland (2006).
It reduces to
(At*A + lambda*I)*x = At*y,
where At is the matrix transpose, see wikipedia, or On the Radial Basis Function Interpolation II by J. Xiao.

On my own examples, the two kinds of smoothing produce significantly different outputs, with the classic Tikhonov approach being much more stable.