Intermittent NaNs appearing in results from calling shared-library blis from Julia

[tkelman]

CPU: Core2 Duo E8400 (old machine)
OS: Ubuntu 14.04, x86-64

Compiled BLIS reference configuration, setting BLIS_ENABLE_DYNAMIC_BUILD := yes. By itself, BLIS passes its own make test.

I'm calling into BLIS from Julia by the following steps:

git clone https://github.com/julialang/julia
cd julia
mkdir -p $PWD/usr/lib
cp /path/to/libblis.so $PWD/usr/lib
echo 'override USE_SYSTEM_BLAS = 1' > Make.user
echo 'override USE_BLAS64 = 0' >> Make.user
echo 'override LIBBLAS = -L$(build_libdir) -lblis -lm' >> Make.user
echo 'override LIBBLASNAME = libblis' >> Make.user
make -j8   # this will take quite a while - Julia has lots of big dependencies
make testall

This gives me a different failure each time I repeat make testall. Here are some examples, from the first couple of files in Julia's test suite (linalg1 and linalg2):
https://gist.github.com/47dbd5517c4a6f56fb2e
https://gist.github.com/51835795c8ada7c0f2a1
https://gist.github.com/552ec09e5e78d1cd3da7
https://gist.github.com/7cce0057bf9a3009a92a
https://gist.github.com/0f01364072634cc95be9
https://gist.github.com/9292fc3fe1a2e9afe09d

I'll see if I can translate a few of these test cases into C to figure out whether the problem is reproducible outside of Julia. I'll also try setting the BLIS integer size to 64-bit and delete the USE_BLAS64 = 0 line, see whether that changes anything.

[fgvanzee]

Tyler took a look at your output and had an idea: is Julia calling BLIS via multiple threads? (The reference configuration is not thread-safe.)

[tkelman]

Aha. Well not exactly multiple threads, but the master julia process launches separate worker julia processes to run the tests in parallel. The subprocesses should have separate memory spaces in which they load dynamic libraries, I think. Even when I run the tests in a single julia process, I still get similar-looking non-reproducible results and intermittent NaNs though.

Is there a list of which configurations are or are not thread-safe anywhere? Or should I assume none of them are at this point?

[fgvanzee]

Is there a list of which configurations are or are not thread-safe anywhere? Or should
I assume none of them are at this point?

Sorry, I should have been more descriptive. The reference configuration isn't thread-safe because it is single-threaded and does not enable OpenMP. (OpenMP is currently the only threading model we support, but POSIX threads could be supported in the future.) In order to achieve thread safety, a critical section in the BLIS memory allocator must be enabled. Currently, it's only enabled when OpenMP is enabled, via BLIS_ENABLE_OPENMP.

Now, I'm not sure if the critical section works when BLIS is called from two different application threads. Tyler thinks that it may not do what we want it to do in that situation.

This all may be unrelated, of course. But it would be interesting to enable OpenMP and see if that fixes some or all of the NaNs.

[tkelman]

But it would be interesting to enable OpenMP and see if that fixes some or all of the NaNs.

Sure, happy to try. Where would that flag be located?

[fgvanzee]

Sure, happy to try. Where would that flag be located?

Just add

#define BLIS_ENABLE_OPENMP

to bli_config.h.

[tkelman]

OK, I also had to uncomment -fopenmp from CMISCFLAGS and add it to LDFLAGS (and LIBBLAS in Julia's Make.user). This is giving a different set of errors - either index out of bounds, or LAPACKException(6597069766656), or SingularException(140011638882304), or determinants of the wrong signs.

Trying with 64-bit integers in BLIS resulted in a segfault, I'll see if I can get a backtrace.

[Maratyszcza]

This kind of error might also result from reading uninitialized memory.
Could you try to run under valgrind memcheck to eliminate this possibility?

On Wednesday, July 2, 2014, Tony Kelman notifications@github.com wrote:

CPU: Core2 Duo E8400 (old machine)
OS: Ubuntu 14.04, x86-64

Compiled BLIS reference configuration, setting BLIS_ENABLE_DYNAMIC_BUILD
:= yes. By itself, BLIS passes its own make test.

I'm calling into BLIS from Julia https://github.com/julialang/julia by
the following steps:

git clone https://github.com/julialang/julia
cd julia
mkdir -p $PWD/usr/lib
cp /path/to/libblis.so $PWD/usr/lib
echo 'override USE_SYSTEM_BLAS = 1' > Make.user
echo 'override USE_BLAS64 = 0' >> Make.user
echo 'override LIBBLAS = -L$(build_libdir) -lblis' >> Make.user
echo 'override LIBBLASNAME = libblis' >> Make.user
make -j8 # this will take quite a while - Julia has lots of big dependencies
make testall

This gives me a different failure each time I repeat make testall. Here
are some examples, from the first couple of files in Julia's test suite (
linalg1 https://github.com/JuliaLang/julia/blob/master/test/linalg1.jl
and linalg2
https://github.com/JuliaLang/julia/blob/master/test/linalg2.jl):
https://gist.github.com/47dbd5517c4a6f56fb2e
https://gist.github.com/51835795c8ada7c0f2a1
https://gist.github.com/552ec09e5e78d1cd3da7
https://gist.github.com/7cce0057bf9a3009a92a
https://gist.github.com/0f01364072634cc95be9
https://gist.github.com/9292fc3fe1a2e9afe09d

I'll see if I can translate a few of these test cases into C to figure out
whether the problem is reproducible outside of Julia. I'll also try setting
the BLIS integer size to 64-bit and delete the USE_BLAS64 = 0 line, see
whether that changes anything.

—
Reply to this email directly or view it on GitHub
#12.

[tkelman]

Not sure whether this is useful: https://gist.github.com/tkelman/f03fb2b9779b7adac894 edit: whoops, definitely not useful, valgrind doesn't apply to processes run by make

Regarding the 64-bit integer segfaults, something seemed to be calling lapack dpotrf at the very end of creating Julia's system image, which then went into BLIS with what looked to be garbage dimensions. I wouldn't expect anything to be doing a Cholesky factorization at that moment, so not sure what was happening there.

[Maratyszcza]

I think you'll need to run it as
valgrind --tool=memcheck --trace-children=yes -- make testall
By default it does not instrument child processes.

On Fri, Jul 4, 2014 at 2:59 PM, Tony Kelman notifications@github.com
wrote:

Not sure whether this is useful:
https://gist.github.com/tkelman/f03fb2b9779b7adac894

Regarding the 64-bit integer segfaults, something seemed to be calling
lapack dpotrf at the very end of creating Julia's system image, which
then went into BLIS with what looked to be garbage dimensions. I wouldn't
expect anything to be doing a Cholesky factorization at that moment, so not
sure what was happening there.

—
Reply to this email directly or view it on GitHub
#12 (comment).

[tkelman]

Thanks yeah I realized that a few minutes after initially posting.

So there's a lot going on here https://gist.github.com/tkelman/c55da37d849d01585739 that I don't quite know how to make sense of, though the calls to lapack sgecon_ and dgecon_ look most relevant.

@ViralBShah now might not be a good time, but in case you want to play with an alternate BLAS at all...

[tkelman]

I think I've figured out what's happening. These first few tests in linalg are often going to lapack, which complicates matters. It was more useful to start at the tests that use blas directly.

Julia offers both in-place and allocating versions of blas calls. The allocating versions do not explicitly initialize the contents of the output array before calling blas, they just send in whatever garbage is in the memory and use beta = 0. BLIS appears to be following the mathematical statement of BLAS operations a bit more literally than the reference and other implementations I'm aware of, and is actually multiplying my input garbage by zero. When my input garbage has NaNs (or presumably +/- Inf as well, but that's not as likely to happen randomly), the output ends up with corresponding NaNs in the same locations.

Should BLIS be checking for beta == 0 and special-casing to an assign-only operation, instead of multiplying the input by 0? Many applications may have grown accustomed to relying on this behavior over the years without realizing it.

[ViralBShah]

@tkelman I want to definitely try this out, but only after the 0.3 release. Would be great to have a BLIS.jl julia package for now. You could just replace all calls to Array with zeros on the julia side to get the initializations zeroed out.

[tkelman]

@ViralBShah that's about what I thought you'd say right now, no worries. I'll see what I can do with Clang auto-wrapping and coming up with a package.

With this patch to Julia https://gist.github.com/a16ce380d174b67561cd to initialize the allocating BLAS calls to 0, I see BLAS tests passing reliably for Float32 and Float64. There appears to be a problem with the imaginary part of the result of cdotc, this is likely due to complex type convention. Julia had switched here JuliaLang/julia#5291 to use cblas for these calls due to some troubles with MKL, but BLIS does't provide a cblas compatibility API. Looks like no difference from setting BLIS_ENABLE_C99_COMPLEX.

The linalg tests still see a few nans, lapack might be making some similar beta == 0 initialization assumptions to what unpatched Julia was making.

[fgvanzee]

Should BLIS be checking for beta == 0 and special-casing to an assign-only operation,
instead of multiplying the input by 0? Many applications may have grown accustomed to
relying on this behavior over the years without realizing it.

Generally speaking, BLIS should already be employing this write vs. update optimization for beta == 0 cases. Could you give me more details about how your are running BLIS? Are you configuring with the reference configuration? Which operation is being called that causes the NaNs to pop up?

Looking at the reference gemm micro-kernel, I can't see any reason it would allow uninitialized memory to get in the way of a beta = 0 invocation.

Maybe it's a different operation altogether (e.g. level-1 / level-2) that is causing the problem?

[tkelman]

Are you configuring with the reference configuration?

Yes, only change here is BLIS_ENABLE_DYNAMIC_BUILD := yes.

Which operation is being called that cause the NaNs to pop up? Looking at the reference gemm micro-kernel

Not sure whether gemm was ever causing trouble, I'll see how many test failures I can cause from direct BLAS calls. syrk definitely was leaving input NaNs in place with beta = 0.