The esa-matchfinder is a C99 library for efficient Lempel-Ziv factorization using enhanced suffix array (ESA).

Copyright (c) 2022 Ilya Grebnov [email protected]

• The esa-matchfinder is block based algorithm with maximum supported block size of 512 megabytes finding matches in range of 2..64 bytes using 12x bytes of memory. Note, ESA_MATCHFINDER_MATCH_BITS definition could be updated to support larger match finding range, but with corresponding reduction in maximum supported block size.
• The esa-matchfinder is fast in best, average and worst cases (see Benchmarks below). But the esa-matchfinder is sensitive to fast memory and software prefetching and might not be suitable for some CPU architectures. The esa-matchfinder might also be slower than other algorithms on specific data types (DNA sequences in particular), so please benchmark yourself.
• The esa-matchfinder works with compilers from Microsoft and GNU, but I recommend Clang for best performance. Additionally, the esa-matchfinder is designed for 64-bit systems and will work suboptimally on 32-bit system.

The esa-matchfinder uses methodology of bottom-up traversal of the Longest Common Prefix (LCP) interval tree written in 2014-2015 by Eric Biggers [email protected] and dedicated to the public domain worldwide.

The esa-matchfinder finds all distance optimal matches (between min_match_length and max_match_length inclusive) for every position of the input block using following algorithm:

1. Suffix (SA) and longest common prefix (LCP) arrays are constructed for the input block. Next, interval tree is constructed on top of SA and LCP arrays.

• The data structure consisting of SA and LCP is often referred as enhanced suffix array (ESA). Hence the name of the match finder.

2. Each interval is a maximum range (could not be further extended to the left or right) of suffixes in SA with common prefix of certain length (LCP). This intervals represent internal nodes of suffix tree.
3. Using interval tree we can now traverse up or down to either wider interval with smaller common prefix or narrower intervals with larger common prefixes.
4. For purpose of Lempel-Ziv factorization we only need to support bottom-up traversal, so during interval tree construction we only need to capture link to parent interval and length of interval's common prefix.
5. LCP array is also pruned by min_match_length and max_match_length to reduce size and depth of interval tree.
6. Additionally, for each position of input block we capture link to a leaf interval corresponding to that position, so we can start bottom-up traversal during factorization phase.
7. SA, LCP and interval tree construction is done during input block parsing phase in linear time with optional multi-threaded optimization using OpenMP.
8. LZ factorization phase is done from left to right by bottom-up traversal of interval tree for each position from input block by reading and updating each corresponding interval with latest offset.

The esa-matchfinder released under the Apache License Version 2.0 and is considered suitable for production use. However, no warranty or fitness for a particular purpose is expressed or implied.

• June 12, 2022 (1.0.0)
  • Initial public release of the esa-matchfinder.

#include "esa_matchfinder.h"

long long multi_pass_optimal_parse(const unsigned char * buffer, int size)
{
    long long total_matches = 0;

    void * mf = esa_matchfinder_create(size, /*min_match_length*/ 2, /*max_match_length*/ 64);
    if (mf != NULL && esa_matchfinder_parse(mf, buffer, size) == ESA_MATCHFINDER_NO_ERROR)
    {
        for (int pass = 0; pass < 2; pass += 1)
        {
            ESA_MATCHFINDER_MATCH matches[ESA_MATCHFINDER_MAX_MATCH_LENGTH];

            esa_matchfinder_rewind(mf, /*position*/ 0);

            for (int position = 0; position < size; position += 1)
            {
                total_matches += esa_matchfinder_find_all_matches(mf, matches) - matches;
            }
        }
    }

    esa_matchfinder_destroy(mf);

    return total_matches;
}

• Input files were capped at 64MB for medium distance tests and 510MB for long distance tests
• MMC source code was modified to increase dictionary size to 64MB
• For all match finders maximum match length was set to 64 bytes (other parameters were not changed)
• The timings are minimum of five runs measuring single-threaded performance in optimal parsing mode

• OS: Microsoft Windows 10 Pro 64-Bit
• CPU: Intel Core i7-9700K Processor (12M Cache, 5GHz)
• RAM: 2x8 GB dual-channel DDR4 (4133 MHz, 17-17-17-37)
• Compiler: Microsoft Visual C++ compiler v14.32
• Optimizations: /MD /DNDEBUG /O2 /GL /arch:AVX2

• OS: Ubuntu 20.04 LTS 64-Bit
• CPU: ODROID-N2+ Amlogic S922X (Cortex-A73 2.4Ghz)
• RAM: 4GB LPDDR4 (2666 MHz)
• Compiler: Clang v10.0.0
• Optimizations: -DNDEBUG -O3 -flto=thin -mcpu=native