Core: Add estimateRowCount for Files and Entries Metadata Tables

[szehon-ho]

Spark queries of Metadata Tables will often try to check row count estimates repeatedly during planning phase, like picking a join strategy.

By default, this involves an extra call to read the manifest file header, as the default logic on the BaseContentScanTask is as follows:

  static long estimateRowsCount(long length, ContentFile<?> file) {
    long[] splitOffsets = splitOffsets(file);
    long splitOffset = splitOffsets != null ? splitOffsets[0] : 0L;
    double scannedFileFraction = ((double) length) / (file.fileSizeInBytes() - splitOffset);
    return (long) (scannedFileFraction * file.recordCount());
  }

However, we actually have this information in the metadata and can just return it directly without reading the file header.

This can save many calls on the Spark driver for planning (one call per manifest file, so potentially many thousands, in serial manner).

This is useful as many Spark maintenance procedures involve querying these tables ( expireSnapshots, removeOrphanFiles)

[szehon-ho]

cc @amogh-jahagirdar @RussellSpitzer does it make sense? Thanks

[amogh-jahagirdar]

@szehon-ho This was a great find, this improvement makes sense to me! I'll wait before merging in case @RussellSpitzer has any comments

[RussellSpitzer]

Looks good to me! Great find

[lurnagao-dahua]

Hi,May I ask why we are not using manifest.addedRowsCount() instead of manifest.addedFilesCount()?
deleted and existing is the same?

[szehon-ho]

This is for a metadata table and not data table. Hence for manifest-read task, each manifest's row count is the number of files it references (added + deleted + existing)

[lurnagao-dahua]

This is for a metadata table and not data table. Hence for manifest-read task, each manifest's row count is the number of files it references (added + deleted + existing)

Got it, thank you for your answer