Hive: HiveCatalog listTables takes minutes if there are thousands tab…

[vanliu-tx]

…les in namespace

A flag is added to indicate whether to filter iceberg table in HiveCatalog#listTables method. Related to #3907

[hililiwei]

Can it be improved if change to parallelStream?

[vanliu-tx]

Can it be improved if change to parallelStream?

I don't think so. Besides, in our production, we use different namespace schema for different table type. For example, tables under namespace iceberg_{biz_id} are all iceberg tables, tables under namespace hive_{biz_id} are all hive partition tables. It's wasting time for method call List<Table> tableObjects = clients.run(client -> client.getTableObjectsByName(database, tableNames)) in our env.

[vanliu-tx]

@rdblue @jackye1995 could you help to review this?

[hililiwei]

Does it only work in Hivecatalog? If so, would it be better to name 'hive.filter-iceberg-table' or 'hive.show-iceberg-table-only'?

[rdblue]

Can it be improved if change to parallelStream?

Iceberg doesn't use streams for parallelism because they are quite limited. When parallelizing operations, be sure to use Tasks instead.

[rdblue]

I think this option is a good idea, but it should be specific to Hive and we should carefully consider naming to make it clear.

[vanliu-tx]

Does it only work in Hivecatalog? If so, would it be better to name 'hive.filter-iceberg-table' or 'hive.show-iceberg-table-only'?

yes, only in HiveCatalog, will change the name for better understanding.

[rdblue]

Thanks, @vanliu-tx. Looking close to ready. I think you just need to rename the property.

[vanliu-tx]

Thanks, @vanliu-tx. Looking close to ready. I think you just need to rename the property.

I was on vacation yesterday, sorry for the delay.

[vanliu-tx]

@rdblue could you help to merge this PR?

[rdblue]

Thanks, @vanliu-tx!