Flink: filters not applied at row-level for non-partition columns

[agent-mo]

Apache Iceberg version

0.13.0

Query engine

Flink

Please describe the bug 🐞

I run a program to read data from iceberg table by flink batch api , it seem the flilters() funtion is unless when iceberg table is stored by PARQUET fromat .
Source table t3 has 4 recods and program add filter expression a >= 3 . And , the program result is incorrect !
Please see follow pictures :

However , I can get correct result when iceberg table stored by ORC format.
Wait for some body give me something useful solutions , thanks !

[agent-mo]

Here is the iceberg table desc information.

[agent-mo]

@stevenzwu Hi, can you get me some helps ?

[Fokko]

I can confirm:

public class TestFokko {
    @Test
    public void testFokkoLoadTable() throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();

        Map<String, String> properties = new HashMap<>();

        properties.put("uri", "http://localhost:8181");
        properties.put("s3.access-key-id", "admin");
        properties.put("s3.secret-access-key", "password");
        properties.put("s3.endpoint", "http://localhost:9000");

        CatalogLoader catalog = CatalogLoader.custom(
                "rest",
                properties,
                new Configuration(true),
                "org.apache.iceberg.rest.RESTCatalog"
        );

        TableLoader tableLoader = TableLoader.fromCatalog(catalog, TableIdentifier.of("nyc", "taxis_vendors"));
        DataStream<RowData> batch = FlinkSource.forRowData()
                .env(env)
                .tableLoader(tableLoader)
                .streaming(false)
                .filters(Arrays.asList(Expressions.greaterThanOrEqual("VendorID", 2)))
                .build();

        batch.addSink(new SinkFunction<RowData>() {
            @Override
            public void invoke(RowData value) {
                System.out.println(value.getInt(0));
            }
        });
        env.execute("Test Iceberg Batch Read");
    }
}

In the output I don't see any filtering.

[Fokko]

It pushes the filter down to skip row groups:

iceberg/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java

Lines 1044 to 1052 in d59e6c8

	return new org.apache.iceberg.parquet.ParquetReader<>(
	file,
	schema,
	options,
	readerFunc,
	nameMapping,
	filter,
	reuseContainers,
	caseSensitive);

The readerFunc passed in there just reads the records. So there is no row-level filtering being done currently.

[Fokko]

I'm not sure if the current behavior is intentional, because the residual filter is passed in the builder:

iceberg/flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/source/RowDataFileScanTaskReader.java

Lines 145 to 165 in d59e6c8

	private CloseableIterable<RowData> newParquetIterable(
	FileScanTask task,
	Schema schema,
	Map<Integer, ?> idToConstant,
	InputFilesDecryptor inputFilesDecryptor) {
	Parquet.ReadBuilder builder =
	Parquet.read(inputFilesDecryptor.getInputFile(task))
	.split(task.start(), task.length())
	.project(schema)
	.createReaderFunc(
	fileSchema -> FlinkParquetReaders.buildReader(schema, fileSchema, idToConstant))
	.filter(task.residual())
	.caseSensitive(caseSensitive)
	.reuseContainers();
	if (nameMapping != null) {
	builder.withNameMapping(NameMappingParser.fromJson(nameMapping));
	}
	return builder.build();
	}

[Fokko]

Did some extensive digging into this today. And it looks like the filter operation returns residuals; that means that row groups that may contain valid rows are read as a whole. Fixing this requires quite a bit of an overhaul of the code.

Currently, we read everything directly using FlinkParquetReaders, and we don't filter currently. If the column isn't part of the requested schema, then we can't filter afterward, so we have to make sure that the columns are included, or skip directly while reading.

[Fokko]

@stevenzwu Gentle ping. What do you think of this?

[doki23]

Parquet doesn't support precise filter, so we'd better add a flink filter operator after source scan even though it has been pushed down.

[zhongyujiang]

I don't know why this gets the correct result when stored as ORC, but I think Iceberg always leave the row level filtering to the post scan. And the java doc for FlinkSource#forRowData said that it's equivalent to TableScan, it is not for row level filtering, so I think this is an expected result.

Initialize a FlinkSource.Builder to read the data from iceberg table. Equivalent to TableScan. See more options in ScanContext.
The Source can be read static data in bounded mode. It can also continuously check the arrival of new data and read records incrementally.
Without startSnapshotId: Bounded
With startSnapshotId and with endSnapshotId: Bounded
With startSnapshotId (-1 means unbounded preceding) and Without endSnapshotId: Unbounded

Returns:
FlinkSource.Builder to connect the iceberg table

[doki23]

so I think this is an expected result.

If it's an expected result, we'd better rename 'filter' to 'unpreciseFilter'(maybe) to avoid making developers confused. If not, just make filter intuitive.

[stevenzwu]

Just looked at the TestFlinkScan class. it only has coverage for filtering on partition column. I don't have the full history on the the residual filter behavior. I do agree the current behavior is a bit confusing to the users.

Wondering what's the behavior in the Spark side?

[stevenzwu]

so we'd better add a flink filter operator after source scan

@doki23 that would requires users to add it explicitly, which is also unintuitive for users that filters have to be provided in two different places. E.g. for FLIP-27 source.

    IcebergSource.Builder<RowData> sourceBuilder =
        IcebergSource.forRowData()
            .tableLoader(tableLoader())
            .table(table)
            .assignerFactory(new SimpleSplitAssignerFactory())
            .flinkConfig(config);
    sourceBuilder.filters(filters);  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    sourceBuilder.properties(options);

    DataStream<Row> stream =
        env.fromSource(
                sourceBuilder.build(),
                WatermarkStrategy.noWatermarks(),
                "testBasicRead",
                TypeInformation.of(RowData.class))
            .filter(<some Flink filter function>); <<<<<<<<<<<<<<<<<<<<<<<<<<

[Fokko]

For other engines, the source also filters down to a row level, instead of a row group level as it is now.

I think we need to do a few things:

• When projecting the schema that's being passed down to the parquet reader, make sure that the fields that that's being filtered on, are being read.
• Read as we do now, which will do the partition pruning, metrics evaluation, and row group filtering.
• Filter using Flink since it is probably heavily optimized (and we don't want to reinvite the wheel here).
• Do a final projection to the requested schema that will exclude the fields that are part of the filter, but not part of the selected columns.