Can not read data when there is an required filed under an optional stuct

[zymap]

Apache Iceberg version

1.8.1 (latest release)

Query engine

None

Please describe the bug 🐞

My table schema is

2025-03-03T17:39:41,451 - INFO  - [main:IcebergTable@131] - Create table with schema table {
  0: address: optional string
  1: age: required int
  2: id: required int
  3: name: optional string
  4: school: optional struct<14: detail: optional string, 15: year: required int>
  5: score: required double
  6: tags: optional map<string, string>
  7: __key: optional binary
  8: __eventTime: optional long
  9: __publishTime: optional long
  10: __properties: optional map<string, string>
  11: __schemaVersion: optional long
  12: __messageOffset: optional long
  13: __partition: optional int
}

When I read data using the IcebergGenerics.read(table).build();, it throws the following exception:

java.lang.IllegalArgumentException: Missing required field: year

	at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.struct(BaseParquetReaders.java:326)
	at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.struct(BaseParquetReaders.java:252)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visit(TypeWithSchemaVisitor.java:148)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visitField(TypeWithSchemaVisitor.java:183)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visitFields(TypeWithSchemaVisitor.java:198)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visit(TypeWithSchemaVisitor.java:47)
	at org.apache.iceberg.data.parquet.BaseParquetReaders.createReader(BaseParquetReaders.java:69)
	at org.apache.iceberg.data.parquet.GenericParquetReaders.buildReader(GenericParquetReaders.java:56)
	at org.apache.iceberg.data.GenericReader.lambda$openFile$2(GenericReader.java:119)
	at org.apache.iceberg.parquet.ReadConf.<init>(ReadConf.java:121)
	at org.apache.iceberg.parquet.ParquetReader.init(ParquetReader.java:71)
	at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:91)
	at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:37)
	at org.apache.iceberg.util.Filter.lambda$filter$0(Filter.java:34)
	at org.apache.iceberg.io.CloseableIterable$2.iterator(CloseableIterable.java:89)
	at org.apache.iceberg.io.CloseableIterable$ConcatCloseableIterable$ConcatCloseableIterator.hasNext(CloseableIterable.java:274)

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[hangc0276]

Looks like this bug is introduced by #11785 @rdblue Would you please help take a look? thanks.

[lliangyu-lin]

I wasn't able to reproduce the issue...
Reproduction step I tried

• Create schema with required id and optional school(with required year and optional details) columns

Schema schema = new Schema(
    required(1, "id", Types.LongType.get()),
    optional(2, "school", Types.StructType.of(
        required(3, "year", Types.IntegerType.get()),
        optional(4, "details", Types.StringType.get())
    ))
);

• Create HadoopTables with the Schema
• Create records with combination of empty school column and school column with empty details.
• Using Table newAppend API to add the parquet files with above records to the table
• Run IcebergGenerics.read(table).build() to check the records

Could you share more details on how the table is created and how data is added to the table? It's possible the issue is happening on write side.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'