Skip to content

Comments

fix(parquet/pqarrow): selective column reading of complex map column#668

Merged
zeroshade merged 3 commits intoapache:mainfrom
zeroshade:fix-map-pqarrow
Feb 20, 2026
Merged

fix(parquet/pqarrow): selective column reading of complex map column#668
zeroshade merged 3 commits intoapache:mainfrom
zeroshade:fix-map-pqarrow

Conversation

@zeroshade
Copy link
Member

Rationale for this change

Upstream fix for the issue identified in apache/iceberg-go#737. When reading maps with nested values using column indices for selective column reading, if the child fields of the map weren't in the list of indices there was a problem:

  • Maps are represented in Parquet as a list of key-value structs (list<struct<key, value>>
  • The struct MUST have exactly 2 fields (key and value) to be converted into a proper Arrow typed Map column
  • When applying the column filtering, if only the key OR value field (but not both) were in the list of columns, the resulting child struct would only have 1 field
  • As a result, the Map.validateData() method would fail with a panic of arrow/array: map array child array should have two fields.

What changes are included in this PR?

In pqarrow/file_reader.go leaf filtering is disabled when reading a map's key-value struct. This will ensure both the key and value columns are always read together, maintaining the required 2-field structure for map array.

Are these changes tested?

Yes a test case is added for the change.

Are there any user-facing changes?

This only affects map type reading with column filter selection ensuring correctness. The only change is that a failure mode has been eliminated.

@zeroshade zeroshade requested review from kou and lidavidm February 19, 2026 19:56
@zeroshade zeroshade changed the title Fix-map-pqarrow fix(parquet/pqarrow): selective column reading of complex map column Feb 19, 2026
@alexandre-normand
Copy link

Thanks for fixing this quickly, @zeroshade! This is very much appreciated.

@zeroshade zeroshade merged commit a886a57 into apache:main Feb 20, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants