Spark: support read of partition metadata column when table is over 1k

[dramaticlly]

support of
SELECT *, _partition from iceberg.foo.bar when foo.bar table has over 1000 columns defined

[dramaticlly]

Looks like PRB failed due to junit clean up of temp directory as mentioned in #10569

[szehon-ho]

Looks good to me, thanks @dramaticlly

[szehon-ho]

Merged, thanks @dramaticlly

[pan3793]

... when foo.bar table has over 1000 columns defined

@dramaticlly can you clarify why it was not supported before? where does the restriction come from? I don't see the magic number 1000 in the codebase

[dramaticlly]

... when foo.bar table has over 1000 columns defined

@dramaticlly can you clarify why it was not supported before? where does the restriction come from? I don't see the magic number 1000 in the codebase

@pan3793 so it will only fix the scenario when selecting all fields together with partition metadata column on iceberg table with more than 1000 columns. The unit test shall reproduce the problem if fix is missing. As for the reasoning, the 1000 is coming from the default field id to be assigned for inner partition struct of iceberg table, more detailed analysis can be found in #9923