Parquet: Fix decimal data reading from ParquetAvroValueReaders

[ajantha-bhat]

During write, Int64 decimal data is written as long and Int32 decimal data is written as int.

iceberg/parquet/src/main/java/org/apache/iceberg/parquet/ParquetAvroWriter.java

Lines 158 to 161 in b64446c

	case INT32:
	return ParquetValueWriters.ints(desc);
	case INT64:
	return ParquetValueWriters.longs(desc);

But during read, Int32 data is read as long and Int64 data is read as int. Hence the exception.

Fixes: #8245

[ajantha-bhat]

Even though I fixed the actual issue,
the test case passes the read part and fails for decimal type for Equals() method.

Line 220:
Assert.assertEquals("Record " + recordNum + " should match expected", expected, actual);

Looks like Avro genericData doesn't have Decimal type. Was the data suppose to be Fixed instead of Decimal?

Unknown datum type java.math.BigDecimal: 0.00040
org.apache.avro.AvroRuntimeException: Unknown datum type java.math.BigDecimal: 0.00040
	at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:933)
	at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:892)
	at org.apache.avro.generic.GenericData.compare(GenericData.java:1178)
	at org.apache.avro.generic.GenericData.compare(GenericData.java:1154)
	at org.apache.avro.generic.GenericData$Record.equals(GenericData.java:287)
	at org.junit.Assert.isEquals(Assert.java:133)
	at org.junit.Assert.equalsRegardingNull(Assert.java:129)
	at org.junit.Assert.assertEquals(Assert.java:112)
	at org.apache.iceberg.spark.data.TestParquetAvroReader.testCorrectness(TestParquetAvroReader.java:220)

Need to dig deeper on this.
But @Fokko, do you know about this? (tagged you since you are involved in all 3 of the project - avro, parquet, iceberg)

[ajantha-bhat]

I did some narrow down. The issue happens only when the schema field is optional. For the required field it passes.

required(25, "foo", Types.DecimalType.of(7, 5)));

Currently, I changed field to required and so, this issue can be merged.
Later need to investigate for optional field in separate PR.

[Fokko]

@ajantha-bhat I'm pretty comfortable with Avro yes. This is an interesting one. In Avro, both int's and long's have the same zigzag encoding. The difference between optional and required is that optional has an integer in front of the actual value that will indicate if the value is null, or not: https://github.com/apache/iceberg/blob/master/python/pyiceberg/avro/reader.py#L250-L262

[Fokko]

Would we cool if we can find out what's going on with the optional, but this is a great catch @ajantha-bhat