Core: Add Variant logical type for Avro

[aihuaxu]

This PR implements a custom logical type - Variant for Avro files in Iceberg. This is to prepare for adding shredded column metadata, which will be written as a Variant in Avro manifest files.

Part of: #10392

[xxubai]

Thanks for the quick update!

[xxubai]

LGTM

[rdblue]

This is close. There are just two blockers for me:

1. This should not expose unnecessary constants in Variant. Cross-project constant use is unnecessary API surface and can cause issues because it entails unnecessary classloading.
2. The custom order schema visitor should also pass metadata and value result futures.

[rdblue]

This check is fairly difficult to understand. When you have complicated logic like this, I'd recommend using a variable to make it more straightforward and readable:

boolean isMalformedVariant = schema.getLogicalType() instanceof VariantLogicalType && !AvroSchemaUtil.isVariantSchema(schema);
Preconditions.checkArgument(!isMalformedVariant, ...);

However, in this case I think the problem is the structure. I commented below that I think Variant behavior should match map and list behavior. Those don't call field (which is specific to structs in this visitor) and that means even less code is going to be shared with records.

I think the best path forward is to handle the variant separately, rather than trying to share the code that visits fields. Then this Precondition can be back inside the block that validates whether the schema is a variant, which should be just after the visitor.recordLevels.push(name); line:

visitor.recordLevels.push(name);

if (schema.getLogicalType() instanceof VariantLogicalType) {
  Preconditions.checkArgument(AvroSchemaUtil.isVariantSchema(schema), ...)
  T metadataResult = new VisitFuture<>(schema.getField(METADATA).schema(), visitor);
  T valueResult = new VisitFuture<>(schema.getField(VALUE).schema(), visitor);
  return visit(schema, metadataResult, valueResult);
}

[aihuaxu]

Make sense. I think we don't need visitor.recordLevels.push(name) and visitor.recordLevels.pop(); for visiting Variant fields since we know they are not records.

BTW: I didn't change but seems we should change the following

visitor.recordLevels.pop();
return visitor.record(schema, names, Iterables.transform(results, Supplier::get));

to

visitor.recordLevels.push(name);
...
Iterable<> itFields = Iterables.transform(results, Supplier::get);
visitor.recordLevels.pop();
return visitor.record(schema, names, itFields);

in order to fail if there is any recursive nodes in Avro schema?

[rdblue]

Shouldn't this be outside the else block? It should apply to both right?

[aihuaxu]

@rdblue I commented above. As I understand that we are trying to error out for the recursive Avro schema. But since we know Variant record will not have records inside, that's why I didn't place that inside visitor.recordLevels.

[rdblue]

I agree that this is unlikely to be a problem given that there is a check for the structure of a variant. But it would still be safer to go ahead and push the name in case anything changes in the future. This is low priority though.

[rdblue]

Same here. The record names should still be tracked to avoid name duplication issues.

[rdblue]

Should this be visitWithName(METADATA, schema, visitor)? That is used for map key/value and array elements.

[rdblue]

The convert method accepts a schema. Why use the Type version instead of the Schema, String version?

[aihuaxu]

I think I should be able to use Schema version. Let me do that in following PR. Thanks for reviewing and merging it.

[rdblue]

I think we can fix the visitWithName issue in a follow up. I'll merge this to unblock other work.

[rdblue]

Thanks for working on this, @aihuaxu! And thanks for reviewing, @XBaith!