[Bug]: flink-optimizer has a probability of java.lang.LinkageError: loader constraint violation.

[XiaoGerGer]

What happened?

I have deployed amoro and flink-optimizer in k8s, there are some error logs in the optimization log, but not all error logs, just probabilistic, can be reproduced in my environment (even if the image is repulled)

arctic163/optimizer-flink:master-snapshot-flink1.15 [7c9855693385]
arctic163/amoro:master-snapshot-hadoop3 [ed95a4b2e74d]

Affects Versions

master-snapshot [ at 2023-12-08 ]

What engines are you seeing the problem on?

Optimizer

How to reproduce

No response

Relevant log output

java.lang.RuntimeException: Run with ugi request failed.
         at com.netease.arctic.table.TableMetaStore.call(TableMetaStore.java:239)
         at com.netease.arctic.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:209)
         at java.base/java.security.AccessController.doPrivileged(Native Method)
         at java.base/javax.security.auth.Subject.doAs(Unknown Source)
         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1816)
         at com.netease.arctic.table.TableMetaStore.doAs(TableMetaStore.java:209)
         at com.netease.arctic.io.ArcticHadoopFileIO.doAs(ArcticHadoopFileIO.java:200)
         at com.netease.arctic.optimizing.AbstractRewriteFilesExecutor.execute(AbstractRewriteFilesExecutor.java:107)
         at com.netease.arctic.optimizing.AbstractRewriteFilesExecutor.execute(AbstractRewriteFilesExecutor.java:63)
         at com.netease.arctic.optimizer.common.OptimizerExecutor.executeTask(OptimizerExecutor.java:105)
         at com.netease.arctic.optimizer.common.OptimizerExecutor.start(OptimizerExecutor.java:34)
         at com.netease.arctic.optimizer.flink.FlinkExecutor.lambda$open$0(FlinkExecutor.java:41)
         at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.LinkageError: loader constraint violation: when resolving method 'void com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(org.objenesis.strategy.InstantiatorStrategy)' the class loader org.apache.flink.util.ChildFirstClassLoader @40849794 of the current class, com/netease/arctic/utils/SerializationUtil$KryoInstantiator, and the class loader 'app' for the method's defining class, com/esotericsoftware/kryo/Kryo$DefaultInstantiatorStrategy, have different Class objects for the type org/objenesis/strategy/InstantiatorStrategy used in the signature (com.netease.arctic.utils.SerializationUtil$KryoInstantiator is in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @40849794, parent loader 'app'; com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy is in unnamed module of loader 'app')
         at com.netease.arctic.utils.SerializationUtil$KryoInstantiator.newKryo(SerializationUtil.java:134)
         at com.netease.arctic.utils.SerializationUtil$KryoSerializerInstance.<init>(SerializationUtil.java:106)
         at java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(Unknown Source)
         at java.base/java.lang.ThreadLocal.setInitialValue(Unknown Source)
         at java.base/java.lang.ThreadLocal.get(Unknown Source)
         at com.netease.arctic.utils.SerializationUtil.kryoSerialize(SerializationUtil.java:79)
         at com.netease.arctic.io.reader.StructLikeFunnel.funnel(StructLikeFunnel.java:20)
         at com.netease.arctic.io.reader.StructLikeFunnel.funnel(StructLikeFunnel.java:11)
         at org.apache.paimon.shade.guava30.com.google.common.hash.AbstractHasher.putObject(AbstractHasher.java:118)
         at org.apache.paimon.shade.guava30.com.google.common.hash.AbstractHashFunction.hashObject(AbstractHashFunction.java:33)
         at org.apache.paimon.shade.guava30.com.google.common.hash.BloomFilterStrategies$2.put(BloomFilterStrategies.java:98)
         at org.apache.paimon.shade.guava30.com.google.common.hash.BloomFilter.put(BloomFilter.java:165)
         at com.netease.arctic.io.reader.CombinedDeleteFilter.applyEqDeletes(CombinedDeleteFilter.java:247)
         at com.netease.arctic.io.reader.CombinedDeleteFilter.applyEqDeletes(CombinedDeleteFilter.java:310)
         at com.netease.arctic.io.reader.CombinedDeleteFilter.filter(CombinedDeleteFilter.java:201)
         at com.netease.arctic.io.reader.GenericCombinedIcebergDataReader.readData(GenericCombinedIcebergDataReader.java:117)
         at com.netease.arctic.optimizing.AbstractRewriteFilesExecutor.rewriterDataFiles(AbstractRewriteFilesExecutor.java:148)
         at com.netease.arctic.table.TableMetaStore.call(TableMetaStore.java:234)
         ... 12 more

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[xxubai]

I have same issue, there is the logs:

java.lang.LinkageError: loader constraint violation: when resolving method "com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(Lorg/objenesis/strategy/InstantiatorStrategy;)V" the class loader (instance of org/apache/flink/util/ChildFirstClassLoader) of the current class, com/netease/arctic/utils/SerializationUtil$KryoInstantiator, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, com/esotericsoftware/kryo/Kryo$DefaultInstantiatorStrategy, have different Class objects for the type org/objenesis/strategy/InstantiatorStrategy used in the signature
	at com.netease.arctic.utils.SerializationUtil$KryoInstantiator.newKryo(SerializationUtil.java:134)
	at com.netease.arctic.utils.SerializationUtil$KryoSerializerInstance.<init>(SerializationUtil.java:106)
	at java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
	at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
	at java.lang.ThreadLocal.get(ThreadLocal.java:172)
	at com.netease.arctic.utils.SerializationUtil.kryoSerialize(SerializationUtil.java:79)
	at com.netease.arctic.utils.SerializationUtil$StructLikeWrapperSerializer.serialize(SerializationUtil.java:189)
	at com.netease.arctic.utils.SerializationUtil$StructLikeWrapperSerializer.serialize(SerializationUtil.java:172)
	at com.netease.arctic.utils.map.SimpleSpillableMap$SimpleSpilledMap.put(SimpleSpillableMap.java:179)
	at com.netease.arctic.utils.map.SimpleSpillableMap.put(SimpleSpillableMap.java:132)
	at com.netease.arctic.utils.map.StructLikeBaseMap.put(StructLikeBaseMap.java:47)
	at com.netease.arctic.io.reader.CombinedDeleteFilter.applyEqDeletes(CombinedDeleteFilter.java:218)
	at com.netease.arctic.io.reader.CombinedDeleteFilter.applyEqDeletes(CombinedDeleteFilter.java:245)
	at com.netease.arctic.io.reader.CombinedDeleteFilter.filter(CombinedDeleteFilter.java:161)
	at com.netease.arctic.io.reader.GenericCombinedIcebergDataReader.readData(GenericCombinedIcebergDataReader.java:123)
	at com.netease.arctic.optimizing.AbstractRewriteFilesExecutor.rewriterDataFiles(AbstractRewriteFilesExecutor.java:146)
	at com.netease.arctic.io.ArcticFileIOAdapter.doAs(ArcticFileIOAdapter.java:45)
	at com.netease.arctic.optimizing.AbstractRewriteFilesExecutor.execute(AbstractRewriteFilesExecutor.java:105)
	at com.netease.arctic.optimizing.AbstractRewriteFilesExecutor.execute(AbstractRewriteFilesExecutor.java:61)
	at com.netease.arctic.optimizer.common.OptimizerExecutor.executeTask(OptimizerExecutor.java:105)
	at com.netease.arctic.optimizer.common.OptimizerExecutor.start(OptimizerExecutor.java:34)
	at com.netease.arctic.optimizer.flink.FlinkExecutor.lambda$open$0(FlinkExecutor.java:41)
	at java.lang.Thread.run(Thread.java:750)

[denizgursoy]

I get the following error.

Caused by: java.lang.LinkageError: loader constraint violation: when resolving method 'void com.esotericsoftware.kryo.Kryo.setInstantiatorStrategy(org.objenesis.strategy.InstantiatorStrategy)' the class loader org.apache.flink.util.ChildFirstClassLoader @1fd56222 of the current class, org/apache/flink/connector/elasticsearch/sink/OperationSerializer, and the class loader 'app' for the method's defining class, com/esotericsoftware/kryo/Kryo, have different Class objects for the type org/objenesis/strategy/InstantiatorStrategy used in the signature (org.apache.flink.connector.elasticsearch.sink.OperationSerializer is in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @1fd56222, parent loader 'app'; com.esotericsoftware.kryo.Kryo is in unnamed module of loader 'app')

Is it related to this? @XiaoGerGer Did you find a solution?