AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client #6746

JonasJ-ap · 2023-02-04T23:46:01Z

Problem Addressed

This PR fix the problem described in issue #6715 by using reflection to instantiate the httpclient configuration impl class to avoid runtime deps of both url-connection-client and apache-httpclient

Test Environment

Spark3.3, Scala 2.12.

scripts to spawn spark shell:

BRANCH_NAME=bug_fix_aws_httpclient_conflict_prefix_map
DEPENDENCIES=""

# add AWS dependnecy
AWS_SDK_VERSION=2.17.257
AWS_MAVEN_GROUP=software.amazon.awssdk
AWS_PACKAGES=(
    "apache-client"
    "s3"
    "glue"
    "kms"
    "iam"
    "sts"
    "dynamodb"
)
for pkg in "${AWS_PACKAGES[@]}"; do
    DEPENDENCIES+="$AWS_MAVEN_GROUP:$pkg:$AWS_SDK_VERSION,"
done

JARS="iceberg-spark-runtime-3.3_$BRANCH_NAME.jar"

# start Spark SQL client shell
spark-shell --packages=$DEPENDENCIES --jars=$JARS\
    --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
    --conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.demo.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
    --conf spark.sql.catalog.demo.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
    --conf spark.sql.catalog.demo.warehouse=s3://gluetestjonas/warehouse \
    --conf spark.sql.catalog.demo.http-client.type=apache

Test commands:

scala> val data = spark.range(0, 10)
scala> data.writeTo("demo.default.test1").create()

Before the fix:

will raise java.lang.NoClassDefFoundError

java.lang.NoClassDefFoundError: software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient$Builder
        at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
        at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3458)
        at java.base/java.lang.Class.getDeclaredMethod(Class.java:2726)
        at java.base/java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1525)
        at java.base/java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:413)
        at java.base/java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:384)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
        at java.base/java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:384)
        at java.base/java.io.ObjectStreamClass$Caches$1.computeValue(ObjectStreamClass.java:110)
        at java.base/java.io.ObjectStreamClass$Caches$1.computeValue(ObjectStreamClass.java:107)
        at java.base/java.io.ClassCache$1.computeValue(ClassCache.java:73)
        at java.base/java.io.ClassCache$1.computeValue(ClassCache.java:70)
        at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:229)
        at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:211)
        at java.base/java.lang.ClassValue.get(ClassValue.java:117)
        at java.base/java.io.ClassCache.get(ClassCache.java:84)
        at java.base/java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:363)
        at

After the fix

Successfully create the table, indicating that url-connection-client is not required

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java

jackye1995 · 2023-02-05T01:28:02Z

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java

+    DynConstructors.Ctor<HttpClientConfigurations> ctor;
+    try {
+      ctor =
+          DynConstructors.builder(HttpClientConfigurations.class).hiddenImpl(impl).buildChecked();


trying to see if we can avoid having to create a new public interface like HttpClientConfigurations. We can have some static methods in non-public class, and dynamically load them using something like

DynMethods.builder("initialize") .impl(impl, Map.class) .buildStaticChecked()

I see Jack's point on not exposing a new public interface. I am wondering if we can define HttpClientConfigurations as a package private abstract class.

Thank you for your suggestions. Currently I switch the implementation to define HttpClientConfigurations.

For the DynMethods way, it seems the final code will be something like:

case HTTP_CLIENT_TYPE_URLCONNECTION: Object httpClientConfigurations = loadHttpClientConfigurations( UrlConnectionHttpClientConfigurations.class.getName(), urlConnectionHttpClientProperties); ((UrlConnectionHttpClientConfigurations)httpClientConfigurations).configureHttpClientBuilder(builder);

since we no longer have a interface or superclass. I personally feel it wierd to return and then cast an Object directly here. So I vote +1 for the having a package-private abstract class.

Also, now I am curious about why the previous impl

interface HttpClientConfigurations { ...

does not work. It seems having a package-private interface also will be similar to a package-private abstract class in this case. Is there any design purpose to not use package-private interface or do I misunderstand something here. Thank you in advance for your help.

good question. it is a miss from my end. I just assumed it is a public interface. Yeah, I agree that package-private interface should be same.

The abstract class looks good to me. But why not just having a static method though? In that case we completely save the need for them to inherit the same abstract class.

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java

aws/src/test/java/org/apache/iceberg/aws/TestHttpClientConfigurations.java

…rations

aws/src/main/java/org/apache/iceberg/aws/ApacheHttpClientConfigurations.java

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java

rdblue · 2023-02-06T20:52:36Z

core/src/main/java/org/apache/iceberg/util/PropertyUtil.java

+   * @param prefix prefix to choose keys from input map
+   * @return subset of input map with keys starting with provided prefix
+   */
+  public static Map<String, String> propertiesWithPrefixNoTrim(


"Trim" typically means removing whitespace. Is there a better way to name this method? Maybe the startsWith could be passed as a Predicate<String> and this could be filterProperties? Or maybe filterPropertiesByPrefix?

…ter keys

stevenzwu

LGTM. I assume latest change is tested with the env that can reproduce the issue.

stevenzwu · 2023-02-08T00:02:31Z

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java

-
-    if (httpClientApacheUseIdleConnectionReaperEnabled != null) {
-      builder.useIdleConnectionReaper(httpClientApacheUseIdleConnectionReaperEnabled);
+  private Object loadHttpClientConfigurations(String impl) {


nit: maybe add a brief Javadoc to explain why we are doing the reflection here and link to the github issue for more context.

JonasJ-ap · 2023-02-08T00:40:42Z

Thanks everyone for reviewing this! I will add a proof of running the latest update on EKS later today or tomorrow

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java

JonasJ-ap · 2023-02-09T09:33:59Z

Sorry for the late update. It took me some time to construct the EKS environment properly.

Test Environment

AWS EKS: 1.24, Spark 3.1.2

Test spark job / k8s job config:

import org.apache.spark.sql.SparkSession

object GlueApp {

  def main(sysArgs: Array[String]) {
    val spark: SparkSession = SparkSession.builder.
      config("spark.master", "local").
      config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions").
      config("spark.sql.catalog.demo", "org.apache.iceberg.spark.SparkCatalog").
      config("spark.sql.catalog.demo.io-impl", "org.apache.iceberg.aws.s3.S3FileIO").
      config("spark.sql.catalog.demo.warehouse", "s3a://gluetestjonas/warehouse").
      config("spark.sql.catalog.demo.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog").
      config("spark.sql.catalog.demo.client.factory", "org.apache.iceberg.aws.AssumeRoleAwsClientFactory").
      config("spark.sql.catalog.demo.client.assume-role.arn", "arn:aws:iam::481640105715:role/jonasjiang_gluecatalog").
      config("spark.sql.catalog.demo.client.assume-role.region", "us-east-1").
      config("spark.sql.catalog.demo.client.assume-role.session-name", "test").
      config("spark.sql.catalog.demo.client.assume-role.external-id", "1234546").
      config("spark.sql.catalog.demo.http-client.type", "apache").
      getOrCreate()

    val data = spark.range(0, 100)
    data.writeTo("demo.default.table100").createOrReplace()
    spark.sql("SELECT * FROM demo.default.table100").show()

    // read using SQL
    // spark.sql("SELECT * FROM demo.reviews.book_reviews").show()
  }
}

name: spark
          image: jonasjiang/spark-eks:v3.1.2
          args: [
              "/bin/sh",
              "-c",
              "/opt/spark/bin/spark-submit \
            --master k8s://https://kubernetes.default.svc.cluster.local:443 \
            --deploy-mode cluster \
            --name spark-eks \
            --class GlueApp \
            --jars s3a://gluetestjonas/jars/iceberg-spark-runtime-3.1_bug_fix_aws_httpclient_conflict_prefix_map.jar \
            --packages software.amazon.awssdk:apache-client:2.17.257,software.amazon.awssdk:s3:2.17.257,software.amazon.awssdk:glue:2.17.257,software.amazon.awssdk:kms:2.17.257,software.amazon.awssdk:iam:2.17.257,software.amazon.awssdk:sts:2.17.257,software.amazon.awssdk:dynamodb:2.17.257 \
            ...
            --conf spark.kubernetes.container.image=jonasjiang/spark-eks:v3.1.2 \
            ...
            --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
            --conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider \
            --conf spark.kubernetes.authenticate.submission.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
            --conf spark.kubernetes.authenticate.submission.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token \
            s3a://gluetestjonas/jars/scalar-glue_apache.jar

Test result:

Current master branch had the following error:

This PR was fine:

Indicating that we no longer need urlconnection when set http-client.type to apache

jackye1995

Looks good to me!

jackye1995 · 2023-02-09T17:42:58Z

I think all the comments are addressed and we have enough vote, I will go ahead and merge this. Thanks for fixing this with such detailed verification! And thanks for the review @stevenzwu @rdblue

… urlconnection and apache client (apache#6746)

JonasJ-ap added 5 commits February 4, 2023 18:18

first draft, tested locally

13b98b3

make new implementation package-private

e5c75a5

add try-catch

2425561

add unit tests

eef197a

rebase

43983f6

github-actions bot added the AWS label Feb 4, 2023

revise javadoc to get rid of importing urlconnectionhttpclient.builder

61574de

jackye1995 reviewed Feb 5, 2023

View reviewed changes

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java Outdated Show resolved Hide resolved

jackye1995 reviewed Feb 5, 2023

View reviewed changes

JonasJ-ap added 2 commits February 5, 2023 21:44

restore property strings

1b242c0

switch to abstract class

d6e899d

github-actions bot added the core label Feb 6, 2023

fix bug of loading unconfigred instance

d20d707

stevenzwu reviewed Feb 6, 2023

View reviewed changes

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java Outdated Show resolved Hide resolved

stevenzwu reviewed Feb 6, 2023

View reviewed changes

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java Outdated Show resolved Hide resolved

stevenzwu reviewed Feb 6, 2023

View reviewed changes

aws/src/test/java/org/apache/iceberg/aws/TestHttpClientConfigurations.java Outdated Show resolved Hide resolved

fix error messages and refactor the unit test for http client configu…

8c62f8e

…rations

jackye1995 reviewed Feb 6, 2023

View reviewed changes

aws/src/main/java/org/apache/iceberg/aws/ApacheHttpClientConfigurations.java Outdated Show resolved Hide resolved

jackye1995 reviewed Feb 6, 2023

View reviewed changes

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java Outdated Show resolved Hide resolved

rdblue reviewed Feb 6, 2023

View reviewed changes

JonasJ-ap added 4 commits February 7, 2023 16:10

combine two maps

d9341ec

use static method to avoid common classes

b74c0cb

use predicate to filter properties

76a37e6

rename helper method to filterProperties and parse a predicate to fil…

d446aea

…ter keys

stevenzwu approved these changes Feb 8, 2023

View reviewed changes

stevenzwu reviewed Feb 8, 2023

View reviewed changes

jackye1995 reviewed Feb 8, 2023

View reviewed changes

aws/src/main/java/org/apache/iceberg/aws/AwsProperties.java Outdated Show resolved Hide resolved

add javadoc and fix the comment

a12e212

jackye1995 approved these changes Feb 9, 2023

View reviewed changes

jackye1995 merged commit 4d43c25 into apache:master Feb 9, 2023

krvikash pushed a commit to krvikash/iceberg that referenced this pull request Mar 16, 2023

AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both…

dedd4a9

… urlconnection and apache client (apache#6746)

AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client #6746

AWS: Load HttpClientBuilder dynamically to avoid runtime deps of both urlconnection and apache client #6746

Uh oh!

Conversation

JonasJ-ap commented Feb 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem Addressed

Test Environment

Test commands:

Before the fix:

After the fix

Uh oh!

Uh oh!

jackye1995 Feb 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenzwu Feb 6, 2023

Choose a reason for hiding this comment

Uh oh!

JonasJ-ap Feb 6, 2023

Choose a reason for hiding this comment

Uh oh!

stevenzwu Feb 6, 2023

Choose a reason for hiding this comment

Uh oh!

jackye1995 Feb 6, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdblue Feb 6, 2023

Choose a reason for hiding this comment

Uh oh!

stevenzwu left a comment

Choose a reason for hiding this comment

Uh oh!

stevenzwu Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

jackye1995 Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

JonasJ-ap commented Feb 8, 2023

Uh oh!

Uh oh!

JonasJ-ap commented Feb 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment

Test spark job / k8s job config:

Test result:

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

jackye1995 commented Feb 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JonasJ-ap commented Feb 4, 2023 •

edited

Loading

jackye1995 Feb 5, 2023 •

edited

Loading

JonasJ-ap commented Feb 9, 2023 •

edited

Loading