AWS: WebIdentityTokenFileCredentialsProvider httpclient issue with EKS service account

[stevenzwu]

Feature Request / Improvement

WebIdentityTokenFileCredentialsProvider from DefaultCredentialsProviderchain uses DefaultSdkHttpClientBuilder for STS client to retrieve IAM credentials for service accounts.

public final class DefaultSdkHttpClientBuilder implements SdkHttpClient.Builder {
    private static final SdkHttpServiceProvider<SdkHttpService> DEFAULT_CHAIN = new CachingSdkHttpServiceProvider<>(
            new SdkHttpServiceProviderChain<>(
                    SystemPropertyHttpServiceProvider.syncProvider(),
                    ClasspathSdkHttpServiceProvider.syncProvider()
  ));

Right now, AwsProperties reference classes from both url-connection-client and apache-httpclient. Hence, we have to include both jars as deps to avoid runtime error of .

      case HTTP_CLIENT_TYPE_URLCONNECTION:
        builder.httpClientBuilder(
            UrlConnectionHttpClient.builder()
                .applyMutation(this::configureUrlConnectionHttpClientBuilder));
        break;
      case HTTP_CLIENT_TYPE_APACHE:
        builder.httpClientBuilder(
            ApacheHttpClient.builder().applyMutation(this::configureApacheHttpClientBuilder));
        break;

When both jars are included in the runtime classpath, we will see the following error from ClasspathSdkHttpServiceProvider. Note that having both jars is not an issue for S3Client as AwsProperties configure httpClientBuilder properly.

"Multiple HTTP implementations were found on the classpath. To avoid non-deterministic loading " +
"implementations, please explicitly provide an HTTP client via the client builders, set the %s " +
"system property with the FQCN of the HTTP service to use as the default, or remove all but one " +
"HTTP implementation from the classpath", implSystemProperty.property()))

As the error message indicates, we can set the impl class explicitly via -D system property. This would work.

-Dsoftware.amazon.awssdk.http.service.impl=software.amazon.awssdk.http.apache.ApacheSdkHttpService

I like to discuss an alternative solution. We can define an interface to configure httpclient and two impl class (url-connection-client and apache-httpclient). In AwsProperties, it can use reflection to instantiate the httpclient configuration impl class. This way, we can avoid runtime deps for both jars and the confusion for ClasspathSdkHttpServiceProvider. Then users don't need to set the system property to choose a httpclient impl. it can be managed by dependency management..

what do you think? or is the system property more explicit and clear?

cc @jackye1995 @danielcweeks

Query engine

None

[jackye1995]

If we have to load both http clients to the claasspath, it seems like a big we should fix. I thought when we introduced the config to switch between url and Apache client we already tested that case, but seems not. I am currently short of bandwidth, ping a few people to see if anyone is interested in picking it up

@amogh-jahagirdar @singhpk234 @JonasJ-ap @xiaoxuandev @aajisaka

[stevenzwu]

I thought when we introduced the config to switch between url and Apache client

That config switch code in AwsProperties introduced the problem of requiring both url and apache client jars, which then confused ClasspathSdkHttpServiceProvider for WebIdentityTokenFileCredentialsProvider.

@jackye1995 and others, what do you think of the proposal of using reflection to avoid the runtime requirements of both jars, while still supporting the switch config? It would be good to agree on the direction first. Then we can figure out who can pick it up. I will look into my availability.

[jackye1995]

what do you think of the proposal of using reflection to avoid the runtime requirements of both jars

Yes that's also the direction I am thinking about.

[JonasJ-ap]

I am interested in picking this up and implementing the proprosal. Could you please assign this to me?

[stevenzwu]

@JonasJ-ap assigned the issue to you. you should be able to reproduce it with latest code and run Spark/Flink with EKS service account.