-
Notifications
You must be signed in to change notification settings - Fork 3k
Make AzureProperties w/ shared-key creds serializable #10045
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
@snazy could you add a test please? |
|
any eta on when this would be merged? |
e4e3fed to
72fd23e
Compare
| }) | ||
| .doesNotThrowAnyException(); | ||
|
|
||
| byte[] serialized = out.toByteArray(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it would be good to actually create an instance of AzureProperties and verify that this matches props. Additionally, you can use
| public static <T> T roundTripSerialize(T type) throws IOException, ClassNotFoundException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
something like
AzureProperties props =
new AzureProperties(
ImmutableMap.<String, String>builder()
.put(ADLS_SAS_TOKEN_PREFIX + "foo", "bar")
.put(ADLS_CONNECTION_STRING_PREFIX + "foo", "bar")
.put(ADLS_READ_BLOCK_SIZE, "42")
.put(ADLS_WRITE_BLOCK_SIZE, "42")
.put(ADLS_SHARED_KEY_ACCOUNT_NAME, "me")
.put(ADLS_SHARED_KEY_ACCOUNT_KEY, "secret")
.build());
AzureProperties serdedProps = TestHelpers.roundTripSerialize(props);
assertThat(serdedProps.adlsReadBlockSize()).isEqualTo(props.adlsReadBlockSize());
assertThat(serdedProps.adlsWriteBlockSize()).isEqualTo(props.adlsWriteBlockSize());
assertThat(serdedProps.adlsSasTokens()).isEqualTo(props.adlsSasTokens());
assertThat(serdedProps.namedKeyCreds()).isEqualTo(props.namedKeyCreds());
assertThat(serdedProps.adlsConnectionStrings()).isEqualTo(props.adlsConnectionStrings());
22629aa to
c069f11
Compare
As `StorageSharedKeyCredential` is not serializable, shared key auth doesn't work with Spark.
c069f11 to
99c1d2f
Compare
…10045) As `StorageSharedKeyCredential` is not serializable, shared key auth doesn't work with Spark.
…10045) As `StorageSharedKeyCredential` is not serializable, shared key auth doesn't work with Spark.
As
StorageSharedKeyCredentialis not serializable, shared key auth doesn't work with Spark.