-
Notifications
You must be signed in to change notification settings - Fork 3k
AWS, GCS: Fix issue with Kryo and empty immutable collections #13216
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWS, GCS: Fix issue with Kryo and empty immutable collections #13216
Conversation
bfba154 to
c06303c
Compare
c06303c to
530300b
Compare
530300b to
6bcd1af
Compare
talatuyarer
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
| }); | ||
|
|
||
| // make sure credentials can be accessed after kryo serde | ||
| ResolvingFileIO resolvingIO = TestHelpers.KryoHelpers.roundTripSerialize(resolvingFileIO); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: the assertions look the same for both Kryo and Java serialisation, would it make sense to use a 'mini-parametrisation', something like a for loop:
for serialisation in {Kryo, Java}
roundTripSerialize with serialisation
perform common assertion logic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes that might make sense. We do have the same issue across the codebase so I've opened #13237 to address this
amogh-jahagirdar
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @nastra , and thank you @talatuyarer @nandorKollar for reviewing. this LGTM
While looking at #13174 I've noticed an issue with Kryo and empty storage credentials.
I've added some tests that reproduce the original issue with Kryo, where the empty immutable collection can't be serded and fails when it is accessed with something like
This applies to the
storageCredentialsinGCSFileIO/S3FileIO/ResolvingFileIO