AWS: set lastEvaluatedKey for listTables in DynamoDb Catalog #6823
Conversation
| public List<TableIdentifier> listTables(Namespace namespace) { | ||
| List<TableIdentifier> identifiers = Lists.newArrayList(); | ||
| Map<String, AttributeValue> lastEvaluatedKey; | ||
| Map<String, AttributeValue> lastEvaluatedKey = null; |
There was a problem hiding this comment.
I don't think this change is needed? if not assigned it's default to null
There was a problem hiding this comment.
For local variables, if used without initialisation, compilation fails (using Jdk8). error: variable lastEvaluatedKey might not have been initialized
There was a problem hiding this comment.
I see, then it looks good to me
|
Could you add an integration test about this? |
|
I thought of updating this test to create more tables, and validate the change. As DynamoDb Query reads max 1MB of data (when limit is not specified), need to create large number of tables (~10K with current setup) to reproduce the issue. Please let me know if I should go ahead update number of tables to higher number in the tests, or please suggest if there are any alternatives ways to test it Locally, I could verify the fix by adding |
jackye1995
left a comment
There was a problem hiding this comment.
I see, that's probably why the bug has never been reported. In that case, I am good for now.
We are working on some refactoring of the AWS integration tests at the moment, and can do something about this case afterwards for scale testing. We can let you know when you can add more tests for the DynamoDB catalog to ensure its correctness.
|
+1 for adding some more AWS integration tests specifically focused on scale testing |
amogh-jahagirdar
left a comment
There was a problem hiding this comment.
This PR looks good to me, since it's more involved to have automated tests for this we can table it for another effort to have better scale tests for our AWS integrations.
|
Thanks for the fix! @munendrasn |
|
Thank you all for the review 🙂 |
While working on issue #6763 found that lastEvaluatedKey was not set for subsequent calls in listTables()
@jackye1995 FYI