CDK deploy: Lambda LogRetention resources fail with rate exceeded errors

[jaapvanblaaderen]

Describe the bug

Deploying multiple stacks with a lot of Lambda functions, results in a lot of CloudWatch Log groups being created by CDK. CDK provides retry options through the logRetentionRetryOptions property of a Lambda Function construct to prevent rate exceeded errors from happening.

Until CDK 2.89.0 this worked fine. In CDK 2.90.0, the retry implementation changed in this PR and the rate exceeded errors occur again.

Expected Behavior

The retry mechanism should just function properly, same way it did in older, CDK 2.89 and lower, versions. Stacks should deploy without issues.

Current Behavior

Stacks cannot be deployed with CDK 2.90.0 and up as they error with a Rate exceeded error:

LogRetentionRateLimitStack3 |  57/183 | 11:34:35 AM | CREATE_FAILED        | Custom::LogRetention        | hello_11/LogRetention (hello11LogRetention0EC20DD0) Received response status [FAILED] from custom resource. Message returned: Rate exceeded (RequestId: b84c54c4-3053-487a-82b9-48671904d05a)

Reproduction Steps

I created a small test project to reproduce the issue: https://github.com/jaapvanblaaderen/log-retention-rate-limit. With this simple setup, the issue can be easily reproduced when deploying the stacks in parallel as described in the readme of the repo. I even increased the amount of retries to 50 but then it still fails.

Testing it against 2.89.0 (which works fine) is possible by checking out the cdk-289 tag.

Possible Solution

Revert the retry mechanism as implemented in CDK 2.89.0 or troubleshoot why the new retry mechanism doesn't work properly.

Additional Information/Context

I'm actually a bit surprised and disappointed that this issue popped up again. I fixed this issue myself in CDK 3 years ago and now we have this regression on it which causes it to fail.

CDK CLI Version

2.90.0

Framework Version

No response

Node.js Version

18

OS

OSX

Language

Typescript

Language Version

No response

Other information

Related issues:

• Support retry strategy on Lambda LogRetention #8257
• custom_resources: log retention rate limit error during deploy #24485

[mrgrain]

Thanks @jaapvanblaaderen for reporting and apologies for this regression. The retry mechanism changed from SDK v2 to SDK v3, and was supposed to be smarter. But clearly something did not work. Looking into it.

[jaapvanblaaderen]

Thanks @mrgrain, please let me know if I can assist in any way.

[mrgrain]

With the patched version from my PR, your reproduction example now succeeds! 🥳

[jaapvanblaaderen]

That sounds great!