RFC: Parameters util - auto transfrom mode when parsing multiple values #136
Description
Key information
- RFC PR: feat(parameters): transform = "auto" #133
- Related issue(s), if known: RFC: parameter retrieval utility #94
- Area: Utilities
- Meet tenets: Yes
Summary
Autodetect the type (json, binary or plain text) of configuration property value by the key name (ending with .json/.binary)
Motivation
When storing multiple configuration settings in a parameter store not all of the values will be encoded the same. Some might be a binary (like a base64 decode binary for a image), others might be regular settings and some have complex data structures and would be encoded as json.
You could do this via one big json text or use something like dynamo db to partition your settings using the "id" and use the "sk" for individual settings. Below is a simple mocked example configuration
| id | sk | value |
|---|---|---|
| wallet | color.settings.json | {"foregroundColor":"rgb(0, 0, 0)","labelColor":"rgb(150, 27, 30)","backgroundColor":"rgb($r, $g, $b)"} |
| wallet | allowed.file.extensions.json | ["jpg", "gif", "png"] |
| wallet | maximum.file.size | 1024 |
| wallet | autoreload.enabled | true |
| wallet | default.image.binary | iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII= |
Proposal
We could support these hybrid configuration by introducing a new "transform" method called "auto" that looks at the key ("sk") to determine how to parse the values.
".json" - for values encoded as json
".binary" - for values encoded as base64 binary
Otherwise we assume the value is plain text.
Above example loaded from dynamo:
>>> from aws_lambda_powertools.utilities.parameters import DynamoDBProvider
>>> ddb_provider = DynamoDBProvider("configuration-prod")
>>>
>>> values = ddb_provider.get_multiple("wallet")
>>>
>>> for key, value in values.items():
... print(key, value)
autoreload.enabled true
allowed.file.extensions.json ['jpg', 'gif', 'png']
color.settings.json {'foregroundColor': 'rgb(0, 0, 0)', 'labelColor': 'rgb(150, 27, 30)', 'backgroundColor': 'rgb($r, $g, $b)'}
default.image.binary b'xxxx..'
maximum.file.size 1024
# etc..So in the above example using the ssm parameter store:
>>> from aws_lambda_powertools.utilities.parameters import get_parameter
>>>
>>> values = get_parameters("/wallet.settings", format="auto")
>>>
>>> for key, value in values.items():
... print(key, value)
/wallet.settings/autoreload.enabled true
/wallet.settings/allowed.file.extensions.json ['jpg', 'gif', 'png']
/wallet.settings/color.settings.json {'foregroundColor': 'rgb(0, 0, 0)', 'labelColor': 'rgb(150, 27, 30)', 'backgroundColor': 'rgb($r, $g, $b)'}
/wallet.settings/default.image.binary b'xxxx..'
/wallet.settings/maximum.file.size 1024
# etc..Drawbacks
Main drawback is that is requires some kind of naming convention in your configuration keys.
Rationale and alternatives
- A solution that would only work for Dynamo is to include an additional column for the type. But this would not work with other storage types and would require additional changes to the current implementation.
- Alternatively you could could encode the value with a prefix like
"json: [1, 2]". But then editing these values is tricky
Unresolved questions
Related to parameters in general we would want to also support kms encrypted values in the dynamodb parameter store and for this we might have a special extension like ".secure" or encode the value "${secure:<base64 kms encrypted value>}