RFC: Setting SSM Parameters and Secret Manager Secrets

[stephenbawks]

Is this related to an existing feature request or issue?

#2826

Which Powertools for AWS Lambda (Python) utility does this relate to?

Parameters

Summary

There are many use cases where the Parameter store is used with PowerTools everywhere. It's an invaluable tool for getting SSM Parameters and the ability to cache them has been fantastic to save time going back to SSM repeatedly. There is one thing that I find to be missing and that is the ability to put SSM parameters as well as the ability to write secrets back to Secrets Manager as well.

Use case

Parameter Store

Adding the function to put_parameter back to SSM with the ability to make sure it's also being cached when it is written back. This should also include the ability to have the overwrite flag as optional. When parameters are being over-written to Parameter store you need to set the overwrite flag because it's set to false by default.

Secrets Manager

Would also like to add the ability to write values to the Secrets Store as well. There are many uses, especially in Lambda where I need to cache/write something secret, (JWT Tokens for example) so that multiple Lambda's are not all going back to get a new token every time a new one is invoked. "Caching" that value in Secret Manager (or as a SecureString in Parameter Store) is a good use case for this.

Proposal

Tasks

• Define the API for creating/updating secrets/parameters
• Define whether we will cache
• Create a PoC to prove the experience.

Parameter Store

For getting values in SSM, Powertools is using get_parameter. I am proposing that we add a new option for put_paramter for the ability to write a value back to a parameter in SSM.

By default, you have to specify the overwrite=true to write a value to a parameter if there is already an existing value. I am wondering if this should be set to true by default or if we should be setting that on every time you need to overwrite a value. When it is written to Parameter store the put_parameter will return a response that includes the new version of the parameter. So in theory there is already the ability to go back versions and it will hold up to 100 of the previous versions before it will start rolling over the oldest.

Secret Manager

Adding the ability to create, set, and update secrets in Secret Manager.

I think create() makes sense for creating a new one.

set() might be translated into boto3 as put_secret_value. The newly created secret would become the latest version of the secret.

update() sounds very similar to set() but could be used for changing the version of the secret that would be retrieved.

update(secret_name="mySuperSecret", secret_value="toomanysecrets")

update(secret_name="mySuperSecret", secret_value="toomanysecrets", stage="development")

get() would be nice to have with the optional way to get different versions of a secret.

generate_random_secret() is also a nice feature for Secrets Manager. This might be a really nice way for people to generate random secrets that can be added to Powertools and saving the need for our fellow Powertools-ers to write custom code to generate random strings or secrets.

Out of scope

Secrets Manager

I think being able to cache that value would be great but I may have to think through if they are updating a secret with a different version.

I have been thinking about how this might eventually be used for versioning around being able to do "canary" or even cutovers to new secrets. But that is probably future work....

Potential challenges

Secrets Manager

When it comes to having the SecretsManager class also cache the new value, I think it would be great but I may have to think through if they are updating a secret with a different version. Do we cache the new version AND the old version if it exists? What would be the behavior around this?

Dependencies and Integrations

No response

Alternative solutions

No response

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[leandrodamascena]

hey @stephenbawks! Congrats on writing your first RFC, this is an incredible work of ideas and solutions, and I really appreciate this work. I will comment on each block you wrote and leave some new ideas.

From here on, I will use boto3.methodname() to identify methods in the boto3 SDK and method() to identify new methods that will be created according to this RFC.

---

Parameter Store

For getting values in SSM, Powertools is using get_parameter. I am proposing that we add a new option for put_paramter for the ability to write a value back to a parameter in SSM.

By default, you have to specify the overwrite=true to write a value to a parameter if there is already an existing value. I am wondering if this should be set to true by default or if we should be setting that on every time you need to overwrite a value.

I think we need to define the API interface for the create/put/set parameter. Looking at the boto3.put_parameter() method I see many options that a customer can configure, I suggest we do something like:

def set(name: str, value: str, description: str = None, overwrite: bool = False, kms_key_id: str = None, type: str = "String", tier: str = "Standard", **kwargs) 

We can define the type annotations in the PR.

I don't think we should set overwrite to True by default. Doing this we might lead the customer to make an inadvertent mistake and override an existing parameter.

When it is written to Parameter store the put_parameter will return a response that includes the new version of the parameter. So in theory there is already the ability to go back versions and it will hold up to 100 of the previous versions before it will start rolling over the oldest.

I'm curious about this point. Did you add this explanation here to contextualize that we should return the version as it is a standard return from the boto3.put_parameter() API or are you thinking of doing something with it? The boto3.get_parameter() API does not support informing the version you want to retrieve, it will always retrieve the last.one.

---

Secret Manager

Adding the ability to create, set, and update secrets in Secret Manager.

I think create() makes sense for creating a new one.

set() might be translated into boto3 as put_secret_value. The newly created secret would become the latest version of the secret.
update() sounds very similar to set() but could be used for changing the version of the secret that would be retrieved.

I think create() is ok, but I didn't understand the difference between set() and update(), can you elaborate more, please?

get() would be nice to have with the optional way to get different versions of a secret.

We already have a method to retrieve secrets specifying the version: https://docs.powertools.aws.dev/lambda/python/latest/utilities/parameters/#passing-additional-sdk-arguments

generate_random_secret() is also a nice feature for Secrets Manager. This might be a really nice way for people to generate random secrets that can be added to Powertools and saving the need for our fellow Powertools-ers to write custom code to generate random strings or secrets.

I like this API and used it a lot, but in my tests, it adds about 25ms to the Lambda execution just to create a random password. I don't know if we bring much-added value just by wrapping this method in Powertools for AWS Lambda. I would like your opinion here @sthulb and @heitorlessa.

---

Cache

After creating/updating a secret and a parameter I think we should cache this for 5 seconds and if the customer uses parameters.get_parameter("param")/parameters.get_secret("secret") this comes from the cache and we avoid a new API call by the client, what do you think?

---

Next steps

I think we need to create a PoC to validate the experience. In this PoC you don't need to do everything like documentation, testing, and others, just how it will look when a customer uses it. You can take this as an example:

from aws_lambda_powertools.utilities import parameters
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools import Logger

logger = Logger()

def lambda_handler(event: dict, context: LambdaContext) -> dict:
    
    endpoint_parameter = parameters.set_parameter(name="/lambda-powertools/endpoint_comments", value="xyz", ....)
    
    logger.info(endpoint_parameter)

Tasks

• Define the API for creating/updating secrets/parameters
• Define whether we will cache
• Create a PoC to prove the experience.

Thank you once more for your dedicated effort in creating this RFC.

[sthulb]

I think the set() and update() functions should probably be merged into one unless there's a significant use case for why both exist. Versioning alone could be another flag in set()/update() depending which one stays.

As for generate_random_secret() func, I'd probably avoid implementing this – there's a tonne of use cases that need to be thought about before implementation – perhaps scope for another RFC in the future. Things to think about for this feature:

• does it need to be FIPS compliant?
• do we care about secret length?
• can we just read from /dev/(u)random?
• do we need to need to generate to a pattern (letters, numbers, symbols and quantity of each)?

[stephenbawks]

@sthulb @leandrodamascena

Yeah, they are pretty close to what they are doing. I think one question I still have is if we are using set() for writing a secret to Secrets Manager, should we also use set() for a similar syntax for SSM Parameter store? In SSM Parameter store, the boto3 command is put_parameter(). I have mixed feelings about Secrets Manager being set() and the SSM Parameter store it would be put().

Wondering if we should make those both set() since people are typically using them interchangeably.