KEP-3610: namespace-wide global env injection

[pacoxu]

• One-line PR description:
  namespace wide env inject into every container

• Issue link: cluster/namespace wide environment variables inject into every container #3610

• Other comments: Create custom cluster wide environment variables inject into every container kubernetes#48180

Closed due to #3612 (comment).

eventually something like mutating CEL admission to do this injection

[pacoxu]

/sig auth

[pacoxu]

/cc @deads2k @fedebongio

for some initial comments

• whether we should add a new admission controller. (There is no new admission controller for years.)
• annotation or configmap or other proposals
• namespace-wide or cluster-wide

[harshanarayana]

I have left a few comment on this based on my prior experience of writing a similar admission controller into the k8s that we managed in house at work. Hope that is all right.

[harshanarayana]

We have an admission controller added to the intree api server on the kubernetes that we manage in house and this format of , and = for separation got too complicated too soon with all the patterns and ability to patch them on the go easily. Instead we decided to switch to using env-to-inject/<env-name>: <env-value> for the annotation. And the values were base64 encoded always to be able to inject complicated patterns.

[pacoxu]

It seems better if we choose to use annotation.
Or we can use configmap for complicated cases.

[harshanarayana]

I would love to see the annotation over the config map as there will be a pre-built assumption that it will have a specific name. Not that it is a bad thing. But, that is just my feeling,

[harshanarayana]

We did the cluster wide by adding the Admission config file that provides a list of cluster wide env that gets injected into all pods across the namespace with a annotation that would let one opt out of getting env injected into the pod if required.

[pacoxu]

In this design, there would be a new admission config yaml for this feature and apiserver should load it. Is there any admission controller that uses this pattern? I need to check.

[harshanarayana]

PodPreset used to be able to do this. If I remember correctly. Where it had its own external config file that you could use ENV values specified right ?

[harshanarayana]

users will get errors ut

I suppose you meant users will not get errors?

[harshanarayana]

We might want to include that this KEP doesn't deal with env that one might want to inject via the valueFrom pattern?

[pacoxu]

Yes. If we want to support that, we need some further design.

[harshanarayana]

Having something like that will be very much useful. Especially since you can enable encryption at rest for sections and injecting the env as valueFrom makes it much safer than actual env in the Pod Spec. Isn't it ?

[harshanarayana]

In case if the global.env values are updated at the namespace level then how would this be pushed down to the POD level ? From the PR tagged to this KEP I don't see that. Might be good to have that in the non-goals ?

[pacoxu]

for pod level, do you mean adding env to every container and init container of the pod?
I mean that env is container-level attribute.

[harshanarayana]

say I create the namespace with ENV x=y,a=b. Over a period of time, I change x=z,a=b How will this value be sent to the pod since the KEP says we won't support the ENV change in POD update case, which is perfectly valid. So, someone will have to delete all the pods in suitable namespace and let them be re-created with new ENV ? case in point is something like proxy with password where the password might changes over a period of time.

[pacoxu]

I think this depens on Should we overwriting existed env? Or do need we to add a flag

Pod is always created by a controller like deployment controller or job controller. If the deployment is without the env, new pods that are created from the deployment would have new env from new injection settings. I think overwriting is too aggressive by default. However, this may be a use case and users may want to overwrite for some reason.

If overwrite is an always behavior, pod update can be supported. Hence, pod env can be overwritten to the latest settings once updated.

[harshanarayana]

True. That begs the question, if we are to do these kind of injection, won't it be better off done at the Deployment/STS/Job/CronJob level than the eventual Pod level ? Since trying to patch a running pod on a namespace to inject or update an env will lead to an error error: failed to patch env update to pod template:....* anyway. One has to delete the pod and let it recreate with new Env. But if this has to be dealt with at the pod level then we kind of end up bypassing all the good bits provided by deployments/sts and such in terms of their rolling update nature.

I am partial to saying, if the global env on the namespace is updated, then users will have to explicitly perform a rollout of the resources in that namespace so that the pods of the said resources can inherit the new env ?

[harshanarayana]

I think overwriting is too aggressive by default.

This I agree on. The default should always be ignoring the overwrite. If the env exists, it should have been added as a conscious decision on the end user part.

But what I was mainly concerned about was the change of env that was injected to the Pod spec by the admission controller. That also means, we would have to have a way to distinguish that if the admission controller did add the ENV or it opted not to because there was an override at the Pod spec level already. We have done something similar by just adding an annotation saying if the ENV was indeed injected by the Admission controller or not. If it was not injected by the admission controller, all possible further actions will ignore such pods honouring the default override nature.

[pacoxu]

This I agree on. The default should always be ignoring the overwrite. If the env exists, it should have been added as a conscious decision on the end user part.

I changed to using API Object instead of annotation. I followed the LimitRange design and the default will be applied if it is not set. If we want to overwrite, I add another forceEnv like hard(default like soft).

[tallclair]

This is a feature that developers might want to leverage, and not something that should have restricted use. Putting this in a namespace annotation requires they have update access on the namespace, but we consider that an elevated permission (e.g. it gives control over pod-security & network policy labels).

It seems like a first-class API would be a better fit for something like this? Using an annotation or configmap feels like a hack to get around the extra work required for an API. A full API object raises the bar, but that might not be a bad thing.

[tallclair]

An API object also addresses the validation concerns, and makes it easier to support the ValueFrom use case.

[pacoxu]

Yes. Like LimitRange or ResourceQuota. It would be better.

I updated the KEP with API object design.

[tallclair]

I would like to see a much more fleshed out motivation for this. Specifically, I don't think this has adequately answered the question of why this should be a built in controller, and not a webhook (or eventually CEL admission policy)

[pacoxu]

Do you mean another proposal like below?

• User can use CEL admission fro a new CRD
• And using a webhook to add env according to the new CRD

[pacoxu]

We can use CEL admission policy to validate if an env value is meeting the requirement.

https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/

object.envars.filter(e, e.name == 'MY_ENV').all(e, e.value.matches('^[a-zA-Z]*)	Validate the 'value' field of a listMap entry where key field 'name' is 'MY_ENV'

[liggitt]

It's not clear to me how this is different in nature from the PodPreset feature that never progressed past alpha and was removed.

Using admission webhooks for now and eventually something like mutating CEL admission to do this injection seems better at first glance than crafting a built-in API with this narrow of a use case.

[pacoxu]

Using admission webhooks for now and eventually something like mutating CEL admission to do this injection seems better at first glance than crafting a built-in API with this narrow of a use case.

@liggitt thanks for your information here and at the sig-auth meeting.
I have searched #2876 and it is in the future plan part.

CEL might be used in Kubernetes for extensibility beyond CRD validation. The future plans section of this KEP explains how CEL might be used for general admission control, defaulting, and conversion. This KEP aims to prove the utility of CEL for both the immediate use case (CRD validation) and these future use cases. This KEP also aims to use CEL in a way that is congruent with these future use cases.

Is there any process for mutating CEL admission? The use case here is one of the defaulting parts IMO.

/close

[liggitt]

Is there any process for mutating CEL admission? The use case here is one of the defaulting parts IMO.

Not yet, but it's a goal... use/integration of CEL is being worked on incrementally to help us gain more experience with integration CEL, with the following items progressing/completing ahead of mutating capabilities:

• CEL validation of CRDs (https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/2876-crd-validation-expression-language) went to beta in 1.25 and is actively progressing toward GA

  Future Work
  CEL for General Admission Control
  The ability to use CEL to extend Kubernetes admission control could allow for both validating and mutating admission control without the use of webhooks.

• CEL validating admission (https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/3488-cel-admission-control), introduced in alpha in 1.26 and actively being worked on in 1.27

  Mutation support. While we believe this enhancement could be extended in the future to support mutations, we believe it is best handled as a separate enhancement. That said, we need to keep mutation in mind in this KEP to ensure we design it in such a way that we don't obviously paint ourselves into a corner where mutation would be difficult to introduce later.

cc @jpbetz @cici37 @tallclair

[jpbetz]

cc @andrewsykim who is exploring CEL for mutating admission

[pacoxu]

• This may have a risk to overwrite system env like KUBERNETES_PORT. Should there be a disallowed list?

[harshanarayana]

If we allow env such as KUBERNETES_SERVICE_HOST to be overwritten, then should we not do the same for KUBERNETES_PORT also? But +1 for having a disallow list.

[pacoxu]

https://github.com/kubernetes/kubernetes/blob/3d67e162a03d0d724dc5a15a0617c5e8572c7b4a/cmd/kubelet/app/options/options.go#L488

AllowedUnsafeSysctls may be something that we can follow. We can mark "KUBERNETES_*" as disallowed.

[harshanarayana]

So what would be the behavior if this is set ?

1. Say the new argument that allows certain unsafe env is not set to KUBERNETES_*
2. Someone tries to create a GlobalEnv resource with KUBERNETS_SOMETHING: somevalue as the env injection.

Do we error out and indicate that you can't even create the GlobalEnv resource until you cleanup the unsafe env or we accept the GlobalEnv but ignore the unsafe one during injection ?

[liggitt]

something related to injecting env into pods seems like it is owned by sig-node more than api-machinery (api-machinery could be participating since it might be involved in the mechanism)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pacoxu
Once this PR has been reviewed and has the lgtm label, please assign johnbelamaric for approval by writing /assign @johnbelamaric in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[harshanarayana]

Should there be an option to suggest ignoring injection of this at the pod level ? Certain usecases like proxy, for example.

If you have a pod needs proxy because it is reaching out to the cluster, but other pods don't need them because they are cluster internal. Injecting the proxy to all pods can cause problems if one misconfigures the entry such as no_proxy that can cause the cluster breakdown. So, would it be good to have a way to annotate a pod and say I am not interested in any of the env injection ? Something global.env/ignore: true?

[pacoxu]

This is another use case. Should I add it to the stories below?

For ValidatingAdmissionPolicy, they use binding mode to add a new binding with matchResources:matchLabels.
We may add a matchLabels field in GlobalEnv. By default, no matchLabels means all.

[harshanarayana]

This is another use case. Should I add it to the stories below?

Would be good to have I think. Having a way to exclude injection can be equally important as having a way to inject something.

By default, no matchLabels means all.

Sure. This will also be inline with how rest of the things do the match. So definitely better than what I mentioned above via the annotation.

[harshanarayana]

If one updated the custom resource GlobalEnv on a cluster with custom values in a given namespace, will that forcefully inject the updated values to the pods on the cluster again or one has to delete the pod manually and force the injection of these env? If the latter, then should we have that under non-goal?

[pacoxu]

Ppd updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds, spec.tolerations

For most other plugins, the injection only happens once the pod is created. We can add this to non-goal.

[harshanarayana]

Ppd updates may not change fields other than spec.containers[].image, spec.initContainers[].image, spec.activeDeadlineSeconds, spec.tolerations

Agreed. The only way to update the envs would be to trigger a delete of the pod. Which is definitely not the right thing to do. A line under non-goal would be good enough I think. Thanks

[k8s-ci-robot]

@pacoxu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-enhancements-test	bb12a9d	link	true	/test pull-enhancements-test
pull-enhancements-verify	bb12a9d	link	true	/test pull-enhancements-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[harshanarayana]

Think this is no more a valid field in the kep.yaml

[harshanarayana]

xref: #3569

[pacoxu]

Thanks for your review.

But according to #3612 (comment), this feature is hard to go to GA like PodPreset. I'd like to close this PR.

[k8s-ci-robot]

@pacoxu: Closed this PR.

In response to this:

Using admission webhooks for now and eventually something like mutating CEL admission to do this injection seems better at first glance than crafting a built-in API with this narrow of a use case.

@LiGgit thanks for your information here and at sig-auth meeting.
I have searched #2876 and it is in the future plan part.

CEL might be used in Kubernetes for extensibility beyond CRD validation. The future plans section of this KEP explains how CEL might be used for general admission control, defaulting and conversion. This KEP aims to prove the utility of CEL for both the immediate use case (CRD validation) and these future use cases. This KEP also aims to use CEL in a way that is congruent with these future use cases.

Is there any process for mutating CEL admission? The use case here is one of the defaulting part IMO.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.