GEP-713 resurrected #3609

guicassolato · 2025-02-11T03:52:57Z

What type of PR is this?
/kind gep

What this PR does / why we need it:
Resurrects GEP-713 (Memorandum) incorporating enhancements discussed at #2927, and burying its forks GEP-2648 and GEP-2649 as Rejected.

Which issue(s) this PR fixes:
Related to #713

Does this PR introduce a user-facing change?:

Enhances GEP-713 (Memorandum) according to top voted suggestions discussed at https://github.com/kubernetes-sigs/gateway-api/discussions/2927, such as:
* merging Direct and Inherited back into a single spec;
* introducing the concept of **merge strategy**

Additionally to:
* targetRef supporting label selectors as an option;
* reduction of targetRef.sectionName to the base case of "it's just another (virtual) resource kind"; and
* algorithm for calculating effective meta resources (effective policies)

And general enhancements to the spec aiming to:
* acknowledge the current known support of the pattern across Gateway API implementations;
* broaden the definitions to potentially welcome known implementations of other meta resource-like concepts into the pattern (or at least acknowledge their similarities with Gateway API)

Moves GEP-713's previous forks GEP-2648 and GEP-2649 to Rejected.

geps/gep-713/index.md

howardjohn · 2025-02-11T04:15:07Z

geps/gep-713/index.md


-Cons:
+#### Target object status


can we give some examples of this? I think I understand it but not certain

Added a simplified example, plus an extension of it for the case including sectionName.

Please let me know if that works or if you expected to see a full YAML.

A full yaml would be nice

geps/gep-713/index.md

howardjohn · 2025-02-11T15:47:10Z

geps/gep-713/index.md

-  going to affect their object, at apply time, which helps a lot with discoverability.
+* **Accepted**: the meta resource passed both syntactic validation by the API server and semantic validation enforced by the controller, such as whether the target objects exist.
+* **Enforced**: the meta resource’s spec is guaranteed to be fully enforced, to the extent of what the controller can ensure.
+* **Partially enforced**: parts of the meta resource’s spec is guaranteed to be enforced, while other parts are known to have been superseded by other specs, to the extent of what the controller can ensure. The status should include details highlighting which parts of the meta resource are enforced and which parts have been superseded, with the references to all other related meta resources.


As long as this is not a MUST then its not a problem, but this seems like it could be quite onerous to compute. For example, imagine I have a global policy and then 1000 namespaces any of which could partially conflict. Its not great to have to 'bubble up' these to the parent.

A concrete example of this in existing Gateway API is attachedRoutes, which is similarly complex for implementations to compute (efficiently)

geps/gep-713/index.md

geps/gep-2648/index.md

geps/gep-2648/metadata.yaml

geps/gep-2649/index.md

geps/gep-2649/metadata.yaml

geps/gep-713/metadata.yaml

mkdocs.yml

geps/gep-713/index.md

youngnick · 2025-02-12T03:41:46Z

geps/gep-713/index.md


-## Background and concepts
+The merge strategies typically include strategies for dealing with conflicting and/or missing specs, such as for applying default and/or override values on the target resources.


I think it's important to note that sometimes the merge strategy may be specified in the design of the object (that is, it's a defaults policy or something), rather than in a field?

In fact, I tend to think that, if the merge strategy is listed in a field, it should be in the status, not the spec, since it's relevant info for users of the Policy more than implementers (who will build the merge strategy into code when handling the Policy anyway).

+1, this feels like something that belongs in status.

+1 that merge strategy may be defined in the metaresource, e.g. the API contract is either only one is allowed per target, or multiple are allowed.

I feel confused about the merge strategy in the status instead of the spec.

Are we talking about the metaresource's status and spec? Or the target's?

The merge strategy, if more than one is supported by the metaresource kind, is a choice of the user that declares an instance of the metaresource. How can it be in the status?

The user literally specify what merge strategy to use when merging that instance of the metaresource. It should be in the spec, no?

Added a few lines about merge strategy as a user choice or not, and reflected in the status stanza of the metaresource.

geps/gep-713/index.md

youngnick · 2025-02-12T04:54:28Z

geps/gep-713/index.md


-**Ana**: _What the hell just happened??_
+If multiple meta resources target the same context, this is considered to be a conflict.


We need to define "context" again here, I think. (I'd forgotten the definition by the time I got to this part).

Suggested change

If multiple meta resources target the same context, this is considered to be a conflict.

If multiple meta resources target the same context (that is, multiple instances of the same or similar policies acting on the same hierarchy have an effective target of the same object), this is considered to be a conflict.

"same", yes; "similar", not a good idea IMO. I think the behavior for different kinds of policies should be undefined.

I'm okay with removing "or similar", but I think that if we're going to leave this as undefined in some cases, we need to be specific in the ones where we do need to have opinions:

For Gateway API Policy objects included in the specification, in the case of intent conflict with some other Policy on Gateway API objects, the Gateway API Policy must take precedence.

For implementation specific Policy objects that affect the same properties across multiple implementations, it's up to the implementations to define behavior. If they don't then the behavior is, necessarily, undefined and could produce differing outcomes depending on unknown factors.

In other words, this is a terrible idea and users should try not to use multiple Policy objects that affect the same things.

youngnick · 2025-02-12T05:00:15Z

geps/gep-713/index.md

-**Chihiro**: _At a guess, all the workloads in the `baker` namespace actually
-fail a lot, but they seem OK because there are retries across the whole
-namespace?_ 🤔
+Conflicts must be resolved by applying a defined *merge strategy* (see further definition in the next section), where the meta resource considered higher between two conflicting specs dictates the merge strategy according to which the conflict must be resolved, defaulting to the lower spec (more specific) beating the higher one if not specified otherwise.


Suggested change

Conflicts must be resolved by applying a defined *merge strategy* (see further definition in the next section), where the meta resource considered higher between two conflicting specs dictates the merge strategy according to which the conflict must be resolved, defaulting to the lower spec (more specific) beating the higher one if not specified otherwise.

Conflicts must be resolved by applying a defined *merge strategy* (see further definition in the next section).

When resolving conflicts, the meta resource higher in the relevant hierarchy dictates the merge strategy - that is, merge strategy conflict resolution works on a least-specific-wins basis. After that the merge strategy's conflict resolution rules apply.

If no merge strategy is specified, then implementations should use more-specific-wins merge strategy by default.

I think this is what you meant here @guicassolato?

Yes, but I happen to find the suggested text more confusing than the original.

"least-specific-wins" and "more-specific-wins" have different subjects in the sentences, and therefore I would phrase it differently to avoid confusion.

A merge strategy is a function that takes as input 2 specs and outputs 1.

One thing is determining the merge strategy. When resolving a conflict posed by 2 metaresources, the least specific metaresource among the two dictates the merge strategy that will be used to solve the conflict, i.e. the function that will take both metaresource specs as input. It's always the least specific metaresource that determines it.

The determined merge strategy can be a merge strategy that resolves to "least-specific-wins" or "more-specific-wins" (and occasionally to things more sophisticated than that, like actual merges).

If the least specific metaresource does not specify a merge strategy, then the merge strategy used to resolve the conflict is "more-specific-wins".

Rephrased a bit to break down as suggested but trying to avoid overloading terminology.

geps/gep-713/index.md

…t object status Signed-off-by: Guilherme Cassolato <[email protected]>

…erm 'strategy' Signed-off-by: Guilherme Cassolato <[email protected]>

Signed-off-by: Guilherme Cassolato <[email protected]>

… metaresources Signed-off-by: Guilherme Cassolato <[email protected]>

Signed-off-by: Guilherme Cassolato <[email protected]>

shashankram · 2025-03-20T21:13:50Z

geps/gep-713/index.md

+The best way to visualize this hierarchy－and therefore the instances of objects organized by the hierarchy－is in the form of a Directed Acyclic Graph (DAG) whose roots are the least specific objects and the leaves are the most specific ones (and ultimately the effective targets of the metaresources). Using a DAG to represent the hierarchy of effective targets ensures that all the relevant objects are represented, and makes the calculation of corresponding combinatorial specs much easier.

-**Chihiro**: _Yeah. Looking a little closer, I think your `baker` rollout this
-morning would have failed without those retries._ 😕
+Lower levels in the hierarchy (e.g., more specific kinds) *inherit* the definitions applied at the higher levels (e.g. less specific kinds), in such a way that higher level rules may be understood as having an “umbrella effect” over everything under that level.


The phrasing of more/less specific objects is quite confusing. Perhaps defining a term to convey their position in the config hierarchy would be clearer.

Rather than changing the terminology, I've added a sentence elaborating on the idea of being higher/lower in the hierarchy. I hope it helps clarify.

geps/gep-713/index.md

shashankram · 2025-03-20T21:52:49Z

geps/gep-713/index.md

+* **Defaults:** more specific specs beats less specific ones.
+* **Overrides:** less specific specs beats more specific ones.


A bit confusing as the value doesn't convey the meaning. Perhaps BottomUp/TopDown, or a value contextualizing the hierarchy is less ambiguous.

The Defaults and Overrides terminology has been employed since the early beginnings of inherited policies in Gateway API. I'm afraid changing this now can lead to even more confusion.

"Bottom-up" and "Top-down" could also clash with the description of the algorithm to calculate effective metaresources, where these terms are employed to describe how one should navigate a path in the hierarchical tree.

The Defaults and Overrides terminology has been employed since the early beginnings of inherited policies in Gateway API. I'm afraid changing this now can lead to even more confusion.

I understand, but that doesn't make it less confusing. The previous GEP was rejected so we should do our best to avoid ambiguity in terminology this time.

We really appreciate the input, @shashankram. And I guess if we can find something better than "defaults" and "overrides" to convey how metaresource/policy rules are applied throughout the hierarchy, I personally think that would be a good addition.

I have to say though that this terminology has so far been well assimilated. This doesn't exclude your criticism of course. I may very well be living in a bubble myself where these words are generally understood, with one exception I will give you: a default value that is specified at a higher, less specific level, in the DAG (i.e. at a node that is closer to the root, including at the root itself) and that is replaced/superseded with a value specified at a lower, more specific level (i.e. at a node that is closer to the leaves, including at a leaf itself) is often described as "overridden at a lower level". I personally find this confusing with otherwise a proper override strategy, i.e., the one where you specify at the less specific level and that yet wins over anything specified at the more specific ones. I made sure to avoid (at least) this confusion in the text.

Other than this, I may need some additional input here to decide how to best move forward. Perhaps @youngnick or @robscott could shine a light.

In time and to be clear, the previous GEP was never rejected. It is actually pretty much alive (as a Memorandum.) We can argue whether "memorandum" is confusing or not for something that is "pretty much alive", but that'd be another conversation 😜

I have to say though that this terminology has so far been well assimilated

Fair enough

geps/gep-713/index.md

Signed-off-by: Guilherme Cassolato <[email protected]>

… semantics rephrased for improved readability Signed-off-by: Guilherme Cassolato <[email protected]>

…'Conflict resolution rules' subsections Signed-off-by: Guilherme Cassolato <[email protected]>

shashankram

My overall feedback is that the GEP is too complex as an initial step to land policy merging semantics in Gateway API. The scenarios described are generally applicable to any form of merging, but I wonder if we can layer merging controls gradually versus in the first pass.

shashankram · 2025-04-03T15:30:36Z

geps/gep-713/index.md

-
-Direct Attached Policies are further specified in the addendum GEP GEP-2648,
-Direct Policy Attachment.
+    This pattern is so far agreed upon only by Gateway API implementers who were in need of an immediate solution and didn't want all their solutions to be completely different and disparate, but does not have wide agreement or review from the rest of Kubernetes (particularly API Machinery).


Could you document the names of the implementers that agreed to this pattern?

It's a rather tacit agreement. I left a list of all known implementations currently doing Policy Attachment at the end of the GEP. I can't tell for sure which ones participated in the early beginning of the pattern, but I dare to say the ones actively maintaining policy kinds with targetRefs stanza are in agreement.

shashankram · 2025-04-07T15:48:36Z

geps/gep-713/index.md

-is more pronounceable than “metaobject”. Additionally, a single word is better
-than a phrase like “wrapper object” or “wrapper resource” overall, although both
-of those terms are effectively synonymous with “metaresource”.
+- _**Metaresource**_: a resource that augments the behavior of another resource without modifying the definition of the resource. Metaresources MUST clearly define a _target_ and an _intent_ as defined in this GEP, and MUST clearly communicate status about whether the augmentation is happening or not.


Unless Metaresource is an actual API Type, referring to it using a one word canonical name is confusing and the name stutters.

Suggested change

- _**Metaresource**_: a resource that augments the behavior of another resource without modifying the definition of the resource. Metaresources MUST clearly define a _target_ and an _intent_ as defined in this GEP, and MUST clearly communicate status about whether the augmentation is happening or not.

- _**Meta-resource**_: a resource that augments the behavior of another resource without modifying the definition of the resource. Metaresources MUST clearly define a _target_ and an _intent_ as defined in this GEP, and MUST clearly communicate status about whether the augmentation is happening or not.

We've been back and forth on this one, but attending to requests it was decided to spell it as a single word. Hence 80d8212.

If this needs to be a single word, I would call this MetaResource not Metaresource for the reason mentioned above. Metaresource reads poor and the name stutters.

shashankram · 2025-04-07T15:55:41Z

geps/gep-713/index.md

-#### Policy type examples
+- Every metaresource MUST include a `targetRefs` stanza specifying which resource(s) the metaresource intends to augment.
+- Every metaresource MUST include one or more implementation-specific fields specifying how the metaresource will augment the behavior of the target resource(s). This is informally referred to as the "spec proper."
+- A metaresource MAY include additional fields specifying a so-called _merge strategy_, i.e., how the metaresource should be combined with other metaresources that affect the same target resource(s). This typically include directives for dealing with conflicting and/or missing specs, such as for applying default and/or override values on the target resources.


Definitions for defaults/overrides should come before this line.

shashankram · 2025-04-07T16:03:42Z

geps/gep-713/index.md

+  - group: gateway.networking.k8s.io/v1
+    kind: Gateway
+    name: my-gateway
+    sectionName: https ## unique name of a listener specified in the object of Gateway kind


What if the targeted Object has distinct sections that need to be referenced using a name? Is a SectionKind/Type necessary to disambiguate (Listener in this case)?

I think this will be necessary at some point for sure. I miss the details that first motivated the current, somewhat simpler approach, based only on name and no section kind/type.

Maybe @robscott can refresh my mind.

shashankram · 2025-04-07T16:04:54Z

geps/gep-713/index.md

+
+##### Targeting virtual types
+
+_Virtual types_ are defined as those with a group unkown by the Kubernetes API server. They can be used to apply metaresources to objects that are not actual Kubernetes resources nor Kubernetes custom resources. Rather, virtual types have a meaning for the metaresource controller responsible for implementing the metaresource.


Suggested change

_Virtual types_ are defined as those with a group unkown by the Kubernetes API server. They can be used to apply metaresources to objects that are not actual Kubernetes resources nor Kubernetes custom resources. Rather, virtual types have a meaning for the metaresource controller responsible for implementing the metaresource.

_Virtual types_ are defined as those with a group unknown by the Kubernetes API server. They can be used to apply metaresources to objects that are not actual Kubernetes resources nor Kubernetes custom resources. Rather, virtual types have a meaning for the metaresource controller responsible for implementing the metaresource.

shashankram · 2025-04-07T17:49:39Z

geps/gep-713/index.md

+To determine which metaresources attached to objects in a hierarchy are higher or lower, use the following rules, continuing on ties:
+1. Between two metaresources at different levels of the hierarchy, the one attached higher wins (i.e. dictates the merge strategy to use to resolve the conflict).
+2. Between two metaresources at the same level of the hierarchy, the older metaresource based on creation timestamp wins.
+3. Between two metaresources at the same level of the hierarchy and identical creation timestamps, the metaresource appearing first in alphabetical order by `{namespace}/{name}` wins.


Suggested change

3. Between two metaresources at the same level of the hierarchy and identical creation timestamps, the metaresource appearing first in alphabetical order by `{namespace}/{name}` wins.

3. Between two metaresources at the same level of the hierarchy and identical creation timestamps, the metaresource appearing first in alphabetical order by `{namespace}/{name}` dictates the merge strategy.

shashankram · 2025-04-07T18:24:03Z

geps/gep-713/index.md

+* **Defaults:** more specific specs beats less specific ones.
+* **Overrides:** less specific specs beats more specific ones.


I have to say though that this terminology has so far been well assimilated

Fair enough

shashankram · 2025-04-07T18:27:00Z

geps/gep-713/index.md

-One more look out at the lake.
+There are 3 *basic merge strategies*:
+* **None:** the metaresource with the oldest creation timestamp that is attached to a target wins, while all the other metaresources attached to the same target are rejected (`Accepted` status condition set to false).
+* **Defaults:** more specific specs beats less specific ones.


Are Defaults and Overrides flipped here?
Defaults: configured higher in the hierarchy, so less specific beats more specific?
Overrides: lower in the hierarchy so is more specific?

Are Defaults and Overrides flipped here?

They are not.

Defaults: configured higher in the hierarchy,

Defaults are configured higher in the hierarchy, yes.

[…] so less specific beats more specific?

However, as stated, it's more specific that beats less specific; not the other way around.

Overrides: lower in the hierarchy so is more specific?

Yes, lower in the hierarchy is more specific.

Think of the Gateway and HTTPRoute kinds, for example. Gateway is higher than HTTPRoute. Gateways specify less specific rules compared to HTTPRoutes, which in turn are more specific.

A default set at the level of a Gateway is initially valid for all HTTPRoutes under the Gateway (less specific). Lower down, it can be replaced for a specific HTTPRoute (more specific.) Therefore the more specific policy rule set at the lower level of HTTPRoute (as in "lower than Gateway") beats the less specific default set at the higher level of Gateway.

That makes sense. The phrasing is confusing and I would change it to what you explained above.

shashankram · 2025-04-07T18:32:33Z

geps/gep-713/index.md

-(There’s also another use case to consider, in that Chihiro should have been able
-to see that the Policy on the namespace was in use in many places before deleting
-it.)
+Metaresource kinds that implement more than one merge strategy MUST define a clear structure for the instances of metaresource to specify which of the supported strategies to apply. Instances of these metaresources MUST NOT be allowed to declare more than one merge strategy at a time, but only one of the supported strategies. If no merge strategy is specified by a given instance of the metaresource, the **Atomic Defaults** merge strategy MUST be assumed.


Instances of these metaresources MUST NOT be allowed to declare more than one merge strategy at a time, but only one of the supported strategies

What does this mean? Are you referring to 2 policy CRs having different merge strategies?

There are two possibilities for the design of metaresource CRDs:

Single merge strategy – all instances of the CRD (i.e., all CRs) are merged/not merged together following a predefined, static merge strategy that is proper of the CRD.

Multiple/user-defined merge strategy: merge strategy specified in a field of each CR. E.g. Kuadrant's AuthPolicy and RateLimitPolicy.

That makes sense, but the original wording is still ambiguous and needs further clarification. It still doesn't answer my question regarding

Instances of these metaresources MUST NOT be allowed to declare more than one merge strategy at a time, but only one of the supported strategies

I meant one CR can only declare one merge strategy. E.g., if the CRD was designed to enable specifying defaults and overrides merge strategies, it's either defaults or overrides in any given CR, but not both simultaneously.

Two CRs of the same CRD can declare different merge strategies between them, if the CRD supports specifying ~~multiple~~ different strategies. E.g., one CR specifies defaults, another CR specifies overrides.

I'll rephrase it to avoid any confusion.

If the mergeStrategy is an Enum, this would be implicit so you could remove this statement.

There is no hard specification for a mergeStrategy field, so to speak. Implementations are free to come up with their own custom strategy names, field names to specify the chosen strategy in a CR, and even schema (should a single field not be enough to specify it). IOW, it is not an enum.

shashankram · 2025-04-07T18:37:06Z

geps/gep-713/index.md

+* A single kind supported in `spec.targetRefs.kind`
+* Effects of the metaresources do not span across the hierarchy, i.e. the _Declared target kind_ is equal to the _Effective target kind_
+* *None* is the only merge strategy supported
+* If supported, could typically be implemented by directly extending the API of the target kind with the fields otherwise defined at the metaresource (e.g. Gateway API filter)

-##### Cluster Admin Discoverability
+#### Inherited

-The Cluster Admin has similar concerns to the Policy Admin, but with a focus on
-being able to determine what's relevant when something is broken.
+* Superset of the above
+* Any metaresource kind that do not comply with at least one characteristic of the Direct class of metaresources


These are quite ambiguous without examples. Could you first define why "meta-resource classes" are necessary and why Direct and Inherited are relevant?

Mainly to ease the transition from the current state of the GEP/s.

GEP-713 was once forked into two GEPs – 2648 and 2649, respectively for Direct and Inherited Policies. We're now merging them back into one single GEP.

The difference between these two classes (Direct and Inherited), if relevant, should emerge from the properties of metaresources specified, including and especially from the concept of merge strategy.

Those 2 GEPs are marked as declined. So this GEP should reintroduce those concepts before including them in the spec.

Sorry. IDK if I follow. There's no spec other than the GEP itself. What do you mean by the "GEP should reintroduce before including in the spec"? This subsection of the guide-level explanation is meant to introduce/define the concepts.

I meant we should describe the motivation for "meta-resource classes` as it pertains to this GEP instead of assuming it is a well understood concept.

shashankram · 2025-04-08T15:39:51Z

geps/gep-713/index.md


-How does the Cluster Admin know what Policy is applied where, and what the content
-of that Policy is?
+## End-to-end examples


Could you add example policies in YAML for all the scenarios described below

shashankram · 2025-04-08T16:03:30Z

geps/gep-713/index.md

+      matchLabels:
+        env: production


Would this still be a local namespace selector?

k8s-ci-robot requested review from robscott and youngnick February 11, 2025 03:53

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Feb 11, 2025

howardjohn reviewed Feb 11, 2025

View reviewed changes