Fix that PodIP field is temporarily removed for a terminal pod

[mimowo]

What type of PR is this?

/kind bug
/kind regression

What this PR does / why we need it:

Additionally, due to the bug there is one more unnecessary API call sent by Kubelet.

Which issue(s) this PR fixes:

Fixes #125370

Special notes for your reviewer:

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  template:
    spec:
      containers:
      - name: pi
        image: howardjohn/shell
        command:
        - bash
        - -c
        - |
          sleep 3
          exit 0
      restartPolicy: Never
  backoffLimit: 1

Output to > kubectl get pod -w -ocustom-columns=NAME:.metadata.name,PHASE:.status.phase,IP:.status.podIP | ts "%Y-%m-%d %H:%M:%.S":
before:

2024-06-10 09:43:05.083475 NAME       PHASE     IP
2024-06-10 09:43:05.083657 pi-bslsx   Pending   <none>
2024-06-10 09:43:05.087107 pi-bslsx   Pending   <none>
2024-06-10 09:43:05.095358 pi-bslsx   Pending   <none>
2024-06-10 09:43:07.197929 pi-bslsx   Running   10.244.1.21
2024-06-10 09:43:10.203104 pi-bslsx   Running   10.244.1.21
2024-06-10 09:43:11.324259 pi-bslsx   Succeeded   <none>
2024-06-10 09:43:12.208545 pi-bslsx   Succeeded   10.244.1.21
2024-06-10 09:43:12.219699 pi-bslsx   Succeeded   10.244.1.21

after fix (note there is no extra API status update):

2024-06-10 09:38:16.270212 NAME       PHASE     IP
2024-06-10 09:38:16.270374 pi-xgtw6   Pending   <none>
2024-06-10 09:38:16.275953 pi-xgtw6   Pending   <none>
2024-06-10 09:38:16.283406 pi-xgtw6   Pending   <none>
2024-06-10 09:38:17.954194 pi-xgtw6   Running   10.244.1.20
2024-06-10 09:38:20.960806 pi-xgtw6   Running   10.244.1.20
2024-06-10 09:38:22.130329 pi-xgtw6   Succeeded   10.244.1.20
2024-06-10 09:38:22.974978 pi-xgtw6   Succeeded   10.244.1.20

Note that the last status update in both cases is the removal of the Job finalizer.

Some questions potential questions based on my priv chats:

1. why in the broken version the last status update contains the IP?
   This is because the last status update is triggered by syncTerminatedPod which generates the apiPodStatus based on the podStatus which is passed from the cache in pod_workers. In contrary, the status update triggered by syncTerminating pods was just based on the container runtime containerRuntime.GetPodStatus.
2. Why does the container runtime not populate the IP information for stopped pods?
   Because of the explicit check for SANDBOX_READY here
3. Why does the podStatus cache contains the IP information when syncTerminatedPod is called. Because it gets updated inside updateCache. Note that, in that case we also fallback to oldPodStatus.IP for stopped containers, see here.
4. Are there more fields that need to be copied?
   It seems not. When calling containerRuntime.GetPodStatus on a stopped pod all fields except for podStatus except for IP and Timestamp are returned. The Timestamp field seems only be used when storing in cache, see here, and here . However, we don't store this status in cache.
5. Why copy the information at the podStatus (*kubecontainer.PodStatus) level, not at the apiPodStatus (v1.PodStatus) level?
   Both would work, but adjusting the underlying structure is less code (single field vs two), and keeps them consistent as commented here
6. Is a more generic approach possible, rather than fixing a specific field?
   Possibly, we could fallback the IPs inside generateAPIPodStatus or convertStatusToAPIStatus, which also have access to oldPodStatus, containing the IPs, but this would affect many more code paths. The proposed fix is minimal to only fix the new status update introduced in Give terminal phase correctly to all pods that will not be restarted #115331: https://github.com/kubernetes/kubernetes/pull/115331/files#diff-67dd9e8f3ee257072765326cb4f242852554a2c0753563fa51e292c0a63a7b94R2022
7. Why after the fix there is one less status update? Because kubelet skips sending status update if there is no difference (skips sending patch if empty inside PatchPodStatus). The only difference between the 2 status updates triggered by syncTerminatingPod and syncTerminatedPod was the presence of the IP fields, so syncTerminatedPod is skipped.

Does this PR introduce a user-facing change?

Fix the bug where PodIP field is temporarily removed for a terminal pod

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[ffromani]

/triage accepted
/priority important-soon

[mimowo]

cc @bobbypage

[ffromani]

/lgtm

/cc @aojea

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 87e79b63ca3f4f611a434e8cf4f88260d03e40ea

[ffromani]

/hold

waiting for review from @bobbypage - feel free to remove anytime

[mimowo]

/test pull-kubernetes-unit
Seems like an unrelated flake in unit tests, opened issue: #125406

[smarterclayton]

/approve

LGTM but will let David or Sergey drive the final ack.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, mimowo, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/kubelet/OWNERS [smarterclayton]
• test/e2e_node/OWNERS [aojea,smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mimowo]

Thank you, so the remaining question is about cherry-pick. I'm not sure we have enough signal based on #125370 (comment). However, I would vote for it because:

1. it is a regression
2. the behavior when IP gets lost for one request and returns back will be confusing for debugging, like oncalling
3. there is one request to the API server per pod lifetime to gain
4. the change is small so the risk is minimal

[aojea]

@mimowo you may need to squash the commits for the backport

LGTM , defer to @SergeyKanzhelev final lgtm

I'm +1 on the backport because the fix is contained and there are a lot of informers that depend on PodIPs field, missing the field can lead to corner and complex cases

[mimowo]

@mimowo you may need to squash the commits for the backport

Thanks! done

[aojea]

/assign @SergeyKanzhelev @bobbypage

[SergeyKanzhelev]

/lgtm

thank you!

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 678395c1479986fbbefeb59c1087a7a91c0fb5bc

[mimowo]

Thank you! I have prepared the cherry-picks for review since 1.27 when the regression started, even though 1.27 is EOL currently. It probably does not hurt to cherry-pick there, otherwise I will just close that PR.