[sig-node] Fix or reduce frequency or switch off perma-failing jobs

[dims]

Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here:
http://storage.googleapis.com/k8s-metrics/failures-latest.json

CI Job	Days Failed
ci-cri-containerd-node-e2e-features	37
ci-cri-containerd-node-e2e-unlabelled	38
ci-crio-cgroupv1-node-e2e-eviction	541
ci-crio-cgroupv1-node-e2e-flaky	504
ci-kubernetes-e2e-node-canary	676
ci-kubernetes-node-arm64-ubuntu-serial	8
ci-kubernetes-node-kubelet-containerd-performance-test	38
pull-kubernetes-node-e2e-containerd-dra	3
pull-kubernetes-node-kubelet-serial-containerd	39
pull-kubernetes-node-kubelet-serial-cpu-manager	1192
pull-kubernetes-node-kubelet-serial-crio-cgroupv1	342
pull-kubernetes-node-kubelet-serial-crio-cgroupv2	384
pull-kubernetes-node-kubelet-serial-pod-disruption-conditions	201
pull-kubernetes-node-kubelet-serial-cpu-manager	677
ci-kubernetes-node-kubelet-serial-cri-o	87

Additional Context:
kubernetes/test-infra#18600

[dims]

/sig node

[dims]

cc @SergeyKanzhelev @mrunalp @derekwaynecarr @dchen1107

[ffromani]

/cc

[akhilerm]

/cc

[pacoxu]

ci-cri-containerd-node-e2e-features 、ci-cri-containerd-node-e2e-unlabelled 、 ci-kubernetes-node-kubelet-containerd-performance-test are similar.

https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-containerd-performance-test can not show recent jobs. And in history, some jobs are ABORTED after 2 days of hanging.

The running job is no response for a long period like:
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-containerd-performance-test/1665741169050521600/

I suspect those jobs failure are related to the CI platform, and not sure who can take a look on those ABORTED long-hanging jobs.

[pacoxu]

• pull-kubernetes-node-e2e-containerd-dra: is pending for DRA: implement e2e node tests #118044 @bart0sh . Link the issue: dynamic resource allocation: Implement node e2e tests #117067
• arm64 is added recently and processes can be found in [umbrella issue] arm ci for node e2e testing test-infra#29693
• ci-crio-cgroupv1-node-e2e-eviction: there are multi-testcase failures
• • fix eviction failing test for nil feature gates assignment #118492
• ci-crio-cgroupv1-node-e2e-flaky is [sig-node] CriticalPod [Serial] [Disruptive] [NodeFeature:CriticalPod] when we need to admit a critical pod [Flaky] should be able to create and delete a critical pod #109296
• ci-kubernetes-e2e-node-canary: I have no context about this job. @endocrimes The failure message is likefailed to run SSH command: out: ssh: connect to host 34.168.1.101 port 22: Connection refused and timeout: failed to run command './ginkgo': Permission denied.
• pull-kubernetes-node-kubelet-serial-containerd is pending for Failure cluster: Density [Serial] [Slow] create a batch of pods latency/resource should be within limit when create 10 pods with 0s interval #118491
• pull-kubernetes-node-kubelet-serial-cpu-manager is using cos-93-lts with containerd 1.4 which failed for CRI v1 validation for cos-93-lts is deprecated and out of service after October 2023 test-infra#29694

[bart0sh]

pull-kubernetes-node-e2e-containerd-dra has been replaced by pull-kubernetes-node-e2e-crio-dra and fixed: https://k8s-testgrid.appspot.com/sig-node-presubmits#pr-node-kubelet-crio-dra

[SergeyKanzhelev]

/assign