Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Opentelemetry propagation using b3multi is not working in latest 1.12.1 version of ingress controller #13155

Open
kumardushyant opened this issue Apr 3, 2025 · 1 comment
Open

Opentelemetry propagation using b3multi is not working in latest 1.12.1 version of ingress controller #13155

kumardushyant opened this issue Apr 3, 2025 · 1 comment
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@kumardushyant
Copy link

What happened:
When nginx ingress is configured to use b3multi propagation at location, controller fails with error - Unsupported propagation type

What you expected to happen:
Expectation is that nginx ingress controller should not fail and produce headers as per b3multi standard

nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller -------------------------------------------------------------------------------
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller E0403 08:03:59.386729      10 queue.go:131] "requeuing" err=<
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller       -------------------------------------------------------------------------------
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller       Error: exit status 1
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller       2025/04/03 08:03:59 [error] 454#454: Unsupported propagation type
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller       nginx: [error] Unsupported propagation type
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller       nginx: configuration file /tmp/nginx/nginx-cfg3417173387 test failed
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller       -------------------------------------------------------------------------------
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller  > key="ingress-nginx/nginxingress-ingress-nginx-controller-admission-6v7wm"
nginxingress-ingress-nginx-controller-76954b54d9-l7p9d controller I0403 08:03:59.386859      10 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"nginxingress-ingress-nginx-controller-76954b54d9-l7p9d", UID:"7b159a9c-ea9b-44be-93a9-d3dd18375ab2", APIVersion:"v1", ResourceVersion:"28467", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX:

It seems nginx ingress controller latest version is not using latest opentelemetry nginx plugin.

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

nginxingress-ingress-nginx-controller-84c4bbf866-5slks:/$ /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.12.1
  Build:         51c2b819690bbf1709b844dbf321a9acf6eda5a7
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: v1.32.3
Kustomize Version: v5.5.0
Server Version: v1.31.1

Environment:

  • Cloud provider or hardware configuration:
    docker desktop
  • OS (e.g. from /etc/os-release):
    windows 11
  • Kernel (e.g. uname -a):
    windows
  • Install tools: docker desktop
    • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
      kind
  • Basic cluster related info:
    • kubectl version
    • kubectl get nodes -o wide
NAME                    STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                       CONTAINER-RUNTIME
desktop-control-plane   Ready    control-plane   16h   v1.31.1   172.18.0.5    <none>        Debian GNU/Linux 12 (bookworm)   5.15.167.4-microsoft-standard-WSL2   containerd://1.7.18
desktop-worker          Ready    <none>          16h   v1.31.1   172.18.0.4    <none>        Debian GNU/Linux 12 (bookworm)   5.15.167.4-microsoft-standard-WSL2   containerd://1.7.18
  • How was the ingress-nginx-controller installed: helm chart
    • If helm was used then please show output of helm ls -A | grep -i ingress

Image

  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
USER-SUPPLIED VALUES:
commonLabels: {}
controller:
addHeaders: {}
admissionWebhooks:
  annotations: {}
  certManager:
    admissionCert:
      duration: ""
    enabled: false
    rootCert:
      duration: ""
  certificate: /usr/local/certificates/cert
  createSecretJob:
    name: create
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true
      runAsGroup: 65532
      runAsNonRoot: true
      runAsUser: 65532
      seccompProfile:
        type: RuntimeDefault
  enabled: true
  extraEnvs: []
  failurePolicy: Fail
  key: /usr/local/certificates/key
  labels: {}
  name: admission
  namespaceSelector: {}
  objectSelector: {}
  patch:
    enabled: true
    image:
      digest: sha256:e8825994b7a2c7497375a9b945f386506ca6a3eda80b89b74ef2db743f66a5ea
      image: ingress-nginx/kube-webhook-certgen
      pullPolicy: IfNotPresent
      tag: v1.5.2
    labels: {}
    networkPolicy:
      enabled: false
    nodeSelector:
      kubernetes.io/os: linux
    podAnnotations: {}
    priorityClassName: ""
    rbac:
      create: true
    securityContext: {}
    serviceAccount:
      automountServiceAccountToken: true
      create: true
      name: ""
    tolerations: []
  patchWebhookJob:
    name: patch
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true
      runAsGroup: 65532
      runAsNonRoot: true
      runAsUser: 65532
      seccompProfile:
        type: RuntimeDefault
  port: 8443
  service:
    annotations: {}
    externalIPs: []
    loadBalancerSourceRanges: []
    servicePort: 443
    type: ClusterIP
affinity: {}
allowSnippetAnnotations: false
annotations: {}
autoscaling:
  annotations: {}
  behavior: {}
  enabled: false
  maxReplicas: 11
  minReplicas: 1
  targetCPUUtilizationPercentage: 50
  targetMemoryUtilizationPercentage: 50
autoscalingTemplate: []
config: {}
configAnnotations: {}
configMapNamespace: ""
containerName: controller
containerPort:
  http: 80
  https: 443
containerSecurityContext: {}
customTemplate:
  configMapKey: ""
  configMapName: ""
disableLeaderElection: false
dnsConfig: {}
dnsPolicy: ClusterFirst
electionID: ""
electionTTL: ""
enableAnnotationValidations: true
enableMimalloc: true
enableTopologyAwareRouting: false
extraArgs: {}
extraContainers: []
extraEnvs: []
extraInitContainers: []
extraModules: []
extraVolumeMounts: []
extraVolumes: []
healthCheckHost: ""
healthCheckPath: /healthz
hostAliases: []
hostNetwork: false
hostPort:
  enabled: false
  ports:
    http: 80
    https: 443
hostname: {}
image:
  allowPrivilegeEscalation: false
  chroot: false
  digest: sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b
  digestChroot: sha256:90155c86548e0bb95b3abf1971cd687d8f5d43f340cfca0ad3484e2b8351096e
  image: ingress-nginx/controller
  pullPolicy: IfNotPresent
  readOnlyRootFilesystem: false
  runAsGroup: 82
  runAsNonRoot: true
  runAsUser: 101
  seccompProfile:
    type: RuntimeDefault
  tag: v1.12.1
ingressClass: nginx
ingressClassByName: false
ingressClassResource:
  aliases: []
  annotations: {}
  controllerValue: k8s.io/ingress-nginx
  default: false
  enabled: true
  name: nginx
  parameters: {}
keda:
  apiVersion: keda.sh/v1alpha1
  behavior: {}
  cooldownPeriod: 300
  enabled: false
  maxReplicas: 11
  minReplicas: 1
  pollingInterval: 30
  restoreToOriginalReplicaCount: false
  scaledObject:
    annotations: {}
  triggers: []
kind: Deployment
labels: {}
lifecycle:
  preStop:
    exec:
      command:
      - /wait-shutdown
livenessProbe:
  failureThreshold: 5
  httpGet:
    path: /healthz
    port: 10254
    scheme: HTTP
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1
maxmindLicenseKey: ""
metrics:
  enabled: false
  port: 10254
  portName: metrics
  prometheusRule:
    additionalLabels: {}
    annotations: {}
    enabled: false
    rules: []
  service:
    annotations: {}
    enabled: true
    externalIPs: []
    labels: {}
    loadBalancerSourceRanges: []
    servicePort: 10254
    type: ClusterIP
  serviceMonitor:
    additionalLabels: {}
    annotations: {}
    enabled: false
    metricRelabelings: []
    namespace: ""
    namespaceSelector: {}
    relabelings: []
    scrapeInterval: 30s
    targetLabels: []
minAvailable: 1
minReadySeconds: 0
name: controller
networkPolicy:
  enabled: false
nodeSelector:
  kubernetes.io/os: linux
podAnnotations: {}
podLabels: {}
podSecurityContext: {}
priorityClassName: ""
progressDeadlineSeconds: 0
proxySetHeaders: {}
publishService:
  enabled: true
  pathOverride: ""
readinessProbe:
  failureThreshold: 3
  httpGet:
    path: /healthz
    port: 10254
    scheme: HTTP
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1
replicaCount: 1
reportNodeInternalIp: false
resources:
  requests:
    cpu: 100m
    memory: 90Mi
scope:
  enabled: false
  namespace: ""
  namespaceSelector: ""
service:
  annotations: {}
  appProtocol: true
  clusterIP: ""
  enableHttp: true
  enableHttps: true
  enabled: true
  external:
    enabled: true
  externalIPs: []
  externalTrafficPolicy: ""
  internal:
    annotations: {}
    appProtocol: true
    clusterIP: ""
    enabled: false
    externalIPs: []
    externalTrafficPolicy: ""
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    loadBalancerClass: ""
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    nodePorts:
      http: ""
      https: ""
      tcp: {}
      udp: {}
    ports: {}
    sessionAffinity: ""
    targetPorts: {}
    type: ""
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  labels: {}
  loadBalancerClass: ""
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  nodePorts:
    http: ""
    https: ""
    tcp: {}
    udp: {}
  ports:
    http: 80
    https: 443
  sessionAffinity: ""
  targetPorts:
    http: http
    https: https
  type: LoadBalancer
shareProcessNamespace: false
sysctls: {}
tcp:
  annotations: {}
  configMapNamespace: ""
terminationGracePeriodSeconds: 300
tolerations: []
topologySpreadConstraints: []
udp:
  annotations: {}
  configMapNamespace: ""
unhealthyPodEvictionPolicy: ""
updateStrategy: {}
watchIngressWithoutClass: false
defaultBackend:
affinity: {}
autoscaling:
  annotations: {}
  enabled: false
  maxReplicas: 2
  minReplicas: 1
  targetCPUUtilizationPercentage: 50
  targetMemoryUtilizationPercentage: 50
containerSecurityContext: {}
enabled: false
extraArgs: {}
extraConfigMaps: []
extraEnvs: []
extraVolumeMounts: []
extraVolumes: []
image:
  allowPrivilegeEscalation: false
  image: defaultbackend-amd64
  pullPolicy: IfNotPresent
  readOnlyRootFilesystem: true
  runAsGroup: 65534
  runAsNonRoot: true
  runAsUser: 65534
  seccompProfile:
    type: RuntimeDefault
  tag: "1.5"
labels: {}
livenessProbe:
  failureThreshold: 3
  initialDelaySeconds: 30
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 5
minAvailable: 1
minReadySeconds: 0
name: defaultbackend
networkPolicy:
  enabled: false
nodeSelector:
  kubernetes.io/os: linux
podAnnotations: {}
podLabels: {}
podSecurityContext: {}
port: 8080
priorityClassName: ""
readinessProbe:
  failureThreshold: 6
  initialDelaySeconds: 0
  periodSeconds: 5
  successThreshold: 1
  timeoutSeconds: 5
replicaCount: 1
resources: {}
service:
  annotations: {}
  externalIPs: []
  loadBalancerSourceRanges: []
  servicePort: 80
  type: ClusterIP
serviceAccount:
  automountServiceAccountToken: true
  create: true
  name: ""
tolerations: []
topologySpreadConstraints: []
unhealthyPodEvictionPolicy: ""
updateStrategy: {}
dhParam: ""
global:
image:
  registry: registry.k8s.io
imagePullSecrets: []
namespaceOverride: ""
portNamePrefix: ""
rbac:
create: true
scope: false
revisionHistoryLimit: 10
serviceAccount:
annotations: {}
automountServiceAccountToken: true
create: true
name: ""
tcp: {}
udp: {}

  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

  • Current State of the controller:

    • kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
             app.kubernetes.io/instance=nginxingress
             app.kubernetes.io/managed-by=Helm
             app.kubernetes.io/name=ingress-nginx
             app.kubernetes.io/part-of=ingress-nginx
             app.kubernetes.io/version=1.12.1
             helm.sh/chart=ingress-nginx-4.12.1
Annotations:  meta.helm.sh/release-name: nginxingress
             meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
❯ k get all -n ingress-nginx -A -o wide
NAMESPACE            NAME                                                         READY   STATUS    RESTARTS        AGE     IP            NODE                    NOMINATED NODE   READINESS GATES
default              pod/httpbin-648cb7896c-vf9dg                                 1/1     Running   0               3h42m   10.244.1.8    desktop-worker          <none>           <none>
ingress-nginx        pod/nginxingress-ingress-nginx-controller-6fc8dbcb96-225r9   0/1     Running   0               29s     10.244.1.25   desktop-worker          <none>           <none>
ingress-nginx        pod/nginxingress-ingress-nginx-controller-84c4bbf866-5slks   1/1     Running   0               16m     10.244.1.24   desktop-worker          <none>           <none>
kube-system          pod/coredns-7c65d6cfc9-7vd4g                                 1/1     Running   5 (5h8m ago)    16h     10.244.0.2    desktop-control-plane   <none>           <none>
kube-system          pod/coredns-7c65d6cfc9-bctjk                                 1/1     Running   5 (5h8m ago)    16h     10.244.0.3    desktop-control-plane   <none>           <none>
kube-system          pod/etcd-desktop-control-plane                               1/1     Running   4 (5h8m ago)    5h35m   172.18.0.5    desktop-control-plane   <none>           <none>
kube-system          pod/kindnet-9fckf                                            1/1     Running   5 (5h8m ago)    16h     172.18.0.5    desktop-control-plane   <none>           <none>
kube-system          pod/kindnet-q422j                                            1/1     Running   5 (5h8m ago)    16h     172.18.0.4    desktop-worker          <none>           <none>
kube-system          pod/kube-apiserver-desktop-control-plane                     1/1     Running   4 (5h8m ago)    5h35m   172.18.0.5    desktop-control-plane   <none>           <none>
kube-system          pod/kube-controller-manager-desktop-control-plane            1/1     Running   5 (5h8m ago)    16h     172.18.0.5    desktop-control-plane   <none>           <none>
kube-system          pod/kube-proxy-qw4dl                                         1/1     Running   5 (5h8m ago)    16h     172.18.0.5    desktop-control-plane   <none>           <none>
kube-system          pod/kube-proxy-x7zjx                                         1/1     Running   5 (5h8m ago)    16h     172.18.0.4    desktop-worker          <none>           <none>
kube-system          pod/kube-scheduler-desktop-control-plane                     1/1     Running   5 (5h8m ago)    16h     172.18.0.5    desktop-control-plane   <none>           <none>
local-path-storage   pod/local-path-provisioner-57c5987fd4-gzv66                  1/1     Running   10 (5h7m ago)   16h     10.244.0.4    desktop-control-plane   <none>           <none>
tempo                pod/grafana-7ccb67fdc9-5z8rf                                 1/1     Running   0               3h16m   10.244.1.17   desktop-worker          <none>           <none>
tempo                pod/tempo-0                                                  1/1     Running   0               3h26m   10.244.1.11   desktop-worker          <none>           <none>

NAMESPACE       NAME                                                      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                   AGE     SELECTOR
default         service/httpbin                                           ClusterIP      10.96.34.253    <none>        80/TCP                                                                                                    3h42m   app=httpbin
default         service/kubernetes                                        ClusterIP      10.96.0.1       <none>        443/TCP                                                                                                   16h     <none>
ingress-nginx   service/nginxingress-ingress-nginx-controller             LoadBalancer   10.96.205.122   172.18.0.6    80:31942/TCP,443:32390/TCP                                                                                3h45m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx
ingress-nginx   service/nginxingress-ingress-nginx-controller-admission   ClusterIP      10.96.62.6      <none>        443/TCP                                                                                                   3h45m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx
kube-system     service/kube-dns                                          ClusterIP      10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP                                                                                    16h     k8s-app=kube-dns
tempo           service/grafana                                           NodePort       10.96.179.171   <none>        80:31234/TCP                                                                                              3h24m   app.kubernetes.io/instance=grafana,app.kubernetes.io/name=grafana
tempo           service/tempo                                             ClusterIP      10.96.33.242    <none>        6831/UDP,6832/UDP,3100/TCP,14268/TCP,14250/TCP,9411/TCP,55680/TCP,55681/TCP,4317/TCP,4318/TCP,55678/TCP   3h26m   app.kubernetes.io/instance=tempo,app.kubernetes.io/name=tempo

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS    IMAGES                                          SELECTOR
kube-system   daemonset.apps/kindnet      2         2         2       2            2           kubernetes.io/os=linux   16h   kindnet-cni   docker.io/kindest/kindnetd:v20241007-36f62932   app=kindnet
kube-system   daemonset.apps/kube-proxy   2         2         2       2            2           kubernetes.io/os=linux   16h   kube-proxy    registry.k8s.io/kube-proxy:v1.31.1              k8s-app=kube-proxy

NAMESPACE            NAME                                                    READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS               IMAGES                                                                                                                     SELECTOR
default              deployment.apps/httpbin                                 1/1     1            1           3h42m   httpbin                  kennethreitz/httpbin                                                                                                       app=httpbin
ingress-nginx        deployment.apps/nginxingress-ingress-nginx-controller   1/1     1            1           3h45m   controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx
kube-system          deployment.apps/coredns                                 2/2     2            2           16h     coredns                  registry.k8s.io/coredns/coredns:v1.11.3                                                                                    k8s-app=kube-dns
local-path-storage   deployment.apps/local-path-provisioner                  1/1     1            1           16h     local-path-provisioner   docker.io/kindest/local-path-provisioner:v20240813-c6f155d6                                                                app=local-path-provisioner
tempo                deployment.apps/grafana                                 1/1     1            1           3h24m   grafana                  docker.io/grafana/grafana:11.6.0                                                                                           app.kubernetes.io/instance=grafana,app.kubernetes.io/name=grafana

NAMESPACE            NAME                                                               DESIRED   CURRENT   READY   AGE     CONTAINERS               IMAGES                                                                                                                     SELECTOR
default              replicaset.apps/httpbin-648cb7896c                                 1         1         1       3h42m   httpbin                  kennethreitz/httpbin                                                                                                       app=httpbin,pod-template-hash=648cb7896c
ingress-nginx        replicaset.apps/nginxingress-ingress-nginx-controller-5c5cddf8df   0         0         0       3h11m   controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5c5cddf8df
ingress-nginx        replicaset.apps/nginxingress-ingress-nginx-controller-6d6858d549   0         0         0       3h10m   controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6d6858d549
ingress-nginx        replicaset.apps/nginxingress-ingress-nginx-controller-6fc8dbcb96   1         1         0       29s     controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6fc8dbcb96
ingress-nginx        replicaset.apps/nginxingress-ingress-nginx-controller-75445dcf47   0         0         0       171m    controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=75445dcf47
ingress-nginx        replicaset.apps/nginxingress-ingress-nginx-controller-76954b54d9   0         0         0       170m    controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=76954b54d9
ingress-nginx        replicaset.apps/nginxingress-ingress-nginx-controller-7fc64b69b8   0         0         0       3h28m   controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=7fc64b69b8
ingress-nginx        replicaset.apps/nginxingress-ingress-nginx-controller-84c4bbf866   1         1         1       16m     controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=84c4bbf866
ingress-nginx        replicaset.apps/nginxingress-ingress-nginx-controller-9d6f5c5f4    0         0         0       3h45m   controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=9d6f5c5f4
ingress-nginx        replicaset.apps/nginxingress-ingress-nginx-controller-cc76674db    0         0         0       19m     controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=cc76674db
ingress-nginx        replicaset.apps/nginxingress-ingress-nginx-controller-fd6b66c5     0         0         0       179m    controller               registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx,pod-template-hash=fd6b66c5
kube-system          replicaset.apps/coredns-7c65d6cfc9                                 2         2         2       16h     coredns                  registry.k8s.io/coredns/coredns:v1.11.3                                                                                    k8s-app=kube-dns,pod-template-hash=7c65d6cfc9
local-path-storage   replicaset.apps/local-path-provisioner-57c5987fd4                  1         1         1       16h     local-path-provisioner   docker.io/kindest/local-path-provisioner:v20240813-c6f155d6                                                                app=local-path-provisioner,pod-template-hash=57c5987fd4
tempo                replicaset.apps/grafana-548df6b444                                 0         0         0       3h24m   grafana                  docker.io/grafana/grafana:11.6.0                                                                                           app.kubernetes.io/instance=grafana,app.kubernetes.io/name=grafana,pod-template-hash=548df6b444
tempo                replicaset.apps/grafana-644fc46df9                                 0         0         0       3h19m   grafana                  docker.io/grafana/grafana:11.6.0                                                                                           app.kubernetes.io/instance=grafana,app.kubernetes.io/name=grafana,pod-template-hash=644fc46df9
tempo                replicaset.apps/grafana-7cb489455f                                 0         0         0       3h23m   grafana                  docker.io/grafana/grafana:11.6.0                                                                                           app.kubernetes.io/instance=grafana,app.kubernetes.io/name=grafana,pod-template-hash=7cb489455f
tempo                replicaset.apps/grafana-7ccb67fdc9                                 1         1         1       3h16m   grafana                  docker.io/grafana/grafana:11.6.0                                                                                           app.kubernetes.io/instance=grafana,app.kubernetes.io/name=grafana,pod-template-hash=7ccb67fdc9

NAMESPACE   NAME                     READY   AGE     CONTAINERS   IMAGES
tempo       statefulset.apps/tempo   1/1     3h26m   tempo        grafana/tempo:2.7.1
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
❯ k -n ingress-nginx describe pod nginxingress-ingress-nginx-controller-6fc8dbcb96-225r9
Name:             nginxingress-ingress-nginx-controller-6fc8dbcb96-225r9
Namespace:        ingress-nginx
Priority:         0
Service Account:  nginxingress-ingress-nginx
Node:             desktop-worker/172.18.0.4
Start Time:       Thu, 03 Apr 2025 08:22:54 +0000
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=nginxingress
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.12.1
                  helm.sh/chart=ingress-nginx-4.12.1
                  pod-template-hash=6fc8dbcb96
Annotations:      kubectl.kubernetes.io/restartedAt: 2025-04-03T08:22:54Z
Status:           Running
IP:               10.244.1.25
IPs:
  IP:           10.244.1.25
Controlled By:  ReplicaSet/nginxingress-ingress-nginx-controller-6fc8dbcb96
Containers:
  controller:
    Container ID:    containerd://ed5679f8099717415228415a2d8607add2d7778fa3053db462cc0e42ceba048e
    Image:           registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b
    Image ID:        registry.k8s.io/ingress-nginx/controller@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b
    Ports:           80/TCP, 443/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/nginxingress-ingress-nginx-controller
      --election-id=nginxingress-ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/nginxingress-ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Thu, 03 Apr 2025 08:23:56 +0000
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 03 Apr 2025 08:22:55 +0000
      Finished:     Thu, 03 Apr 2025 08:23:56 +0000
    Ready:          False
    Restart Count:  1
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginxingress-ingress-nginx-controller-6fc8dbcb96-225r9 (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-52dzz (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginxingress-ingress-nginx-admission
    Optional:    false
  kube-api-access-52dzz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age   From                      Message
  ----     ------     ----  ----                      -------
  Normal   Scheduled  81s   default-scheduler         Successfully assigned ingress-nginx/nginxingress-ingress-nginx-controller-6fc8dbcb96-225r9 to desktop-worker
  Warning  RELOAD     80s   nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:22:56 [error] 30#30: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg3474157338 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  76s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:22:59 [error] 32#32: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg472318488 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  73s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:23:03 [error] 33#33: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg2495582254 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  70s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:23:06 [error] 36#36: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg973644942 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  66s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:23:09 [error] 37#37: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg309122785 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  63s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:23:13 [error] 38#38: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg2952555819 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  60s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:23:16 [error] 39#39: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg1337209232 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  56s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:23:19 [error] 40#40: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg4090577555 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  53s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:23:23 [error] 41#41: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg2283506313 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  33s (x6 over 50s)  nginx-ingress-controller  (combined from similar events): Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:23:43 [error] 47#47: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg475679345 test failed

-------------------------------------------------------------------------------
  Normal   Killing  31s                kubelet                   Container controller failed liveness probe, will be restarted
  Normal   Pulled   20s (x2 over 81s)  kubelet                   Container image "registry.k8s.io/ingress-nginx/controller:v1.12.1@sha256:d2fbc4ec70d8aa2050dd91a91506e998765e86c96f32cffb56c503c9c34eed5b" already present on machine
  Normal   Started  20s (x2 over 81s)  kubelet                   Started container controller
  Normal   Created  20s (x2 over 81s)  kubelet                   Created container controller
  Warning  RELOAD   19s                nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:23:57 [error] 32#32: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg3399543475 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  15s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:24:01 [error] 34#34: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg2449803272 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  12s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:24:04 [error] 35#35: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg3009513166 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  9s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:24:07 [error] 36#36: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg888382374 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  5s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:24:11 [error] 37#37: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg424506146 test failed

-------------------------------------------------------------------------------
  Warning  RELOAD  2s  nginx-ingress-controller  Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2025/04/03 08:24:14 [error] 38#38: Unsupported propagation type
nginx: [error] Unsupported propagation type
nginx: configuration file /tmp/nginx/nginx-cfg990277332 test failed

-------------------------------------------------------------------------------
  Warning  Unhealthy  1s (x6 over 71s)  kubelet  Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy  1s (x8 over 71s)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 500
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
Name:                     nginxingress-ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=nginxingress
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.12.1
                          helm.sh/chart=ingress-nginx-4.12.1
Annotations:              meta.helm.sh/release-name: nginxingress
                          meta.helm.sh/release-namespace: ingress-nginx
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginxingress,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.205.122
IPs:                      10.96.205.122
LoadBalancer Ingress:     172.18.0.6 (Proxy)
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31942/TCP
Endpoints:                10.244.1.24:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  32390/TCP
Endpoints:                10.244.1.24:443
Session Affinity:         None
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:                   <none>

  • Current state of ingress object, if applicable:

    • kubectl -n <appnamespace> get all,ing -o wide
    • kubectl -n <appnamespace> describe ing <ingressname>
    • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

  • Others:

    • Any other related information like ;
      • copy/paste of the snippet (if applicable)
      • kubectl describe ... of any custom configmap(s) created and in use
      • Any other related information that may help
❯ k get cm -n ingress-nginx nginxingress-ingress-nginx-controller -o yaml
apiVersion: v1
data:
  allow-snippet-annotations: "true"
  enable-opentelemetry: "true"
  location-snippet: |
    opentelemetry_propagate b3multi;
  otel-sampler: AlwaysOn
  otel-sampler-parent-based: "true"
  otel-sampler-ratio: "1.0"
  otlp-collector-host: tempo.tempo.svc
  otlp-collector-port: "4317"
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: nginxingress
    meta.helm.sh/release-namespace: ingress-nginx
  creationTimestamp: "2025-04-03T04:38:02Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginxingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.12.1
    helm.sh/chart: ingress-nginx-4.12.1
  name: nginxingress-ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "44526"
  uid: ab31ee81-b259-4d31-bde8-3348b67bb865

How to reproduce this issue:

Anything else we need to know:
@kumardushyant kumardushyant added the kind/bug Categorizes issue or PR as related to a bug. label Apr 3, 2025
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Apr 3, 2025
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
[SIG Network] Ingress NGINX
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kumardushyant @k8s-ci-robot