Nginx Controller using endpoints instead of Services

[nalum]

Is there any reason as to why the Nginx controller is set up to get the endpoints that a Service uses rather than using the Service?

When I run updates to a deployment and they get rolled out some requests are being sent to the pods that are being terminated which results in a 5XX error because nginx still thinks it's available. Doesn't the use of the Service take care of this issue?

I'm using this image: gcr.io/google_containers/nginx-ingress-controller:0.8.3

I'd be happy to look into changing this so it works with the Services rather than the Endpoints if it makes sense to everyone that it does that.

[yissachar]

The old docs explained:

The NGINX ingress controller does not uses Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.

[nalum]

Ah okay.

Thanks for that, I didn't find it in my searches.

Are there efforts being made to have it more responsive to the readiness and liveness checks do you know?

[aledbf]

Are there efforts being made to have it more responsive to the readiness and liveness checks do you know?

@nalum yes. We need to release a final version before changing the current behavior.
The idea (we need a POC first) is to use the kong load balancer code that allows the update of the nginx upstreams without requiring a reload.

[nalum]

@aledbf anything I can do to help with this?

[nalum]

Just dropping these comments here:

aledbf Slack 11:59 AM 2017-02-22
@nalum about that, I just need time 🙂 . We need to release 0.9 first. I hope after beta 2 #303 we can release 0.9. After that I will start looking the kong load balancer lua code. Basically this is the code we need to understand and “port” Kong/kong#1735

aledbf Slack 12:00 PM 2017-02-22
@nalum please take this just as an idea and POC. We really need to found a way to avoid reloads and better handling of nginx upstreams

[rikatz]

OK, I was taking a look at this:

https://github.com/yaoweibin/nginx_upstream_check_module

Maybe we can open a feature request to configure this, but I think this is not the case of this issue.

About using services instead of PODs, we also have to take in mind that serviceIPs are valid only inside a Kubernetes Cluster, while POD IPs might be valid in your network (using Calico or some other solution).

Anyway, ingress gets the upstream IP from Service (connect to service, watch the service and check for POD IP changes in that service) so it might be the case to health check the Upstream POD with the module above, instead of using the Service IP :)

@aledbf Is this the case to open a feature request using the upstream healthcheck module or should we keep this open?

Thanks!

[nalum]

@rikatz correct me if I'm wrong, the brief look I took at that repo didn't show that it is using the kubernetes health checks, I would think it better to take advantage of those rather than adding additional checks. So have nginx be more responsive to the changes of the service.

[aledbf]

@rikatz adding an external module for this just duplicates the probes feature already provided by kubernetes.

[aledbf]

@rikatz what this issue is really about how we update the configuration (change in pods) without reloading nginx

[rikatz]

Yeap, Kube health check is the best approach. I was just thinking about an alternative :)

Will take a look in how to change the config without reload the nginx also.

[rikatz]

@aledbf and about this module: https://github.com/yzprofile/ngx_http_dyups_module

And ingress adding / removing upstreams based in HTTP request for this? This is the same approach that NGINX Plus uses to add/remove upstreams without reload / kill nginx process

[aledbf]

@rikatz I tried that module like a year ago (without success). Maybe we need to check again

[rikatz]

@aledbf I did some quick checks here and it worked 'fine'.

But the following situations still 'reloads' the nginx:

• New vHost
• New Path (changed Ingress)
• New SSL Cert

I couldn't verify if, for each upstream change, NGINX got reloaded or if this doesn't happens.

So, is this module of upstreams still applicable to the whole problem? I can start writing a PoC of this approach (it includes changing the nginx.tmpl, the core and a lot of other things) but if you think it's still a valuable change, we can do this :D

Thanks