-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CRI] Containerd switching to HTTPS for TLS verification for a HTTP Image Registry on Localhost #4826
Comments
@tejasrao97 Could you try to use
|
@fuweid but, when we are using managed Kubernetes services we won't have access to add any fields into the config.toml of containerd, by doing an ssh into the nodes. We are not facing this issue in managed Kuberentes clusters which are using CRI-O and Docker as the container runtime interface and we haven't added any entry in the config files for CRI-O/Docker for the insecure registry that is hosted inside the cluster. |
Even i am facing the same issue in one of the kubernetes clusters provisioned with container runtime as containerd. We need to find a fix for this ASAP. All these days we had this setup in docker and weren't facing any issues as such. |
link to containerd/cri#1328 |
@fuweid in 1.4.3 version of conatinerd, as of now if the cri is using http while pulling an image from localhost, but after that it's trying to verify the certificates. So if we skip the tls verification by default, when the image is being pulled by the CRI from localhost then I guess that would fix the issue. |
@tejasrao97 We can't skip the tls verification by default for security reason. But I think it sounds good to skip for localhost. |
@fuweid yeah it would be good if we skip it only for localhost. Thank you! 🙂 |
I wonder if it would make sense to have a listing of ips for which tls verificaiton will be skipped.. and default that to localhost ipv4/6. Hmm. |
Any Updates on this ? |
cc @adisky |
/assign |
I tried to setup an insecure docker registry and using ctr/crictl to pull the images without any additional config in /etc/containerd/config.toml, I am able to pull images. |
Hi @adisky, yeah we can pull images but we would have to explicitly pass the skip tls verification flag to ctr/crictl to pull the image from an insecure registry. |
@tejasrao97 I am not passing skip flag. Still able to pull the images, can you check if this works?#5100? |
@adisky I came across this issue few months ago when I tried to create a Kubernetes cluster using kubeadm(1.19.4) with containerd(1.4.3) as the CRI. I don't have the containerd config because I destroyed that cluster. But I remember not making any changes to the contianerd config file, it had the default values. |
@tejasrao97 try the master branch since #5100 has been merged. If it works for you, please close the ticket. Thanks |
@fuweid @adisky @mikebrow @dims I think it's still not working Below is the screenshot of gitlab docker registry in which the image is present Below is the screenshoft of ctr client inside the kubernetes master node failing to pull image from gitlab If I add -k flag to ctr I'm able to pull the images as I had mentioned in the previous comments I built containerd from the master branch and used the resulting containerd binaries to bootstrap the kubernetes cluster using kubeadm. Please find the below screenshots of the binary versions. Thanks |
@tejasrao97 ctr doesn't work for you, please try crictl, thanks |
@fuweid sure I'll try that, but how come it isn't working in Jenkins where ctr is not being used instead a pod template is used. |
Moving to 1.6 milestone since we've released 1.5. |
would like to see the containerd debug log (containerd -l debug) for this scenario We added a bunch of interesting debug out and refactored this code around the time and after this was reported.. it is possible your scenario is failing for a non-obvious reason and then trying your configured mirror/default and showing the non-interesting error and thus you don't get to see the actual reason for failure, esp. noting the referenced pull image secret. |
I tried creating a cluster using kubeadm and contianerd as the CRI, came across a new error!
conatainerd version which I have used is 1.4.3 and the kubeadm version is 1.19.4
I'm getting certificate error from the containerd runtime when the images that are hosted on gitlab tool running as a pod are being pulled using localhost(127.0.0.1) as the registry domain, the request is going through http at first, but later the localhost is getting resolved to the loadbalancer IP on which the gitlab pod is exposed and its trying to validate the certificate as the certificates are not added to the loadbalancer.
Steps to reproduce the issue:
1.Create a kubernetes cluster using kubeadm(version 1.19.4) and containerd (1.4.3).
2.Run gitlab pod in the cluster and try pushing images to the container registry hosted on gitlab using skopeo
3.Create a sample deployment with the image pushed to the gitlab container registry
Describe the results you received:
![image](https://user-images.githubusercontent.com/55922259/101728555-9dd30300-3adc-11eb-86a0-f64c710332f8.png)
![image](https://user-images.githubusercontent.com/55922259/101728595-acb9b580-3adc-11eb-9363-5f01c37a39d1.png)
RESULTS RECEIVED:--->
Containerd fails to pull the image present in gitlab due to certificate error for the gitlab endpoint(load balancer).
Gitlab internally authenticates the user with registry credentials, without tls verification, but containerd CRI is tracing the authentication redirection with tls enabled because of which the image is not getting pulled due to certificate issue.
Describe the results you expected:
![image](https://user-images.githubusercontent.com/55922259/101729036-79c3f180-3add-11eb-88a1-e6cdf22f325d.png)
RESULTS EXPECTED:->>
Docker and CRI-O CRI's are able to pull the image from the similar setup as mentioned above, it would be helpfull if containerd has a similar functionality.
Output of
containerd --version
:Any other relevant information:
The text was updated successfully, but these errors were encountered: