Keep the watch action working all the time

[vklonghml]

Edited by @mbohlool:
This question lead to an action item to add a retry mechanism to watch class. It should be controlled by a flag and will result in keeping the watch open all the time.

Original post:

below is how i use the client-python in list.py:

config.load_kube_config()
v1 = client.CoreV1Api()
w = watch.Watch()
for event in w.stream(v1.list_persistent_volume_claim_for_all_namespaces):
print("Event: %s %s" % (event['type'], event['object'].metadata.name))

when i run the script with command "python list.py", it will show the event normally,
however i will exited automatic in several minitues.

does anybody konws how could i keep this watch action working all the time?

[mbohlool]

Do you get any error? did you try to set timeout_seconds parameter?

[webwurst]

I see this happening with one certain cluster, but it works fine with another. So this might be some timeout on somthing in between, like load-balancer or so. But it would be cool if the watch could just do a retry in case the connection is lost.

[vklonghml]

@mbohlool i don't set the timeout_seconds parameter, use the default value, but how can i keep the watch work all the time, like the @webwurst said.

[ivar-lazzaro]

+1.
I have several threads using the watcher to monitor different types of K8S resources, and I see their connection close every 30-35 min[0]. My application then resumes the watch but when that happens all the items for a given type are returned, which is undesirable. Not sure what is causing this 30 min timeout, the API server runs in a pod on the same host as my watchers do.

Is there a way to store a watch "version number" every time I receive an event that can be used whenever I resume the stream, so that I only get events subsequent to that point?

[0] I can tell looking at the list response "closed" attribute.

[mbohlool]

look at resource_version parameter. the returned object's metadata should also have resource_version. pass resource_version of the last received object to the new call.

[ivar-lazzaro]

Thanks for the answer.
I'm not sure how to use resource_version without doing a full list for that object type first. When the streaming process is just started it makes sense, but I want to avoid a full list call every time the connection breaks. When I get an event from the watch.stream function I get something like this:

{'raw_object': {u'status': {u'phase': u'Active'}, u'kind': u'Namespace', u'spec': {u'finalizers': [u'kubernetes']}, u'apiVersion': u'v1', u'metadata': {u'name': u'demo', u'resourceVersion': u'921', u'creationTimestamp': u'2017-04-28T22:23:54Z', u'selfLink': u'/api/v1/namespacesdemo', u'uid': u'5d5cd386-2c61-11e7-9d39-84b261c2790e'}}, u'object': {u'status': {u'phase': u'Active'}, u'kind': u'Namespace', u'spec': {u'finalizers': [u'kubernetes']}, u'apiVersion': u'v1', u'metadata': {u'name': u'demo', u'resourceVersion': u'921', u'creationTimestamp': u'2017-04-28T22:23:54Z', u'selfLink': u'/api/v1/namespacesdemo', u'uid': u'5d5cd386-2c61-11e7-9d39-84b261c2790e'}}, u'type': u'ADDED'}

The only resourceVersion I see here is the one concerning the specific k8s object. Should I set that value as the resource_version argument?

[mbohlool]

every time you get an event, store event['object'].metadata.resource_version into a variable (let say last_seen_version) and when you want to reconnect, pass it like this w.stream(v1.list_namespace, resource_version=last_seen_version)