Compulsorily close coordinator if task is stopped by the connect framework. #13756
Conversation
… stopping as during stopping the consumer will be closed and it will lead to partition being moved
|
@bryanck can you please review. |
|
Hey Pritam, Is there any update of the review status for this PR? |
|
Hi @bryanck can you please review this once. |
|
To add some color here, the referenced issues that this is supposed to fix are very critical and are blocking production workloads due to data loss. I pulled this branch down and ran a build - the built file for Kafka Connect says it's 1.7.0 in the kafka-connect/kafka-connect-runtime/build/distributions/iceberg-kafka-connect-runtime-1.7.0-SNAPSHOT. However I have this running in an MSK Connect environment that is using the DebeziumTransform SMT, which is only supported in 1.9.x (AFAIK). So I believe this is a 1.9.x version? I've been running it for about 24 hours now and I haven't seen any errors related to |
|
@kumarpritam863 Thanks for the contribution, and thanks for you patience on the review. |
|
@bryanck will this fix be part of |
Currently this is not planned for 1.10.x. |
|
We are also loosing data because of this issue, so I would also say this is rather critical for a Kafka Connect connector. |
|
@bryanck - We’re seeing the same behavior on our side and are looking forward to this PR being merged. |
|
@bryanck What would the process be to get this included in the 1.10.2 or 1.11.0 milestone? |
|
@Pionerd This was merged so it will go into 1.11. |
|
hey @bryanck , is there a release calendar for iceberg? when can we expect the version 1.11 with this fix to be available? |
While this PR handled the a bunch of issues in ICR mode:
To make the interface consistent we called the close method with context.assignment() and close method always checks if the 0th partition is being lost then only close the coordinator. But in task stopping scenario, just after closing a task the partition will move as after task closing the consumer is also closed, here is the reference kafka code:
@Override protected void close() { // FIXME Kafka needs to add a timeout parameter here for us to properly obey the timeout // passed in try { task.stop(); } catch (Throwable t) { log.warn("Could not stop task", t); } taskStopped = true; Utils.closeQuietly(consumer, "consumer"); Utils.closeQuietly(headerConverterPlugin, "header converter"); Utils.closeQuietly(keyConverterPlugin, "key converter"); Utils.closeQuietly(valueConverterPlugin, "value converter"); Utils.closeQuietly(pluginMetrics, "plugin metrics"); /* Setting partition count explicitly to 0 to handle the case, when the task fails, which would cause its consumer to leave the group. This would cause onPartitionsRevoked to be invoked in the rebalance listener, but not onPartitionsAssigned, so the metrics for the task (which are still available for failed tasks until they are explicitly revoked from the worker) would become inaccurate. */ sinkTaskMetricsGroup.recordPartitionCount(0); }To handle this scenario and permanently closing the coordinator, in this PR I am using empty partition list as marker to task being stopped as kafka connect never calls open or close with empty partitions.
Addresses : #13593, #13763, #13763