Kafka Connect: Handle no coordinator and data loss in ICR mode #12372

kumarpritam863 · 2025-02-21T16:16:25Z

Few Observations:

In the ICR mode "open" receives only the "newly added partitions", and "open" will "not be called" by connect framework if there are No New Partitions Assigned to the Task.
Similarly in case of "Close" the Task receives only the "removed partitions" but we blindly close the "Co-ordinator".
The coordinator is created only in case "open" is called but in case when a partition is revoked and no partition is added on the task then only close will be called with that revoked partition without any open call.

How this is leading to NO-Coordinator Scenario:

Consider the case when a partition other than partition ZERO is removed from the leader task and is assigned to some other task.
In this case a close call on leader will close the Co-ordinator but since this task will not get open call this will not lead to leader-election on this task.
As the other task which received the removed task has not received partition zero, leader election on that task will also not lead to that task being elected as leader.

Let's see this with the below example:

Initially we had one worker "W0" with two tasks "T0" and "T1" consuming from two partitions of one topic namely "P0" and "P1", so the initial configuration is:

W0 -> [{T0,P0}, {T1, P1}] -> this will elect "T0" as the co-ordinator as it has "P0"

Now another worker "W1" joins:-> this will lead to rebalancing on both tasks as well as topic partitions within those tasks.

Connect Framework will stop T1 on W0.
This will cause partition level rebalance and partition P1 will be assigned to T0.

State at this point of time:

W0 -> [{T0,[P0, P1]}]
W1 -> []

Now,

Connect FrameWork will start the T1 on W1.
This will again lead to a rebalance at the partition level.

Assume P1 is removed from T0:

Connect Framework will call "close" call on T0 with Partition P1.
This will close the Co-ordinator on T0.
An "open" call we be made by the Connect FrameWork on T1 with partition P1.
This will lead to leader-election on T1 but since it is assigned P1, T1 will not be elected as Co-ordinator.

Hence this leads to a No-Coordinator scenario.

Data Loss Scenario:

In Incremental Cooperative Rebalancing (ICR) mode, when rebalance happens, consumers do not stop consuming as their is no stop the world like in "Eager" mode of rebalancing.
In case a partition is removed from a task, Consumer co-ordinator calls "close(Collection()) of the sink Task. In this call since we are blindly dumping all the files, this will dump the records also for the partitions still retained by this task. Moreover close call will make the committer null and since we have not null check for commiter in the put(Collection) , this will silently ignore the records. Once we get an open(Collection) call from Kafka this will start the commiter without resetting the offsets which leads to data loss.

Document explaining both the scenario: https://docs.google.com/document/d/1okqGq1HXu2rDnq88wIlVDv0EmNFZYB1PhgwyAzWIKT8/edit?tab=t.0#heading=h.51qcys2ewbsa

…rrent design itself

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/ResourceType.java

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

bryanck · 2025-02-23T22:27:53Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

-      coordinatorThread.terminate();
-      coordinatorThread = null;
+  @Override
+  public void stop(ResourceType resourceType) {


Likewise, we can just call the type-specific methods directly instead.

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

bryanck · 2025-02-23T22:39:47Z

We should add some comments, and also tests if feasible. Also looks like there are some code formatting issues.

…rent_design_in_ICR_mode

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/Committer.java

bryanck · 2025-02-24T15:38:05Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java

-    committer = CommitterFactory.createCommitter(config);
-    committer.start(catalog, config, context);
+    // We should be starting co-ordinator only the list of partitions has the zeroth partition.
+    if(committer.isCoordinator(partitions)) {


We should move this logic into the committer implementation.

If we want to move this logic to the committer then we need to somehow pass the the partitions information and that would require modification to the commuter method signatures

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java

…handle backward compatibility

… owned offsets and this avoida extra work

…_data_loss_in_current_design_in_ICR_mode_test_latest Handling no coordinator and data loss in current design in icr mode test latest

bryanck · 2025-03-13T14:00:12Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CommitterFactory.java


 class CommitterFactory {
-  static Committer createCommitter(IcebergSinkConfig config) {
+  static Committer createCommitter() {


We should leave this as-is for now and address changes in the PR to support different committers.

bryanck · 2025-03-13T14:03:12Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java

  @Override
  public void start(Map<String, String> props) {
    this.config = new IcebergSinkConfig(props);
+    // Catalog and committer are global resources and do not depend on the topic partition;


I don't feel this comment is necessary

bryanck · 2025-03-13T14:06:54Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

+  private Collection<MemberDescription> membersWhenWorkerIsCoordinator;
+  private final AtomicBoolean isCommitterInitialized = new AtomicBoolean(false);
+
+  void initializeCommitter(Catalog catalog, IcebergSinkConfig config, SinkTaskContext context) {


This should be private.

bryanck · 2025-03-13T14:12:20Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

+  private final AtomicBoolean isCommitterInitialized = new AtomicBoolean(false);
+
+  void initializeCommitter(Catalog catalog, IcebergSinkConfig config, SinkTaskContext context) {
+    if (isCommitterInitialized.compareAndSet(false, true)) {


I feel checking for initialization is being overly conservative, given this would only happen in cases where Kafka Connect isn't following the sink task contract. We can add a precondition check instead, in methods that require initialization. Also, we shouldn't need an atomic as methods should be called from the main task thread.

This check is kind of required as the objects being created here are passed as reference to the worker and coordinator and every open call is changing that especially the KafkaClient. Also this is not a lock and this will be a synchronous call for all the task. This also prevents redundant assignment at every open call.

Also the atomic ds are very efficient and this is just a check and set and will happen only once in the lifetime of a task. Also all task are independent of each other so this will not block anything as this is not a lock just a compare and set.

I see, it might be a bit confusing calling committer start from the task open. What do you think of naming the new committer methods open and close instead, to align with the task API?

(BTW the atomic is fine, that's safer anyway, thanks)

Yeah, I was thinking that initially but did not made change. Should I rename the new methods as open and close for committer interface.

bryanck · 2025-03-13T14:36:07Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

+  private SinkTaskContext context;
+  private KafkaClientFactory clientFactory;
+  private Collection<MemberDescription> membersWhenWorkerIsCoordinator;
+  private final AtomicBoolean isCommitterInitialized = new AtomicBoolean(false);


We can name this initialized or isInitialized.

bryanck · 2025-03-13T14:36:46Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

+  private Collection<MemberDescription> membersWhenWorkerIsCoordinator;
+  private final AtomicBoolean isCommitterInitialized = new AtomicBoolean(false);
+
+  private void initializeCommitter(Catalog catalog, IcebergSinkConfig config, SinkTaskContext context) {


Let's name this just initialize

bryanck · 2025-03-13T14:58:59Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

-  public void start(Catalog catalog, IcebergSinkConfig config, SinkTaskContext context) {
-    KafkaClientFactory clientFactory = new KafkaClientFactory(config.kafkaProps());
-
+  public boolean hasLeaderPartition(Collection<TopicPartition> currentAssignedPartitions) {


This should be private

Also, nitpick, I feel we should remove isLeader and fold the logic in here, having the two methods with similar names is somewhat confusing.

I left that just for better readability of the code. I can make that change.

Also that method is being used in one of the tests, and did not wanted to make extra changes.

bryanck · 2025-03-13T15:00:25Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

    }
  }
+
+  public void startWorker() {


The following new methods should be private

…sLeaderPartition and isLeader

bryanck · 2025-03-14T15:52:40Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java

  public void start(Map<String, String> props) {
    this.config = new IcebergSinkConfig(props);
+    catalog = CatalogUtils.loadCatalog(config);
+    committer = CommitterFactory.createCommitter(config);


When assigning member variables, you should use the this. prefix.

Ohh, sure, missed this.

bryanck · 2025-03-14T15:54:33Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

+    if (isInitialized.compareAndSet(false, true)) {
+      this.icebergCatalog = catalog;
+      this.icebergSinkConfig = config;
+      this.sinkTaskContext = context;


We should revert the variable names to what they were.

bryanck · 2025-03-14T16:14:35Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java

+  }
+
+  private void startCoordinator() {
+    LOG.info("Task elected leader, starting commit coordinator");


We should protect against multiple coordinator threads here

bryanck · 2025-03-14T17:10:14Z

Thanks @kumarpritam863 for the research on this and the contribution!

kumarpritam863 · 2025-03-14T17:13:05Z

Thanks @bryanck for all the support, insights and reviews.

mblesak · 2025-07-22T11:32:31Z

@kumarpritam863, can you please check: #13593
It is issue related to the changes in this Pull Request.
Method CommitterImpl.stop() has been marked as deprecated but I guess we need to distinguish between close() and stop() where IcebergSinkTask closes the assets like Catalog.

kumarpritam863 · 2025-08-05T17:00:37Z

@mblesak this is already handled in this PR.

Pritam Kumar Mishra added 2 commits October 10, 2024 15:45

handling no_coordinator scenario and the data loss scenario in the cu…

60dc6fa

…rrent design itself

open admin always in try for better resource managemennt

6bad004

github-actions bot added the KAFKACONNECT label Feb 21, 2025

bryanck reviewed Feb 23, 2025

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java Outdated Show resolved Hide resolved

bryanck reviewed Feb 23, 2025

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/ResourceType.java Outdated Show resolved Hide resolved

bryanck reviewed Feb 23, 2025

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java Outdated Show resolved Hide resolved

bryanck reviewed Feb 23, 2025

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java Outdated Show resolved Hide resolved

Pritam Kumar Mishra added 5 commits February 24, 2025 08:34

Merge branch 'main' into handling_no_coordinator_and_data_loss_in_cur…

b2e6799

…rent_design_in_ICR_mode

resolved some of the PR comments and code formatting

bd35bb1

added test case for no-coordinator scenario

826a698

added comments

85c25ad

fixed comments

6c25311

bryanck changed the title ~~Handling no coordinator and data loss in current design in icr mode~~ Handling no coordinator and data loss in ICR mode Feb 24, 2025

bryanck reviewed Feb 24, 2025

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/Committer.java Outdated Show resolved Hide resolved

bryanck reviewed Feb 24, 2025

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/Committer.java Outdated Show resolved Hide resolved

bryanck reviewed Feb 24, 2025

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java Outdated Show resolved Hide resolved

bryanck reviewed Feb 24, 2025

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java Outdated Show resolved Hide resolved

bryanck reviewed Feb 24, 2025

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkTask.java Outdated Show resolved Hide resolved

Pritam Kumar Mishra added 9 commits March 3, 2025 22:28

introduced two default methods without changing the exist methods to …

69ee942

…handle backward compatibility

introduced two default methods without changing the exist methods to …

c3f76a1

…handle backward compatibility

added comments and java doc to the committer interface

e33b92c

refactored

6a5995c

moved offset sync call to close as offset sync is only needed for the…

688462c

… owned offsets and this avoida extra work

refactored offset syncing part

9ca2b73

testing

71bbfd3

refactoring spaces

5368850

reverted exception handling made for testing purpose

c26bd70

Merge pull request #3 from kumarpritam863/handling_no_coordinator_and…

e1d7c30

…_data_loss_in_current_design_in_ICR_mode_test_latest Handling no coordinator and data loss in current design in icr mode test latest

bryanck changed the title ~~Handling no coordinator and data loss in ICR mode~~ Kafka Connect: Handling no coordinator and data loss in ICR mode Mar 13, 2025

bryanck reviewed Mar 13, 2025

View reviewed changes

resolved pr comments

1b6311c

bryanck reviewed Mar 13, 2025

View reviewed changes

resolved pr comments and applied spotless

c5abc58

bryanck reviewed Mar 13, 2025

View reviewed changes

Pritam Kumar Mishra added 3 commits March 13, 2025 20:33

made class specific methods private in committer impl

733bafa

changed isLeader to containsFirstPartition to avaoid confusion btw ha…

903ece0

…sLeaderPartition and isLeader

resolved checkstyle issues

7658be4

bryanck approved these changes Mar 14, 2025

View reviewed changes

added spotless fix

1548e24

bryanck reviewed Mar 14, 2025

View reviewed changes

reverted member variables names and added this to assigning variables

845f3cc

bryanck reviewed Mar 14, 2025

View reviewed changes

added extra safeguard to avoid duplicate coordinator

2667a56

bryanck merged commit 51abab1 into apache:main Mar 14, 2025
14 checks passed

bryanck changed the title ~~Kafka Connect: Handling no coordinator and data loss in ICR mode~~ Kafka Connect: Handle no coordinator and data loss in ICR mode Mar 14, 2025

kumarpritam863 mentioned this pull request Aug 7, 2025

Compulsorily close coordinator if task is stopped by the connect framework. #13756

Merged

fenil25 mentioned this pull request Oct 22, 2025

Kafka Connect: Don't check that consumer group is stable for coordinator leader election #14395

Merged

Kafka Connect: Handle no coordinator and data loss in ICR mode #12372

Kafka Connect: Handle no coordinator and data loss in ICR mode #12372

Uh oh!

Conversation

kumarpritam863 commented Feb 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bryanck commented Feb 23, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryanck Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryanck commented Mar 14, 2025

Uh oh!

Uh oh!

kumarpritam863 commented Mar 14, 2025

Uh oh!

mblesak commented Jul 22, 2025

Uh oh!

kumarpritam863 commented Aug 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

bryanck Mar 13, 2025 •

edited

Loading