Skip to content

Kafka Connect: Add SMTs for Debezium and AWS DMS #11936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
bryanck merged 19 commits into from
Feb 25, 2025
Merged

Kafka Connect: Add SMTs for Debezium and AWS DMS #11936

bryanck merged 19 commits into from
Feb 25, 2025

Conversation

@ismailsimsek
Copy link
Contributor

@ismailsimsek ismailsimsek commented Jan 9, 2025 •
edited
Loading

resolves #10844
resolves #11914

Copied over kafka-connect-transforms code. no code changes made
applied code formatting
and updated build.gradle accordingly

cc: @bryanck could you please take look at this when you have chance?

ismailsimsek
ismailsimsek commented Jan 9, 2025
View reviewed changes
ismailsimsek
ismailsimsek commented Jan 9, 2025
View reviewed changes
@ismailsimsek
Copy link
Contributor Author

ismailsimsek commented Jan 10, 2025

@jbonofre @bryanck its ready for review

@Fokko Fokko requested a review from bryanck January 15, 2025 14:04
@ismailsimsek ismailsimsek force-pushed the kafka-smt-copy branch 2 times, most recently from 2e2c727 to 9e64c5b Compare January 15, 2025 16:45
danielcweeks
danielcweeks reviewed Jan 15, 2025
View reviewed changes
danielcweeks
danielcweeks reviewed Jan 15, 2025
View reviewed changes
@bryanck
Copy link
Contributor

bryanck commented Jan 16, 2025

Thanks @ismailsimsek for porting this over!

@ismailsimsek ismailsimsek force-pushed the kafka-smt-copy branch 2 times, most recently from 823cc83 to 34445e6 Compare January 16, 2025 17:56
ismailsimsek
ismailsimsek commented Jan 16, 2025
View reviewed changes
@braislchao braislchao mentioned this pull request Jan 17, 2025
Closed
danielcweeks
danielcweeks reviewed Jan 27, 2025
View reviewed changes
@liko9
Copy link
Contributor

liko9 commented Feb 15, 2025

Can someone please add this to the milestone for 1.9.0?

@Fokko Fokko added this to the Iceberg 1.9.0 milestone Feb 15, 2025
@ismailsimsek
Copy link
Contributor Author

ismailsimsek commented Feb 17, 2025

@Fokko @bryanck @danielcweeks @jbonofre could you please review?

Currently, i believe only open question is on LICENSE/NOTICE changes, not sure what to do on that part. should we merge and followup with the new PR for LICENSE/NOTICE updates?

@jbonofre
Copy link
Member

jbonofre commented Feb 17, 2025

@ismailsimsek let me do a pass and suggest update on LICENSE/NOTICE if needed.

@jbonofre
Copy link
Member

jbonofre commented Feb 18, 2025

I did a pass about LICENSE/NOTICE.

First, in kafka-connect-transforms, I see use of debezium package, but for specific code (not code copied from debezium), so it's OK.

As kafka-connect-transforms is used in kafka-connect-runtime distributions, I did a pass on the transitive dependencies.
Here's what I see in the PR:

  1. The versions in LICENSE doesn't match the one in the distributions (I suggest to do a rebase to fix that)
  2. bson should be in LICENSE. As bson comes from https://github.com/mongodb/mongo-java-driver and there's no NOTICE there, no need to update NOTICE in the distributions. That should be in this PR.
  3. detector-resources-support (from Google Opentelemetry) and exporter-metrics (from Google Opentelemetry) are not in LICENSE. I will check that (on main) as I think I fixed it already.

So, @ismailsimsek specifically to this PR, bson should be added in LICENSE. If you want I can create a commit in this PR for that.

@ismailsimsek
Copy link
Contributor Author

ismailsimsek commented Feb 18, 2025

I did a pass about LICENSE/NOTICE.

First, in kafka-connect-transforms, I see use of debezium package, but for specific code (not code copied from debezium), so it's OK.

As kafka-connect-transforms is used in kafka-connect-runtime distributions, I did a pass on the transitive dependencies. Here's what I see in the PR:

1. The versions in `LICENSE` doesn't match the one in the distributions (I suggest to do a rebase to fix that)

2. `bson` should be in `LICENSE`. As bson comes from https://github.com/mongodb/mongo-java-driver and there's no `NOTICE` there, no need to update `NOTICE` in the distributions. That should be in this PR.

3. `detector-resources-support` (from Google Opentelemetry) and `exporter-metrics` (from Google Opentelemetry) are not in `LICENSE`. I will check that (on `main`) as I think I fixed it already.

So, @ismailsimsek specifically to this PR, bson should be added in LICENSE. If you want I can create a commit in this PR for that.

Thanks for the review, @jbonofre, I appreciate your feedback. yep its ok for me, feel free to commit it directly to the PR, otherwise I can incorporate them soon.

bryanck and others added 15 commits February 18, 2025 17:20
@bryanck @ismailsimsek
Add SMT project with copy value SMT (#50)
754fdb3 
(cherry picked from commit 639b0d5b41b827d984aae04efe594315ec2b2b91)
@bryanck @ismailsimsek
Add SMT to tranform DMS messages for CDC (#56)
5241c60 
(cherry picked from commit 63cde8e7f6c12392c7741922d5e6ad807051f24a)
@bryanck @ismailsimsek
Update dependencies and SMT readme (#58)
f1f62b9 
(cherry picked from commit b9cd15de938e57bf178e1ccb443e481fde881224)
@bryanck @ismailsimsek
Style updates to match Iceberg project (#99)
4230b30 
(cherry picked from commit c17c6f734dac96975959e6165416396e4058332c)
@bryanck @ismailsimsek
Update to Iceberg 1.4.0-rc.2 (#101)
3801025 
(cherry picked from commit 03dcf40b484f40f62c551b9ddf5cefea93a3440a)
@bryanck @ismailsimsek
Add experimental Debezium SMT (#104)
7503d68 
(cherry picked from commit d0adaf9f961ceb89aaa408c03874788b3cf2c422)
@bryanck @ismailsimsek
Add key to Debezium CDC metadata (#107)
dffc911 
(cherry picked from commit 5812322e595cee663d920aedaed21998fffa9bdf)
@bryanck @ismailsimsek
Config to set target in Debezium SMT (#111)
2660a0f 
(cherry picked from commit bf82d607dc2b5e816c8b6f59bcbdc48281154e98)
@bryanck @ismailsimsek
Place CDC metadata in struct/map in SMTs (#116)
bd4fa6f 
(cherry picked from commit 89f533b2e689cbd1935c3bd1b82eea5e9dc0cd07)
@bryanck @ismailsimsek
Update readme for transforms (#158)
d003307 
(cherry picked from commit 92e4d984fe41c20faf68b1c36e6fd20759e0a19f)
@tabmatfournier @ismailsimsek
SMT for json parsing (apache#214)
2f07a6f 
* smt-nested-json-as-map

- parse json objects into Maps rather than Structs prior to handing to the iceberg connector, for users with unstructured json data.

(cherry picked from commit 303435aa794d8df1728f83ca5179e896b17ca4ff)
@tabmatfournier @ismailsimsek
Option to inject kafka metadata (apache#205)
71a4500 
* option-to-inject-kafka-metadata

- SMT to add Kafka metadata (topic, partition, offset, timestamp) to Struct and Map types

(cherry picked from commit 423b4a8b0f2e42f2dd7de315631e944c285dcb09)
@tabmatfournier @ismailsimsek
Non-flattening MongoDb Debezium SMT (apache#204)
9de3c65 
* matf-non-flattening-mongodb-debezium-smt

- adds debezium mongo SMT for converting BSON before/after into typed Struct before/after

(cherry picked from commit 21d741e53ce77547edbb5838f1b2b49db619be0c)
@ismailsimsek
Kafka Connect: Add SMTs for Debezium and AWS DMS from tabular kafka c…
190d0b1 
…onnect project
@ismailsimsek
Kafka Connect: Add SMTs documentation to Iceberg docs
dfc34c0
@ismailsimsek
Copy link
Contributor Author

ismailsimsek commented Feb 18, 2025

@jbonofre rebased and updated LICENSE file. please feel free commit/change if anything missing.

@jbonofre
Copy link
Member

jbonofre commented Feb 18, 2025

@ismailsimsek awesome ! thanks. I'm preparing a PR for your branch 😄

@ismailsimsek
Copy link
Contributor Author

ismailsimsek commented Feb 24, 2025

@jbonofre @danielcweeks @Fokko @ajantha-bhat could you please review? I believe all open points are addressed.

Note: We'll want to merge this without squashing.

It looks good! We'll want to merge this without squashing.

@bryanck
Copy link
Contributor

bryanck commented Feb 24, 2025

It is OK with me if we lose my history. Could you look into the failing tests though?

@jbonofre
Copy link
Member

jbonofre commented Feb 24, 2025

@ismailsimsek I created a PR to "fix" the LICENSE/NOTICE: ismailsimsek#201 Maybe worth to consider ?

@ismailsimsek
Update kafka-connect-runtime distributions LICENSE and NOTICE with ad…
2c8dbb3 
…dition of transforms

Update kafka-connect-runtime distributions LICENSE and NOTICE with addition of transforms
@ismailsimsek
Copy link
Contributor Author

ismailsimsek commented Feb 24, 2025 •
edited
Loading

@jbonofre merged it. and test are passing now

@bryanck
Copy link
Contributor

bryanck commented Feb 24, 2025

LGTM, I'll leave it open for one more day and see if there is any more feedback.

@bryanck bryanck merged commit 7d9e96f into apache:main Feb 25, 2025
43 checks passed
@bryanck
Copy link
Contributor

bryanck commented Feb 25, 2025

Thanks @ismailsimsek for porting this over!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bryanck bryanck bryanck approved these changes

@danielcweeks danielcweeks Awaiting requested review from danielcweeks

Assignees

No one assigned

Labels

build docs INFRA KAFKACONNECT

Projects

None yet

Milestone

Iceberg 1.9.0

Development

Successfully merging this pull request may close these issues.

Official iceberg kafka-connect is missing SMTs from original Databricks/Tabular repository Kafka Connect: Add SMTs for Debezium and AWS DMS

7 participants

@ismailsimsek @bryanck @liko9 @jbonofre @danielcweeks @Fokko @tabmatfournier