Encryption for REST catalog #13225

smaheshwar-pltr · 2025-06-03T21:08:13Z

This PR implements client-side support for REST catalog encryption. With it, clients interacting with a REST catalog can read and write encrypted data.

It is similar to #13066, that integrates encryption with the Hive catalog.

cc @rdblue @RussellSpitzer @ggershinsky

smaheshwar-pltr · 2025-06-03T21:22:42Z

spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestTableEncryption.java

+// TODO(smaheshwar): This test is taken from https://github.com/apache/iceberg/pull/13066, with the
+// exception of testCtas, but adapted for the REST catalog. When that merges, we can directly use
+// those tests for the REST catalog as well by  adding to the parameters method there, to have a
+// single test class for table encryption.


Highlighting this - this file is largely taken from #13066

smaheshwar-pltr · 2025-06-03T21:26:48Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+              : Integer.parseInt(dekLength);
+    }
+
+    // Force re-creation of encryptingFileIO and encryptionManager


See #13066 (comment)

github-actions · 2025-07-07T00:19:57Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

smaheshwar-pltr · 2025-07-07T12:37:19Z

(Bump to remove staleness)

github-actions · 2025-08-07T00:19:41Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

smaheshwar-pltr · 2025-08-07T13:06:57Z

(Bump to remove staleness)

github-actions · 2025-09-08T00:17:56Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

smaheshwar-pltr · 2025-09-11T17:35:59Z

(Bump to remove staleness)

smaheshwar-pltr · 2025-09-23T15:29:31Z

core/src/main/java/org/apache/iceberg/encryption/StandardEncryptionManager.java

+   * @param dataKeyLength length of data encryption key (16/24/32 bytes)
+   * @param kmsClient Client of KMS used to wrap/unwrap keys in envelope encryption
+   */
+  public StandardEncryptionManager(


Could deprecate the old constructor

smaheshwar-pltr · 2025-09-23T15:32:52Z

spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestTableEncryption.java

+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.TestTemplate;
+
+// TODO(smaheshwar-pltr): This test is taken from https://github.com/apache/iceberg/pull/13066, with


I added a CTAS test to the ones authored by @ggershinsky here. Perhaps there's a world in which this PR merges first in which case can undo this comment. There are also review comments on these tests in #13066 that are potentially relevant

smaheshwar-pltr · 2025-09-26T13:48:06Z

Given #7770 is merged, curious for thoughts on this PR.
REST integration sounds on the cards. Also happy for this PR to be superseded if other folks have worked on it.

cc @huaxingao @RussellSpitzer @ggershinsky @rdblue

RussellSpitzer · 2025-09-27T00:04:50Z

Could you elaborate on the api additions? I think it would help to have some more description on the general direction of this or

huaxingao · 2025-10-25T01:46:02Z

@smaheshwar-pltr Could you please resolve the conflicts?

ggershinsky · 2025-10-27T13:43:36Z

@huaxingao @smaheshwar-pltr Our team has a person who works on encryption with the REST catalog. If @smaheshwar-pltr does not object, we can follow up on this patch.

ggershinsky · 2025-10-27T13:50:30Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+    return encryptionManager;
+  }
+
+  private void encryptionPropsFromMetadata(TableMetadata metadata) {


Is this method applied on the TableMetadata that is fetched directly from the REST catalog, and not from the metadata.json file? Both are possible, but the former must override (and check) the latter, to protect against the key removal and other attacks.

Yes this method will always be applied on a metadata field of a LoadTableResponse object received directly from the REST catalog (its only usage within this class is as such, and you can check the constructor usages within RESTSessionCatalog to confirm that the metadata coming in from there is as such too.

Question (not sure if there have been discussions here or if your team have thoughts): we want the key ID to come from the REST catalog service directly for security reasons.

It's typical for REST catalogs to provide metadata that corresponds to the metadata file in storage and not modify it apart from that. Given this, would it be preferable to have this field returned within the LoadTableResponse itself, to encourage catalogs to track it explicitly?

The concrete proposal here might be: ENCRYPTION_TABLE_KEY and ENCRYPTION_DEK_LENGTH become properties on the LoadTableResponse's config (mentioned in the REST spec here).

@ggershinsky @huaxingao LMKWYT!

Well I see two scenarios when thinking about this:

metadata.json is something that both the server and the clients can read (although clients wouldn't need to, given they get the metadata with the LoadTableResponse )

metadata.json can only be accessed on the server side and clients are not given FS credentials (either vended or not) to reach it

For case (1) I totally agree, we can't rely on just metadata.json to store these encryption properties, and the catalog should store it separately too, and eventually populating (i.e. doing the override logic referred by @ggershinsky) the properties in the LoadTableResponse to be created.
For case (2) I'm not 100% sure, but still leaning toward the catalog taking on this responsibility.

Either way, for the client side there's not much we can do other than recommending clients to consider the metadata from LoadTableResponse only. The rest (no pun intended) is on the server side to be decided and will be implementation-specific. For this code snippet above, irrelevant IMHO.

Let me know your thoughts.

Afaik, HMS is not optimized for JSON storage. But maybe someone in the community will take on storing the full metadata object there, to improve table security. Having only the table properties is barely sufficient. I think we should recommend REST catalogs for encrypted tables.

It's not. But would storing a hash suffice? If so we could generate the hash of the whole JSON content and store it via an additional (Hive) table property. Then during table loading we can verify that the TableMetadata we just read in from a potentially untrusted storage (and yet metadata.json is not encrypted) is original or has been tampered with.

One aspect of having an encrypted metadata.json is when the table schema is also considered a sensitive piece of information. I haven't found this in the discussions but do you know if this has ever been considered @ggershinsky ?

would storing a hash suffice? If so we could generate the hash of the whole JSON content and store it via an additional (Hive) table property. Then during table loading we can verify that the TableMetadata we just read in from a potentially untrusted storage (and yet metadata.json is not encrypted) is original or has been tampered with.

I think it's a good idea

One aspect of having an encrypted metadata.json is when the table schema is also considered a sensitive piece of information. I haven't found this in the discussions but do you know if this has ever been considered

Not sure. Though, it should be possible to have a REST implementation that hides the metadata.json file from the storage.

I think it's a good idea

Sounds good, I can take this on and will produce a PR shortly.

Not sure. Though, it should be possible to have a REST implementation that hides the metadata.json file from the storage.

Yes, with REST that's true, I just meant it in a general sense, e.g. it's not currently possible to hide the schema of an encrypted table with Hive catalog. It may just be one more thing to note/document as a limitation of encryption wrt. Hive catalog - just merely wanted to highlight this though.

ggershinsky · 2025-10-27T14:00:05Z

Also, it would be good to refactor (if possible) a code common to this PR and to #13066 , so that other catalogs will be able to re-use it.

ggershinsky

Thanks @smaheshwar-pltr !

github-actions · 2026-01-09T00:19:06Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions · 2026-01-16T00:19:33Z

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

smaheshwar-pltr · 2026-01-17T10:31:17Z

Not stale - I have a PR to separately fix the failing test this PR flagged that I'm working on

smaheshwar-pltr · 2026-01-20T17:08:24Z

@singhpk234 @huaxingao, may I get this PR reopened please?

huaxingao · 2026-01-20T17:45:22Z

@smaheshwar-pltr I removed the stale label, but somehow the reopen button is dim for me. Can you reopen on your side?

huaxingao · 2026-01-20T17:45:42Z

reopen

smaheshwar-pltr · 2026-01-20T18:34:28Z

@huaxingao apologies, I think some force pushing I did messed things up - I believe things should be good now, may you check again if you can reopen now?

smaheshwar-pltr · 2026-01-21T19:37:52Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

   * @param mutationHeaderSupplier a supplier for additional HTTP headers to include in mutation
   *     requests (POST/DELETE)
   * @param fileIO the FileIO implementation for reading and writing table metadata and data files
+   * @param kmsClient the {@link KeyManagementClient} for encrypted tables


Pointing out that I'm changing the signature of this method. #14740 did the same recently, and I think #14465 that hasn't released yet so there shouldn't be problems with this

smaheshwar-pltr · 2026-01-21T19:40:37Z

spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java

+    // This tests that encryption keys are maintained on refreshing different metadata - if
+    // they are not, the table will be unreadable and this will fail.


(I've verified that this test fails without the encryption-key tracking changes described in #14752. In other words, if we instead always construct the encryption manager using the current metadata in the table operations class which is wrong, this test fails and is the only one to fail. I've also documented the nuance here)

smaheshwar-pltr · 2026-01-21T20:48:02Z

@huaxingao @singhpk234, I believe that all comments and discussions are addressed, this is ready for another review!

It would be great to get this in soon if possible, to prevent conflicts growing and possible breaks (#13225 (comment)).

szlta

+1 non-binding - thanks @smaheshwar-pltr, this looks good to me

github-actions bot added spark core labels Jun 3, 2025

smaheshwar-pltr commented Jun 3, 2025

View reviewed changes

smaheshwar-pltr marked this pull request as ready for review June 3, 2025 21:24

smaheshwar-pltr mentioned this pull request Jun 3, 2025

Encryption integration and test #13066

Merged

smaheshwar-pltr commented Jun 3, 2025

View reviewed changes

smaheshwar-pltr mentioned this pull request Jun 8, 2025

AWS: KeyManagementClient implementation that works with AWS KMS #13136

Merged

github-actions bot added the stale label Jul 7, 2025

github-actions bot removed the stale label Jul 8, 2025

github-actions bot added the stale label Aug 7, 2025

github-actions bot removed the stale label Aug 8, 2025

github-actions bot added the stale label Sep 8, 2025

github-actions bot removed the stale label Sep 12, 2025

smaheshwar-pltr force-pushed the rest-encrypt-on-main-cherry-pick branch from d007aa2 to 889cbbf Compare September 23, 2025 15:23

smaheshwar-pltr mentioned this pull request Sep 23, 2025

Encryption for REST catalog #13224

Closed

smaheshwar-pltr commented Sep 23, 2025

View reviewed changes

ggershinsky reviewed Oct 27, 2025

View reviewed changes

github-actions bot added the build label Oct 27, 2025

smaheshwar-pltr requested review from ggershinsky, huaxingao and singhpk234 December 5, 2025 19:47

Suppress test method length

9ed8d8e

ggershinsky approved these changes Dec 8, 2025

View reviewed changes

smaheshwar-pltr added 2 commits December 9, 2025 16:40

Merge branch 'main' into rest-encrypt-on-main-cherry-pick

2d18a6a

Spotless

745a959

singhpk234 mentioned this pull request Dec 29, 2025

Add Polaris blog about KMS apache/polaris#3331

Merged

6 tasks

github-actions bot added the stale label Jan 9, 2026

github-actions bot closed this Jan 16, 2026

huaxingao removed the stale label Jan 20, 2026

huaxingao reopened this Jan 20, 2026

smaheshwar-pltr added 3 commits January 21, 2026 19:21

Merge branch 'main' into rest-encrypt-on-main-cherry-pick

1106a26

Fix conflicts and update test

8ac5228

Undo 4.0 changes

73e0980

smaheshwar-pltr commented Jan 21, 2026

View reviewed changes

Fix checkstyle

a774a9c

smaheshwar-pltr added 2 commits January 21, 2026 21:00

empty

86abfab

Nit: Simplify adding encryption keys

1e42400

smaheshwar-pltr force-pushed the rest-encrypt-on-main-cherry-pick branch from 645c75f to 1e42400 Compare January 21, 2026 23:21

szlta approved these changes Jan 22, 2026

View reviewed changes

		// This tests that encryption keys are maintained on refreshing different metadata - if
		// they are not, the table will be unreadable and this will fail.

Encryption for REST catalog #13225

Are you sure you want to change the base?

Encryption for REST catalog #13225

Uh oh!

Conversation

smaheshwar-pltr commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

smaheshwar-pltr commented Jul 7, 2025

Uh oh!

github-actions bot commented Aug 7, 2025

Uh oh!

smaheshwar-pltr commented Aug 7, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

smaheshwar-pltr commented Sep 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smaheshwar-pltr commented Sep 26, 2025

Uh oh!

RussellSpitzer commented Sep 27, 2025

Uh oh!

huaxingao commented Oct 25, 2025

Uh oh!

ggershinsky commented Oct 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggershinsky commented Oct 27, 2025

Uh oh!

ggershinsky left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 9, 2026

Uh oh!

github-actions bot commented Jan 16, 2026

Uh oh!

smaheshwar-pltr commented Jan 17, 2026

Uh oh!

smaheshwar-pltr commented Jan 20, 2026

Uh oh!

huaxingao commented Jan 20, 2026

Uh oh!

huaxingao commented Jan 20, 2026

Uh oh!

smaheshwar-pltr commented Jan 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smaheshwar-pltr Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

smaheshwar-pltr commented Jun 3, 2025 •

edited

Loading

smaheshwar-pltr Jan 21, 2026 •

edited

Loading