JDBC catalog fix namespaceExists check #8340

ismailsimsek · 2023-08-16T20:42:31Z

resolves #8321

Currently checking testDb and test_b with sql LIKE clause will return true for both even when second namespace is not exists. _,% characters in namespace causing issue, which should be escaped.

Extended Query to Escape {\,_,%} special characters with LIKE {pattern} ESCAPE '\' clause

The ESCAPE statement is supported by all major Databases:

dramaticlly

thank you @ismailsimsek for the fix!

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java

dramaticlly

LGTM!

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java

dramaticlly · 2023-08-18T16:44:37Z

@rdblue @nastra @jackye1995 can you help review the change for JDBC bug fix #8321 ?

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java

amogh-jahagirdar

Thanks for this PR @ismailsimsek and sorry for the delay on reviewing this, it looks like the issue was reported again #8832 . I think we would want to get rid of the pattern based "LIKE" search though. I'd expect getNamespace should always be an exact lookup based on the specified name. What are your thoughts? @ismailsimsek @dramaticlly @nastra

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java

rdblue · 2023-10-20T00:18:04Z

@ismailsimsek it looks like this has unnecessary changes and refactors quite a bit. Can you revert the unnecessary changes and add to the PR description how this solves the problem? Thank you!

ismailsimsek · 2023-10-20T10:01:40Z

@rdblue @dramaticlly resolved the issue with pure sql, and rolled back previous changes. could you please review it again?

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java

rdblue · 2023-10-20T15:27:07Z

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java

+    assertThat(catalog.namespaceExists(Namespace.of("testDb."))).isFalse();
+    assertThat(catalog.namespaceExists(Namespace.of("testDb.ns"))).isFalse();
+    assertThat(catalog.namespaceExists(Namespace.of("testDb.ns_"))).isFalse();
+    assertThat(catalog.namespaceExists(Namespace.of("testDb%"))).isFalse();


For these false cases, can you run them twice? Once like this and once with a namespace that would match if the special character were not escaped.

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java

rdblue · 2023-10-20T15:29:04Z

Thanks, @ismailsimsek! Looks a lot better now. I like the new fix using ESCAPE. I think we just need to have more test cases now.

dramaticlly · 2023-10-20T17:36:36Z

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java

+          + " ( "
          + TABLE_NAMESPACE
-          + " LIKE ? LIMIT 1";
+          + " = ? OR "


thank you for separate out for both equality check and prefix check with escape.

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java

rdblue · 2023-10-28T20:18:25Z

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java

+    assertThat(catalog.namespaceExists(testNamespace)).isTrue();
+    assertThat(catalog.namespaceExists(Namespace.of("test.Db", "ns1"))).isTrue();
+    // TODO FIX handle Dot (its namespace SEPERATOR)? in the namespace levels. currently its
+    // accepted and threaded as a level


I'm not sure what problem you're referring to, but in general Iceberg avoids needing to escape the . character by mandating that ambiguous names are not allowed. That is, you can either have ["a", "b"] -> "a.b" or ["a.b"] -> "a.b" but you can't have both objects since they have a conflicting name ("a.b"). This is how many places work, including column indexing, and no one ever has a problem.

thank you @rdblue , I didn't know this as well. Is this mandate come from implementation of createNamespace() method? Trying to understand how it was enforced

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java

rdblue

Looks good other than the test cases for . that I think should be removed.

…haracter

ismailsimsek · 2023-11-30T14:56:32Z

@nastra @Fokko if you have time, could you please review it too? it should be ready.

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java

amogh-jahagirdar

Just some nits., thanks @ismailsimsek for following up on this. From a testing perspective, it looks like you're covering all the cases so thanks for adding all those.

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java

dramaticlly · 2023-12-11T19:13:46Z

@rdblue @amogh-jahagirdar can you take another look? I think it's awesome if we can merge this fix.

amogh-jahagirdar

Thanks for the patience and follow through on this PR @ismailsimsek , from my side everything looks good. I'll hold if @rdblue has any comments before merging

amogh-jahagirdar · 2023-12-13T18:40:09Z

Merging, thanks again for this fix @ismailsimsek and your follow-through. Thanks for reviews @dramaticlly @nastra @rdblue

…he#8340) * JDBC catalog fix namespaceExists check * Improve testCreateNamespace test * Improve testCreateNamespace test * Improve testCreateNamespace test * Improve testCreateNamespace test * Improve testCreateNamespace test * Improve testCreateNamespace test * Split tests to testCreateNamespaceWithSpecialCharacter * Organize and split testCreateNamespace tests * Removed test with `.`, and extended testCreateNamespaceWithBackslashCharacter * Add review improvements (cherry picked from commit 11608e1)

ismailsimsek · 2023-12-13T20:26:43Z

Thank you all for reviewing it, @amogh-jahagirdar should we also merge it to 1.4.x branch, for next release?
created PR for it if makes sense #9291

… (#9291)

…he#8340) * JDBC catalog fix namespaceExists check * Improve testCreateNamespace test * Improve testCreateNamespace test * Improve testCreateNamespace test * Improve testCreateNamespace test * Improve testCreateNamespace test * Improve testCreateNamespace test * Split tests to testCreateNamespaceWithSpecialCharacter * Organize and split testCreateNamespace tests * Removed test with `.`, and extended testCreateNamespaceWithBackslashCharacter * Add review improvements

github-actions bot added the core label Aug 16, 2023

ismailsimsek force-pushed the fix_jdbc_ctalog_namespace_exists branch 2 times, most recently from 4e19e0b to c2dedf0 Compare August 16, 2023 21:07

dramaticlly reviewed Aug 16, 2023

View reviewed changes

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java Outdated Show resolved Hide resolved

dramaticlly approved these changes Aug 17, 2023

View reviewed changes

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java Outdated Show resolved Hide resolved

ismailsimsek commented Aug 18, 2023

View reviewed changes

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java Outdated Show resolved Hide resolved

amogh-jahagirdar mentioned this pull request Oct 16, 2023

JDBC: JDBC Catalog should do exact namespace search for get namespace queries #8833

Closed

amogh-jahagirdar reviewed Oct 16, 2023

View reviewed changes

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 20, 2023

View reviewed changes

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java Outdated Show resolved Hide resolved

JDBC catalog fix namespaceExists check

2046a40

ismailsimsek force-pushed the fix_jdbc_ctalog_namespace_exists branch from bfbd41e to 2046a40 Compare October 20, 2023 09:50

ismailsimsek requested review from dramaticlly and rdblue October 20, 2023 09:59

rdblue reviewed Oct 20, 2023

View reviewed changes

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 20, 2023

View reviewed changes

core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java Show resolved Hide resolved

Improve testCreateNamespace test

a012950

dramaticlly reviewed Oct 20, 2023

View reviewed changes

ismailsimsek added 5 commits October 21, 2023 09:17

Improve testCreateNamespace test

4aa7289

Improve testCreateNamespace test

cfc874c

Improve testCreateNamespace test

ec6d062

Improve testCreateNamespace test

14c2a5e

Improve testCreateNamespace test

6508e2b

ismailsimsek requested a review from rdblue October 21, 2023 08:36

ismailsimsek added 2 commits October 25, 2023 09:08

Split tests to testCreateNamespaceWithSpecialCharacter

0718c83

Organize and split testCreateNamespace tests

05d6d84

ismailsimsek commented Oct 25, 2023

View reviewed changes

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 28, 2023

View reviewed changes

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java Show resolved Hide resolved

rdblue requested changes Oct 28, 2023

View reviewed changes

Temoved test with ., and extended testCreateNamespaceWithBackslashC…

ec812f9

…haracter

ismailsimsek requested a review from rdblue October 28, 2023 20:30

ismailsimsek requested a review from amogh-jahagirdar November 30, 2023 11:28

nastra reviewed Dec 7, 2023

View reviewed changes

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java Outdated Show resolved Hide resolved

amogh-jahagirdar reviewed Dec 7, 2023

View reviewed changes

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java Outdated Show resolved Hide resolved

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java Outdated Show resolved Hide resolved

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java Show resolved Hide resolved

Add review improvements

61dcb2e

amogh-jahagirdar approved these changes Dec 12, 2023

View reviewed changes

amogh-jahagirdar merged commit 11608e1 into apache:main Dec 13, 2023

amogh-jahagirdar mentioned this pull request Dec 13, 2023

[JdbcCatalog] Issue with Namespace Exists #8832

Closed

nastra pushed a commit that referenced this pull request Dec 19, 2023

JDBC Catalog: Fix namespaceExists check with special characters (#8340)…

793f815

… (#9291)

nastra mentioned this pull request Apr 11, 2024

UncheckedSQLException: Failed to execute exists query: SELECT table_namespace FROM iceberg_tables WHERE catalog_name = ? AND (table_namespace = ? OR table_namespace LIKE ? ESCAPE '\') LIMIT 1 #10056

Closed

JDBC catalog fix namespaceExists check #8340

JDBC catalog fix namespaceExists check #8340

Uh oh!

Conversation

ismailsimsek commented Aug 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dramaticlly left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dramaticlly left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dramaticlly commented Aug 18, 2023

Uh oh!

Uh oh!

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rdblue commented Oct 20, 2023

Uh oh!

ismailsimsek commented Oct 20, 2023

Uh oh!

Uh oh!

rdblue Oct 20, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rdblue commented Oct 20, 2023

Uh oh!

dramaticlly Oct 20, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rdblue Oct 28, 2023

Choose a reason for hiding this comment

Uh oh!

dramaticlly Oct 31, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rdblue left a comment

Choose a reason for hiding this comment

Uh oh!

ismailsimsek commented Nov 30, 2023

Uh oh!

Uh oh!

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dramaticlly commented Dec 11, 2023

Uh oh!

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Uh oh!

amogh-jahagirdar commented Dec 13, 2023

Uh oh!

ismailsimsek commented Dec 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ismailsimsek commented Aug 16, 2023 •

edited

Loading