Update grammar to rely on indexPattern instead of identifier in join target #120494

idegtiarenko · 2025-01-21T07:28:33Z

This replaces identifier with indexPattern in joinTarget grammar.

This change is needed to make index selection consistent between FROM and [LOOKUP] JOIN commands:

Both should use the same quotes " (currently join relies on `)
Both should allow specifying indices with - without having to quote them (not possible with join at the moment)
Both should conform to allowed index names (there are number of differences today, for example it is possible to specify test? or +test in join even though it is not a valid index name.)

…target

elasticsearchmachine · 2025-01-21T07:28:57Z

Hi @idegtiarenko, I've created a changelog YAML for you.

elasticsearchmachine · 2025-01-21T09:50:12Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/190_lookup_join.yml

# Conflicts: # x-pack/plugin/esql/qa/security/src/javaRestTest/java/org/elasticsearch/xpack/esql/EsqlSecurityIT.java

ivancea

LGTM! I would wait for Costin or Alex to also check it, in case I missed something around indexPattern, as I'm relatively new in this part of ESQL 👀

costin

The issue description doesn't specify why the change was done.
I presume it's to allow proper syntax ("index") instead of (index) however there are side effects - changing to indexPattern (instead of indexString) allows for remote clusters to be specified and for index patterns (index*) to be specified, including date math.

My expectation is that patterns should be rejected (this doesn't seem to be the case), date math could be allowed if the syntax points to just an alias or index - the unit test doesn't seem to cover this case.

Thanks!

costin · 2025-01-22T03:51:48Z

...ugin/esql/qa/security/src/javaRestTest/java/org/elasticsearch/xpack/esql/EsqlSecurityIT.java

Add randomization to have the index specified with and without double quotes.

I would like to keep this test simple. All possible index name variations are covered in dedicated parser test.

costin · 2025-01-22T03:55:48Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java

@@ -2939,4 +2941,69 @@ public void testNamedFunctionArgumentWithUnsupportedNamedParameterTypes() {
            );
        }
    }
+
+    public void testValidJoinPattern() {


Using the index pattern makes sense for parser (vs grammar) validation however I would have expected for patterns to be rejected from the start instead of waiting to resolve them first.
That is LOOKUP JOIN foo works during parsing (but might fail at runtime if it's an alias pointing to multiple indices) however LOOKUP JOIN foo* fails at parsing since it's pattern (regardless of whether it gets resolved to 1 or more indices).

Added in 2b642b9

idegtiarenko · 2025-01-22T08:24:13Z

The issue description doesn't specify why the change was done.

I have added several examples highlighting the difference in the syntax

I presume it's to allow proper syntax ("index") instead of (index) however there are side effects - changing to indexPattern (instead of indexString) allows for remote clusters to be specified and for index patterns (index*) to be specified, including date math.

Remote clusters are handled in #120277. I would like to merge it after to ensure that works for both left and right patterns of the query string.

My expectation is that patterns should be rejected (this doesn't seem to be the case), date math could be allowed if the syntax points to just an alias or index - the unit test doesn't seem to cover this case.

Added in 2b642b9

# Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join.csv-spec

alex-spies

I'm nearly done reviewing. So far, this looks very good. I have a couple notes on test cases, but the general grammar change LGTM.

x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join.csv-spec

alex-spies · 2025-01-23T13:23:30Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join.csv-spec

Since we're also enabling quoting, could we please add test cases demonstrating the usage of " and """ quotes in LOOKUP JOIN?

I understand the quotes shouldn't be necessary right now, but someone could start using them and therefor we should have tests to guard against regression.

x-pack/plugin/esql/src/main/antlr/EsqlBaseLexer.g4

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java

alex-spies · 2025-01-23T15:56:07Z

The issue description doesn't specify why the change was done.

@costin the csv tests show that we currently require a lot of backtick quoting that's not required in (and is inconsistent with) FROM, mostly due to minuses in index names. I do believe this PR's change is very important to address this.

however there are side effects - changing to indexPattern (instead of indexString) allows for remote clusters to be specified and for index patterns (index*) to be specified, including date math.

This is true, and we'll address this with proper validation. CCS will be addressed once #120277 is update for the changes here, date math will get its own follow-up PR with validation, and wildcards are already addressed.

Also note that even in the absence of parse-time validation, all of these cases (wildcards, date math, and probably also ccs) should also be rejected due to Luigi's PR which disables aliases, specifically here where we ensure that the concrete index name is equal to the index pattern supplied by the user.

costin

LGTM

alex-spies

Nice, thanks @idegtiarenko !

I have mostly minor remarks and think this could be merged as-is. I'd add an integration test for " and """ usage though, as discussed.

We already identified a follow-up to validate against date math. Feel free to postpone additional enhancements for the next PR, and let's get this wrapped up and merged.

alex-spies · 2025-01-23T16:01:50Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java

+        var type = randomFrom("", "LOOKUP ");
+


The empty string looks wrong - FROM idx | JOIN other_idx ON field is not supported, but by the test below, we parse this as a lookup join o.O

Consider removing the plain join, but it's also fine as-is due to this only testing the parsing itself.

alex-spies · 2025-01-23T16:38:55Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/IdentifierGenerator.java

+            if (ESTestCase.randomBoolean()) {
+                index.append('*');
+            } else {
+                index.insert(ESTestCase.randomIntBetween(0, index.length() - 1), '*');


nit: there could be multiple wildcards in the middle as well.

alex-spies · 2025-01-23T16:39:43Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/IdentifierGenerator.java

+            } else {
+                index.insert(ESTestCase.randomIntBetween(0, index.length() - 1), '*');
+            }
+        } else if (canAdd(Features.DATE_MATH, features)) {


For date math, specifically the case with a pipe | character is interesting in ESQL.

But we can consider this in the follow-up PR for date math invalidation.

It is a bit tricky as using | requires mandatory quoting. I would like to followup on it separately.

alex-spies · 2025-01-23T16:40:40Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/IdentifierGenerator.java

+     * Identifier could be an index or alias. It might be hidden or remote or use a pattern.
+     * See @link <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params">valid index patterns</a>
+     */
+    public static String randomIndexPattern(Feature... features) {


This thing is really cool. Thanks for researching all the different ways that patterns can look like!

# Conflicts: # x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/191_lookup_join_text.yml

elasticsearchmachine · 2025-01-24T10:38:36Z

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 120494

… join target (#120784) * Update grammar to rely on indexPattern instead of identifier in join target (#120494) This replaces identifier with indexPattern in joinTarget grammar. This change is needed to make index selection consistent between FROM and [LOOKUP] JOIN commands: * Both should use the same quotes " (currently join relies on `) * Both should allow specifying indices with - without having to quote them (not possible with join at the moment) * Both should conform to allowed index names (there are number of differences today, for example it is possible to specify test? or +test in join even though it is not a valid index name.) (cherry picked from commit 9ffe3c8)

Update grammar to rely on indexPattern instead of identifier in join …

fa962c4

…target

idegtiarenko added >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) auto-backport Automatically create backport pull requests when merged :Analytics/ES|QL AKA ESQL v9.0.0 v8.18.0 labels Jan 21, 2025

Update docs/changelog/120494.yaml

6049006

idegtiarenko added 5 commits January 21, 2025 10:02

initial test

610b3bc

add join type

8675905

upd

2770dbb

upd

58b2f27

cleanup

bf9c556

idegtiarenko marked this pull request as ready for review January 21, 2025 09:49

fix quotes usage

a7a91b9

idegtiarenko requested a review from costin January 21, 2025 12:09

idegtiarenko assigned alex-spies and unassigned alex-spies Jan 21, 2025

idegtiarenko requested a review from alex-spies January 21, 2025 12:10

ivancea reviewed Jan 21, 2025

View reviewed changes

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/190_lookup_join.yml Outdated Show resolved Hide resolved

idegtiarenko added 2 commits January 21, 2025 14:11

inc lookup version

318fd1d

add pattern test cases

60444a8

idegtiarenko requested a review from ivancea January 21, 2025 13:18

Merge branch 'main' into es-10559

62d53f9

# Conflicts: # x-pack/plugin/esql/qa/security/src/javaRestTest/java/org/elasticsearch/xpack/esql/EsqlSecurityIT.java

ivancea approved these changes Jan 21, 2025

View reviewed changes

costin requested changes Jan 22, 2025

View reviewed changes

Merge branch 'main' into es-10559

a5f713f

prohibit * in join patterns

2b642b9

idegtiarenko requested a review from costin January 22, 2025 10:17

Merge branch 'main' into es-10559

52d98e6

idegtiarenko mentioned this pull request Jan 22, 2025

Disallow CCS with lookup join #120277

Merged

idegtiarenko added 5 commits January 23, 2025 13:15

random identifier generator

7b38bec

use random identifier generator

2cc2643

Merge branch 'main' into es-10559

9529874

# Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/resources/lookup-join.csv-spec

fix merge

e7d794d

add date math generation

62ba2fe

alex-spies reviewed Jan 23, 2025

View reviewed changes

alex-spies self-requested a review January 23, 2025 13:52

idegtiarenko added 4 commits January 23, 2025 14:55

rename

f562bc1

upd

9e7f125

Merge branch 'main' into es-10559

a9ecc24

fix new tests

4574fd1

add IT with quotes

856a6ec

costin approved these changes Jan 23, 2025

View reviewed changes

alex-spies approved these changes Jan 23, 2025

View reviewed changes

idegtiarenko added 2 commits January 24, 2025 08:05

Merge branch 'main' into es-10559

0a39bb2

add missing capability

c1b399d

idegtiarenko mentioned this pull request Jan 24, 2025

Add missing capability #120771

Merged

Merge branch 'main' into es-10559

1116eb7

# Conflicts: # x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/191_lookup_join_text.yml

idegtiarenko merged commit 9ffe3c8 into elastic:main Jan 24, 2025
16 checks passed

elasticsearchmachine added the backport pending label Jan 24, 2025

idegtiarenko mentioned this pull request Jan 24, 2025

[8.x] Update grammar to rely on indexPattern instead of identifier in join target #120784

Merged

idegtiarenko removed the backport pending label Jan 24, 2025

idegtiarenko deleted the es-10559 branch January 24, 2025 12:33

Update grammar to rely on indexPattern instead of identifier in join target #120494

Update grammar to rely on indexPattern instead of identifier in join target #120494

Uh oh!

Conversation

idegtiarenko commented Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 21, 2025

Uh oh!

elasticsearchmachine commented Jan 21, 2025

Uh oh!

Uh oh!

ivancea left a comment

Choose a reason for hiding this comment

Uh oh!

costin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

idegtiarenko commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-spies commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

costin left a comment

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alex-spies Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 24, 2025

💔 Backport failed

Uh oh!

Uh oh!

idegtiarenko commented Jan 21, 2025 •

edited

Loading

idegtiarenko commented Jan 22, 2025 •

edited

Loading

alex-spies commented Jan 23, 2025 •

edited

Loading

alex-spies Jan 23, 2025 •

edited

Loading