ESQL: Change the order of the optimization rules #124335

astefan · 2025-03-07T14:44:50Z

In a query like INLINESTATS max_lang = MAX(languages) BY y = gender the "rename" of gender as y was replaced by an eval y = gender and then integrated back as a "projection" inside the aggregate itself. ReplaceAggregateNestedExpressionWithEval and CombineProjections seem to work one against the other. The former adds the eval while the latter eliminates it and puts it back in the groupings of the aggregation.

I have changed the order in which ReplaceOrderByExpressionWithEval and PropagateInlineEvals rules were called in the logical optimizer so that the eval mentioned above is not integrated back in the aggregation.

elasticsearchmachine · 2025-03-07T14:45:30Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-03-07T14:45:31Z

Hi @astefan, I've created a changelog YAML for you.

astefan · 2025-03-07T14:49:00Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/inlinestats.csv-spec

@@ -695,3 +695,115 @@ emp_no:integer | languages:integer | gender:keyword | max_lang:integer | y:keywo
         10012 |                 5 | null           | 5                | null
         10014 |                 5 | null           | 5                | null
 ;
+


Some of the following tests fail with the same error. This is likely due to something that @alex-spies caught in a previous review, meaning too much is sent to the data nodes for processing, instead of being done on the coordinator. This is also tied to scenarios where multiple inlinestats commands are used in a query.

astefan · 2025-03-07T14:49:15Z

...src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PropagateInlineEvals.java

@@ -34,7 +34,6 @@ protected LogicalPlan rule(InlineJoin plan) {
        // check if there's any grouping that uses a reference on the right side


Only a small cleanup here.

…ticsearch into inlinestats_pickup3

…inlinestats_pickup3

costin

Good catch.
To double check my understanding : the eval being integrated as a project is fine (since the eval is just an alias) however in the case of inline the eval is needed stand alone to be propagated on the left side so the join works by returning the grouped column with the right alias.

Did I miss anything?
A logical optimization test might be useful that calls the two rules (three) rules one after the other to showcase the side effect.

Thanks!

alex-spies

I think this is nice and should be merged for the added test coverage alone.

However, did you maybe consider a solution where we make ReplaceAggregateNestedExpressionWithEval aware of whether it is applied inside the right hand side of InlineJoin? The root problem seems to me that this rule creates an Eval that should be applied not on the the right hand side, but to the left hand side of an InlineJoin - and we require a whole additional optimizer rule, namely PropagateInlineEvals applied in the correct order without runs of ReplaceAliasingEvalWithProject resp. CombineProjections for this to work - which I find a little brittle.

...src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PropagateInlineEvals.java

alex-spies · 2025-03-11T11:09:30Z

...src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PropagateInlineEvals.java

+            if (groupingRefs.isEmpty()) {
+                return p;
+            }
+
            // find their declaration and remove it
            // TODO: this doesn't take into account aliasing


This comment may be stale now.

...src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PropagateInlineEvals.java

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizer.java

alex-spies · 2025-03-11T15:37:42Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/inlinestats.csv-spec

+FROM employees
+| KEEP emp_no, languages, gender, first_name
+| RENAME first_name as f
+| INLINESTATS max_lang = MAX(languages) BY y = gender, l = languages, f = left(f, 1)


Subtle semantic question:

The implementation assumes that expressions in the BY clause take effect in the main branch of the join as well. I.e. INLINESTATS ... BY ..., f = left(f, 1) means that the attribute f in the left hand side gets overwritten before being aggregated by/joined on.

But there's a different way to understand this, and it's actually useful: INLINESTATS ... BY ..., f = left(f, 1) could mean that f is overwritten only on the right hand side where the aggregation happens, and then is matched up with the left hand side.

I don't think we want the latter to be the semantics here. But that could be useful in case that e.g. the left hand side has a counter field that indicates the current row (or similar) - then INLINESTATS ... BY counter = counter + 1 would actually shift the stats by 1 row.

…inlinestats_pickup3

astefan · 2025-03-12T14:10:55Z

To double check my understanding : the eval being integrated as a project is fine (since the eval is just an alias) however in the case of inline the eval is needed stand alone to be propagated on the left side so the join works by returning the grouped column with the right alias.

Yes.

However, did you maybe consider a solution where we make ReplaceAggregateNestedExpressionWithEval aware of whether it is applied inside the right hand side of InlineJoin? The root problem seems to me that this rule creates an Eval that should be applied not on the the right hand side, but to the left hand side of an InlineJoin - and we require a whole additional optimizer rule, namely PropagateInlineEvals applied in the correct order without runs of ReplaceAliasingEvalWithProject resp. CombineProjections for this to work - which I find a little brittle.

I did consider another solution, but I discarded it because it didn't feel the right approach.
My first solution to this bug was to change PropagateInlineEval to also consider Projects like the ones created by ReplaceAliasingEvalWithProject for an eval that comes from ReplaceAggregateNestedExpressionWithEval. This is for grouping by expressions, ie inlinestats ... by y = gender. But this rule is about evals in any form, extending it to Project even if this type of eval was only an aliasing felt wrong. Testing more complicated queries that perform various forms of aliasing in the inlinestats ... by .... groupings part seem to support the current chosen solution in the PR.

alex-spies · 2025-03-12T15:13:30Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/inlinestats.csv-spec

I just realized that it's probably a good idea to add tests for INLINESTATS ... BY x = bucket(...) and INLINESTATS ... BY x = CATEGORIZE(...). I think the latter cannot work because the join key in BY x = CATEGORIZE(...) is computed during the aggregation, whereas INLINESTATS requires the join key to be present before that.

Cc @jan-elastic , I think we'll have to start out with a limitation where INLINESTATS can't use CATEGORIZE, at least at first; to enable this, I think we'd somehow have to grab the categorizer from the first phase of the query (which computes the STATS) and make it available to the second phase of the query (which performs the joining with every row we see).

FWIW, I can live with the fact that the first version of INLINESTATS doesn't work with CATEGORIZE.

Just open a GitHub issue for that and it can be resolved later.

I opened #124717

alex-spies · 2025-03-12T15:19:02Z

I did consider another solution, but I discarded it because it didn't feel the right approach.
My first solution to this bug was to change PropagateInlineEval to also consider Projects like the ones created by ReplaceAliasingEvalWithProject for an eval that comes from ReplaceAggregateNestedExpressionWithEval. This is for grouping by expressions, ie inlinestats ... by y = gender. But this rule is about evals in any form, extending it to Project even if this type of eval was only an aliasing felt wrong. Testing more complicated queries that perform various forms of aliasing in the inlinestats ... by .... groupings part seem to support the current chosen solution in the PR.

This is not the alternative I'm proposing. What I observed is that PropagateInlineEvals just fixes a mistake made by ReplaceAggregateNestedExpressionWithEval: the latter is not aware of InlineJoin and thus adds its extra EVAL for an expression like INLINESTATS ... BY y = foo(...) to the wrong side of the join. Then PropagateInlineEvals scoops that EVAL up and throws it onto the correct side, but has to be more defensive and needs to carefully pick apart the EVAL that we could've known doesn't belong there in the first place.

Or to put it differently and bluntly: I think PropagateInlineEvals shouldn't exist, rather we should fix ReplaceAggregateNestedExpressionWithEval.

But that blows up the scope of this PR. I'll make a note to revisit this later - also, the general shape of the code for INLINESTATS may change over the course of the next weeks, so it's not worth pursuing this now.

alex-spies

LGTM. Thanks for the iterations, Andrei!

elasticsearchmachine · 2025-03-13T05:46:55Z

💚 Backport successful

Status	Branch	Result
✅	9.0

Change the order of the optimization rules

9828afc

astefan added >bug :Analytics/ES|QL AKA ESQL auto-backport Automatically create backport pull requests when merged v9.0.1 v9.1.0 labels Mar 7, 2025

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Mar 7, 2025

Update docs/changelog/124335.yaml

8843569

astefan commented Mar 7, 2025

View reviewed changes

astefan requested review from alex-spies and costin March 7, 2025 14:49

elasticsearchmachine and others added 6 commits March 7, 2025 14:52

[CI] Auto commit changes from spotless

a2c33fa

Fix import

18ebafe

Merge branch 'inlinestats_pickup3' of https://github.com/astefan/elas…

8beb022

…ticsearch into inlinestats_pickup3

More tests

26f2349

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

68c31b7

…inlinestats_pickup3

[CI] Auto commit changes from spotless

16311c2

costin approved these changes Mar 10, 2025

View reviewed changes

alex-spies reviewed Mar 11, 2025

View reviewed changes

astefan added 2 commits March 12, 2025 16:07

Address reviews

a036535

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

89e76ef

…inlinestats_pickup3

update

eddde2d

alex-spies mentioned this pull request Mar 12, 2025

ES|QL INLINESTATS after STATS optimized incorrectly #124654

Closed

alex-spies reviewed Mar 12, 2025

View reviewed changes

Merge branch 'main' into inlinestats_pickup3

dc8c35d

alex-spies approved these changes Mar 12, 2025

View reviewed changes

Merge branch 'main' into inlinestats_pickup3

ef8303c

astefan merged commit c48f9a9 into elastic:main Mar 13, 2025
17 checks passed

astefan deleted the inlinestats_pickup3 branch March 13, 2025 05:46

astefan mentioned this pull request Mar 13, 2025

[9.0] ESQL: Change the order of the optimization rules (#124335) #124694

Merged

astefan added a commit to astefan/elasticsearch that referenced this pull request Mar 13, 2025

ESQL: Change the order of the optimization rules (elastic#124335)

2e8d983

elasticsearchmachine pushed a commit that referenced this pull request Mar 13, 2025

ESQL: Change the order of the optimization rules (#124335) (#124694)

d226608

alex-spies mentioned this pull request Mar 13, 2025

ESQL: Enable multiple INLINESTATS + remaining tests #124715

Open

7 tasks

albertzaharovits pushed a commit to albertzaharovits/elasticsearch that referenced this pull request Mar 13, 2025

ESQL: Change the order of the optimization rules (elastic#124335)

2c0c258

alex-spies mentioned this pull request Mar 13, 2025

ESQL: Simplify PropagateInlineEvals #124754

Open

jfreden pushed a commit to jfreden/elasticsearch that referenced this pull request Mar 13, 2025

ESQL: Change the order of the optimization rules (elastic#124335)

d0f925e

costin assigned alex-spies Mar 14, 2025

costin removed v9.0.1 auto-backport Automatically create backport pull requests when merged labels Mar 14, 2025

alex-spies added the v9.0.1 label Mar 17, 2025

astefan mentioned this pull request Jun 12, 2025

ESQL: INLINESTATS implementation with multiple LogicalPlan updates #128917

Merged

		@@ -34,7 +34,6 @@ protected LogicalPlan rule(InlineJoin plan) {
		// check if there's any grouping that uses a reference on the right side

ESQL: Change the order of the optimization rules #124335

ESQL: Change the order of the optimization rules #124335

Uh oh!

Conversation

astefan commented Mar 7, 2025

Uh oh!

elasticsearchmachine commented Mar 7, 2025

Uh oh!

elasticsearchmachine commented Mar 7, 2025

Uh oh!

astefan Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

astefan Mar 7, 2025

Choose a reason for hiding this comment

Uh oh!

costin left a comment

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alex-spies Mar 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-spies Mar 11, 2025

Choose a reason for hiding this comment

Uh oh!

astefan commented Mar 12, 2025

Uh oh!

alex-spies Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

jan-elastic Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

alex-spies Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

alex-spies commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 13, 2025

💚 Backport successful

Uh oh!

Uh oh!

astefan Mar 7, 2025 •

edited

Loading

alex-spies commented Mar 12, 2025 •

edited

Loading