[ES|QL] Support some stats on aggregate_metric_double #120343

limotova · 2025-01-17T05:17:19Z

Adds non-grouping support for min, max, sum, and count, using
CompositeBlock as the underlying block type and an internal
FromAggregateMetricDouble function to handle converting from
CompositeBlock to the correct metric subfields.

Closes #110649

Adds support for min, max, sum, and count

martijnvg

Thanks Larisa this look good! I left a few comments.

About the failing yamlRestCompatTest test suite, the following error is returned by aggregate double metric field mapper: Must have all subfields to use aggregate double metric in ESQL

The yamlRestCompatTest test suite runs 8.x versions of the same yaml test against current main in this branch. This error is now returned, because before only min and max metric were configured to be stored. This fails the assumption in aggregate double metric field mapper. Maybe be less strict here (see comment in AggregateDoubleMetricFieldMapper)?

x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DataType.java

.../compute/src/main/java/org/elasticsearch/compute/data/AggregateDoubleMetricBlockBuilder.java

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Count.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/AggregateMapper.java

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/CompositeBlock.java

...g/elasticsearch/xpack/esql/expression/function/scalar/convert/FromAggregateDoubleMetric.java

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java

dnhatn

I've left some comments, but the approach looks good. Thanks Larisa!

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/CompositeBlock.java

.../compute/src/main/java/org/elasticsearch/compute/data/AggregateDoubleMetricBlockBuilder.java

...g/elasticsearch/xpack/esql/expression/function/scalar/convert/FromAggregateDoubleMetric.java

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java

dnhatn

I left some comments and questions. Thanks Larisa for iterating on this.

.../compute/src/main/java/org/elasticsearch/compute/data/AggregateDoubleMetricBlockBuilder.java

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/CompositeBlock.java

.../compute/src/main/java/org/elasticsearch/compute/data/AggregateDoubleMetricBlockBuilder.java

dnhatn · 2025-01-27T18:49:38Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/BlockUtils.java

@@ -233,6 +233,17 @@ private static Block constantBlock(BlockFactory blockFactory, ElementType type,
            case BYTES_REF -> blockFactory.newConstantBytesRefBlockWith(toBytesRef(val), size);
            case DOUBLE -> blockFactory.newConstantDoubleBlockWith((double) val, size);
            case BOOLEAN -> blockFactory.newConstantBooleanBlockWith((boolean) val, size);
+            case COMPOSITE -> {


Hmm, a composite type can be more than aggregated_metric_double? Can we just leave it unsupported here?

I had to add this to be able to support the unit tests and wasn't really sure of a way to work around it

If composite can be more than just aggregated_metric_double, then should aggregated_metric_double have its own element type?

I'm not sure it really needs it's own type. Maybe there's something funny around constants though?

Maybe we instanceof on the val and do an AggregateMetricDouble if it's one of those constants. Again, this feels like it's the kind of thing we'd use for just tests and ROW. Which is ok.

dnhatn · 2025-01-27T19:01:39Z

.../esql/compute/src/main/java/org/elasticsearch/compute/data/AggregateMetricDoubleLiteral.java

+
+package org.elasticsearch.compute.data;
+
+public class AggregateMetricDoubleLiteral {


Did we introduce this because of tests?

Yeah, there's a couple of other places in tests where the need for something like this also popped up in the first attempt at implementing aggregate_metric_double

Maybe I'm misreading, but is this only needed for tests? If so should this be moved to the test sources?

Yeah it's only used in the tests so far but one function that required it is outside of the test code

That code was originally for ROW but we've used it more in tests since. I think, conceptually at least, we could use this thing for ROW support for aggregate metric double. It's a lot more convenient than a Map representation or something.

...sticsearch/xpack/esql/expression/function/scalar/convert/FromAggregateDoubleMetricTests.java

…(in that order)

martijnvg · 2025-01-28T15:06:11Z

.../esql/compute/src/main/java/org/elasticsearch/compute/data/AggregateMetricDoubleLiteral.java

+
+package org.elasticsearch.compute.data;
+
+public class AggregateMetricDoubleLiteral {


Maybe I'm misreading, but is this only needed for tests? If so should this be moved to the test sources?

martijnvg · 2025-01-28T15:09:36Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/BlockUtils.java

@@ -233,6 +233,17 @@ private static Block constantBlock(BlockFactory blockFactory, ElementType type,
            case BYTES_REF -> blockFactory.newConstantBytesRefBlockWith(toBytesRef(val), size);
            case DOUBLE -> blockFactory.newConstantDoubleBlockWith((double) val, size);
            case BOOLEAN -> blockFactory.newConstantBooleanBlockWith((boolean) val, size);
+            case COMPOSITE -> {


If composite can be more than just aggregated_metric_double, then should aggregated_metric_double have its own element type?

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/40_unsupported_types.yml

nik9000 · 2025-01-28T18:49:25Z

server/src/main/java/org/elasticsearch/index/mapper/BlockLoader.java

@@ -501,4 +503,10 @@ interface SingletonOrdinalsBuilder extends Builder {
         */
        SingletonOrdinalsBuilder appendOrd(int value);
    }
+
+    interface AggregateMetricDoubleBuilder extends Builder {


Interesting! This is not something in server but we have to add it here for this. Huh. I suppose that's ok.

server/src/main/java/org/elasticsearch/index/mapper/BlockLoader.java

.../esql/compute/src/main/java/org/elasticsearch/compute/data/AggregateMetricDoubleLiteral.java

nik9000 · 2025-01-28T18:54:09Z

.../esql/compute/src/main/java/org/elasticsearch/compute/data/AggregateMetricDoubleLiteral.java

+
+package org.elasticsearch.compute.data;
+
+public class AggregateMetricDoubleLiteral {


That code was originally for ROW but we've used it more in tests since. I think, conceptually at least, we could use this thing for ROW support for aggregate metric double. It's a lot more convenient than a Map representation or something.

nik9000 · 2025-01-28T18:55:36Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/BlockUtils.java

@@ -233,6 +233,17 @@ private static Block constantBlock(BlockFactory blockFactory, ElementType type,
            case BYTES_REF -> blockFactory.newConstantBytesRefBlockWith(toBytesRef(val), size);
            case DOUBLE -> blockFactory.newConstantDoubleBlockWith((double) val, size);
            case BOOLEAN -> blockFactory.newConstantBooleanBlockWith((boolean) val, size);
+            case COMPOSITE -> {


I'm not sure it really needs it's own type. Maybe there's something funny around constants though?

nik9000 · 2025-01-28T18:56:28Z

...ck/plugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/EsqlTestUtils.java

-            case UNSUPPORTED, OBJECT, DOC_DATA_TYPE, TSID_DATA_TYPE, PARTIAL_AGG -> throw new IllegalArgumentException(
-                "can't make random values for [" + type.typeName() + "]"
-            );
+            case UNSUPPORTED, OBJECT, DOC_DATA_TYPE, TSID_DATA_TYPE, PARTIAL_AGG, AGGREGATE_METRIC_DOUBLE ->


At some point we'll have to be able to make random AggregateMetricDoubles. But later is fine.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/PositionToXContent.java

...g/elasticsearch/xpack/esql/expression/function/scalar/convert/FromAggregateMetricDouble.java

server/src/main/java/org/elasticsearch/index/mapper/BlockLoader.java

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java

nik9000

I'm happy when the others are happy.

nik9000 · 2025-01-29T00:52:56Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/BlockUtils.java

@@ -233,6 +233,17 @@ private static Block constantBlock(BlockFactory blockFactory, ElementType type,
            case BYTES_REF -> blockFactory.newConstantBytesRefBlockWith(toBytesRef(val), size);
            case DOUBLE -> blockFactory.newConstantDoubleBlockWith((double) val, size);
            case BOOLEAN -> blockFactory.newConstantBooleanBlockWith((boolean) val, size);
+            case COMPOSITE -> {


Maybe we instanceof on the val and do an AggregateMetricDouble if it's one of those constants. Again, this feels like it's the kind of thing we'd use for just tests and ROW. Which is ok.

nik9000 · 2025-01-29T00:55:31Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Count.java

@@ -141,6 +144,9 @@ protected TypeResolution resolveType() {
    public Expression surrogate() {
        var s = source();
        var field = field();
+        if (field.dataType() == DataType.AGGREGATE_METRIC_DOUBLE) {
+            return new Sum(s, FromAggregateMetricDouble.withMetric(source(), field, AggregateMetricDoubleBlockBuilder.Metric.COUNT));


This is a really clear explanation of what this field is - COUNT(aggregate_metric_double) will is a SUM of the preaggregated counts. It's really quite educational.

It does ask "should we have methods to pick apart the sub-fields?" like GET_COUNT(aggregate_metric_double). Or something. Not now, but eventually? Like, if we want to treat the field like it's just a container of numbers.

Which is odd. We don't have container type fields until this one.

Wait, you already wrote it and called it FromAggregateMetricDouble. Of course! Do we plug it in? It looks like not. That's fine and good. But we should talk about if it's appropriate to expose it in the language as a function one day.

We plugged it in, but it is not yet exposed at the language level. I think we should consider exposing this method so that users can retrieve values from aggregate_metrics_double individually, as we currently don't have a good way to return all values at once. Let's address this in a follow-up.

nik9000 · 2025-01-29T00:58:28Z

...ck/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/LocalExecutionPlanner.java

@@ -372,7 +372,7 @@ private PhysicalOperation planTopN(TopNExec topNExec, LocalExecutionPlannerConte
                case GEO_POINT, CARTESIAN_POINT, GEO_SHAPE, CARTESIAN_SHAPE, COUNTER_LONG, COUNTER_INTEGER, COUNTER_DOUBLE, SOURCE ->
                    TopNEncoder.DEFAULT_UNSORTABLE;
                // unsupported fields are encoded as BytesRef, we'll use the same encoder; all values should be null at this point
-                case PARTIAL_AGG, UNSUPPORTED -> TopNEncoder.UNSUPPORTED;
+                case PARTIAL_AGG, UNSUPPORTED, AGGREGATE_METRIC_DOUBLE -> TopNEncoder.UNSUPPORTED;


This means you can't SORT if one of these is used later. That's fine for now, but may not be appropriate later.

Yeah I think that's something we want to support later on/not in this PR
and might require some discussion since I feel like it's not clear what we would sort on exactly

nik9000 · 2025-01-29T00:59:35Z

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java

@@ -288,7 +291,7 @@ public AggregateDoubleMetricFieldType(String name) {
        }

        public AggregateDoubleMetricFieldType(String name, Map<String, String> meta, MetricType metricType) {
-            super(name, true, false, false, TextSearchInfo.SIMPLE_MATCH_WITHOUT_TERMS, meta);
+            super(name, true, false, true, TextSearchInfo.SIMPLE_MATCH_WITHOUT_TERMS, meta);


Were we just incorrectly saying "no, this doesn't have doc values" before? That feels silly.

nik9000 · 2025-01-29T01:01:01Z

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java

+                        }
+                    }
+
+                    private void copyDoubleValuesToBuilder(Docs docs, BlockLoader.DoubleBuilder builder, NumericDocValues values)


I suppose you could reuse one of the other builders somehow maybe? Like the one that numerics uses for doubles. It has code just like this, right? Too tricky to share?

dnhatn

Great work. Thanks Larisa!

martijnvg

LGTM

martijnvg · 2025-01-29T14:23:43Z

.../esql/compute/src/main/java/org/elasticsearch/compute/data/AggregateMetricDoubleLiteral.java

+    private final Double sum;
+    private final Integer count;
+
+    public AggregateMetricDoubleLiteral(Double min, Double max, Double sum, Integer count) {


I think this can be turned into a record?

Yeah I think it can; I just did that and moved it to be under AggregateMetricDoubleBlockBuilder

elasticsearchmachine · 2025-01-29T22:10:02Z

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 120343

Adds non-grouping support for min, max, sum, and count, using CompositeBlock as the underlying block type and an internal FromAggregateMetricDouble function to handle converting from CompositeBlock to the correct metric subfields. Closes elastic#110649

) Adds non-grouping support for min, max, sum, and count, using CompositeBlock as the underlying block type and an internal FromAggregateMetricDouble function to handle converting from CompositeBlock to the correct metric subfields. Closes #110649

idegtiarenko · 2025-02-03T15:46:59Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/AggregateMapper.java

@@ -212,6 +212,9 @@ private static Stream<AggDef> groupingAndNonGrouping(Tuple<Class<?>, Tuple<Strin
        if (tuple.v1().isAssignableFrom(Rate.class)) {
            // rate doesn't support non-grouping aggregations
            return Stream.of(new AggDef(tuple.v1(), tuple.v2().v1(), tuple.v2().v2(), true));
+        } else if (tuple.v2().v1().equals("AggregateMetricDouble")) {


I am a bit puzzled by this condition.
I think (related pr: #121542) corresponds to extra configs. I do not think they are ever set to AggregateMetricDouble.
I also do not think we enter this branch in CsvTests.

Could you please help me understand when this is happening?

Okay, it looks like we define case AGGREGATE_METRIC_DOUBLE -> "AggregateMetricDouble"; below, but I do not think dataTypeToString is called when building AggDef here

You're correct we don't enter this branch; this PR was a 2nd/3rd iteration on adding aggregate metric double to ES|QL and it was necessary in an older iteration, and I mistakenly left it in in this PR. I plan to remove this in the next phase/PR

elasticsearchmachine added the v9.0.0 label Jan 17, 2025

limotova force-pushed the add-aggregate-double-metric-aggregates branch from a3123a7 to c12cf21 Compare January 17, 2025 06:57

[ES|QL] Support some stats on aggregate_double_metric

ae16694

Adds support for min, max, sum, and count

limotova force-pushed the add-aggregate-double-metric-aggregates branch from c12cf21 to ae16694 Compare January 18, 2025 04:14

limotova requested review from martijnvg and dnhatn January 18, 2025 08:31

martijnvg reviewed Jan 20, 2025

View reviewed changes

dnhatn reviewed Jan 22, 2025

View reviewed changes

addressed most comments minus tests

a5be73b

limotova force-pushed the add-aggregate-double-metric-aggregates branch from 0064bc5 to a5be73b Compare January 23, 2025 02:17

limotova added 3 commits January 23, 2025 12:38

Merge branch 'main' into add-aggregate-double-metric-aggregates

80b317a

FromAggregateDoubleMetricTests

a2a6092

Merge branch 'main' into add-aggregate-double-metric-aggregates

9094f62

limotova requested review from dnhatn and martijnvg January 27, 2025 16:33

dnhatn reviewed Jan 27, 2025

View reviewed changes

limotova and others added 8 commits January 27, 2025 13:14

bring back unsupported exceptions

b3b27aa

Merge branch 'main' into add-aggregate-double-metric-aggregates

5acb5a8

change valueCount to Integer and revert other getValueCount()

71a05b2

and revert getFirstValueIndex()

648e0af

test all metrics in aggregate double metric test

83612ec

refactor things touched in this PR to all be aggregate metric double …

38659b9

…(in that order)

[CI] Auto commit changes from spotless

bfe84c1

renaming stragglers

022eebe

limotova changed the title ~~[ES|QL] Support some stats on aggregate_double_metric~~ [ES|QL] Support some stats on aggregate_metric_double Jan 28, 2025

limotova mentioned this pull request Jan 28, 2025

[ES|QL] Aggregate Metric Double field type #119171

Closed

limotova marked this pull request as ready for review January 28, 2025 05:25

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Jan 28, 2025

limotova added >enhancement :StorageEngine/TSDB You know, for Metrics labels Jan 28, 2025

limotova added auto-backport Automatically create backport pull requests when merged and removed needs:triage Requires assignment of a team area label labels Jan 28, 2025

dnhatn requested a review from nik9000 January 28, 2025 05:38

fix changelog

6dd6e36

martijnvg reviewed Jan 28, 2025

View reviewed changes

limotova added 2 commits January 28, 2025 07:34

Merge branch 'main' into add-aggregate-double-metric-aggregates

5dd5ad9

Add downsampled yaml test

af0e242

nik9000 reviewed Jan 28, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/index/mapper/BlockLoader.java Outdated Show resolved Hide resolved

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapper.java Outdated Show resolved Hide resolved

limotova added 3 commits January 28, 2025 12:03

address some comments

1838a82

change Double etc to double

335fe3b

Merge branch 'main' into add-aggregate-double-metric-aggregates

5608bce

nik9000 approved these changes Jan 29, 2025

View reviewed changes

dnhatn approved these changes Jan 29, 2025

View reviewed changes

limotova added 2 commits January 28, 2025 15:34

add instanceof check

c417b3c

Merge branch 'main' into add-aggregate-double-metric-aggregates

b5837f3

martijnvg approved these changes Jan 29, 2025

View reviewed changes

limotova added 3 commits January 29, 2025 09:25

Merge branch 'main' into add-aggregate-double-metric-aggregates

fba979f

change AggMetDoubLit to record

827eaf3

move function to make checkstyle work

fbdb24d

limotova merged commit bcd8d15 into elastic:main Jan 29, 2025
16 checks passed

limotova deleted the add-aggregate-double-metric-aggregates branch January 29, 2025 22:08

elasticsearchmachine added the backport pending label Jan 29, 2025

limotova removed the backport pending label Jan 30, 2025

alex-spies mentioned this pull request Jan 30, 2025

ESQL: Enable LOOKUP JOIN in non-snapshot builds #121193

Merged

idegtiarenko reviewed Feb 3, 2025

View reviewed changes

limotova mentioned this pull request May 2, 2025

Add support for Downsampling in ES|QL #120759

Open

15 tasks


		package org.elasticsearch.compute.data;

		public class AggregateMetricDoubleLiteral {

[ES|QL] Support some stats on aggregate_metric_double #120343

[ES|QL] Support some stats on aggregate_metric_double #120343

Uh oh!

Conversation

limotova commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dnhatn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dnhatn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nik9000 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

limotova commented Jan 17, 2025 •

edited

Loading

limotova Jan 29, 2025 •

edited

Loading

idegtiarenko Feb 3, 2025 •

edited

Loading