ESQL: Enable async get to support formatting #111104

kanoshiou · 2024-07-19T14:00:09Z

An async request in text mode (POST _query/async?format=txt) previously returned an empty body without providing an async ID. I have now implemented the second approach mentioned in the issue #110926. Now, the async parameters (Async-ID & Async-running) are returned as headers.

However, I discovered an issue: when using the async ID from a text mode request to query /_query/async/{id}, the returned data format is still JSON. This might be a bug, and perhaps the response should be in plain text format, matching the initial request.

Closes #110926

elasticsearchmachine · 2024-07-23T08:04:50Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

bpintea · 2024-07-25T13:59:43Z

Thanks for your interest, @kanoshiou.

We'll want to make sure that all operations are possible and work in txt mode with the async querying, not just starting the query (see other async APIs); since if that's not an option for some reason, we'll want to reject the async "txt query" all together, rather than add the headers.

Besides that any change would require tests. For txt mode we'll want (besides possible unit tests) integration tests. Have a look at x-pack/plugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/RestEsqlTestCase.java.

kanoshiou · 2024-07-25T16:40:58Z

all operations are possible and work in txt mode with the async querying, not just starting the query

@bpintea I agree with this. I will work on making sure that when the request is in txt mode, /_query/async/{id} returns correctly. I will also add tests once I have completed all the code changes.

However, after reviewing the code for /_query/async/{id}, I realize that the changes required might be quite extensive. As a new contributor, I might need some additional time to implement these changes, and I would greatly appreciate any guidance.

Currently, /_query/async/{id} uses RestRefCountedChunkedToXContentListener. Would it be possible to simply replace it with EsqlResponseListener as used in /_query/async, or is there a better approach? For example, creating a new listener that extends RestRefCountedChunkedToXContentListener like EsqlResponseListener does and overrides the processResponse method. It seems I also need to store the initial request format in StoredAsyncResponse.

…support format and delimiter in params

This reverts commit e81487a.

…pe in request

kanoshiou · 2024-07-31T16:51:45Z

Hi @bpintea ,

I believe I have completed the code. Could you please review it when you have some time?

I've updated the listener for GET /_query/async/{id} to EsqlResponseListener, so it now accepts parameters (delimiter, drop_null_columns and format) like the POST /_query API. Additionally, I have added tests to verify the correctness of the code.

You can now set the format in the request parameters to specify the return style. I’m not sure if this meets your requirements. If there are any other issues with the code, please let me know, and I will continue to modify it.

…mode

…rns-as-headers-in-text-mode

kanoshiou · 2024-10-10T09:32:10Z

Hi @bpintea ,

I hope this message finds you well. I wanted to gently remind you about #111104. It has been a while since the submission, and I wanted to check if there are any updates or if any further actions are required from my side.

Thank you for your time and assistance!

bpintea · 2024-11-27T15:51:00Z

@kanoshiou sorry for getting back late.
Meanwhile, some of the issues covered in your PR had already been addressed, like the headers insertion, now also aligned with the _search API (see #111840).
However, not all of it, specifically the async get, which still doesn't support formatting, neither the originally asked for one, nor one requested as URL parameters (the format one). Worst, it doesn't even consider the Accept header formats. Your PR does address this and would be worth continuing.

...ugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/RestEsqlTestCase.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlResponseListener.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/EsqlMediaTypeParser.java

bpintea

Just some minor notes, looking good otherwise.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/EsqlMediaTypeParser.java

...ugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/RestEsqlTestCase.java

kanoshiou · 2024-12-04T10:33:38Z

@bpintea Thank you very much! I have updated the branch based on your comments.

…mode

bpintea · 2024-12-04T10:49:58Z

@elasticsearchmachine test this please

kanoshiou · 2024-12-04T12:04:07Z

Hi @bpintea, I would like to know how I should resolve these test failures, as I can't see the details.

Default to JSON in this case.

…eaders-in-text-mode

bpintea · 2024-12-05T10:17:41Z

I would like to know how I should resolve these test failures

I'm not sure how/if you can access the test results. But most of them will be revealed if you run locally ./gradlew -p x-pack/plugin/esql/ check. That would take a while, though since it runs BWC tests as well. You can probably run individual tests too (see ./gradlew tasks --all).
Besides that, another failed test was ./gradlew ":test:external-modules:test-esql-heap-attack:javaRestTest".

The tests failed because of missing media type indication. We're a bit inconsistent here, but ES allows a (search) GET with no media type indication whatsoever, so I need to revise my previous comment about supporting a default -- I've pushed a fix for that. However, I'd kindly ask you to add tests for this (i.e. an async GET with no media indication) then too, both RestEsqlTestCase and EsqlMediaTypeParserTests. Thanks.

bpintea · 2024-12-05T10:17:51Z

@elasticsearchmachine test this please

kanoshiou · 2024-12-05T10:25:56Z

@bpintea Thank you for letting me know.

I'd kindly ask you to add tests for this (i.e. an async GET with no media indication) then too, both RestEsqlTestCase and EsqlMediaTypeParserTests. Thanks.

Sure thing! I will implement the tests ASAP.

kanoshiou · 2024-12-06T09:32:05Z

@bpintea I've added the tests. Please review.

bpintea · 2024-12-06T18:06:30Z

@elasticsearchmachine test this please

…mode

bpintea · 2024-12-09T09:51:24Z

@elasticsearchmachine test this please

…mode

bpintea · 2024-12-09T10:07:07Z

@elasticsearchmachine test this please

bpintea

Looks good, thanks @kanoshiou.

bpintea · 2024-12-06T17:57:32Z

...ugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/RestEsqlTestCase.java

+
+    }
+
+    static Request prepareRequestWithOptions(RequestObjectBuilder requestObject, Mode mode) throws IOException {


bpintea · 2024-12-06T17:58:01Z

...ugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/RestEsqlTestCase.java

+        var json = entityToMap(entity, requestObject.contentType());
+        checkKeepOnCompletion(requestObject, json, true);
+        String id = (String) json.get("id");
+        // results won't be returned since keepOnCompletion is true


Nit: if they're not returned, it's because the waitForCompletion() is provided a very small interval (see addAsyncParameters()), so ES won't have the time to query and respond with results in time.

bpintea · 2024-12-06T17:58:25Z

...ugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/RestEsqlTestCase.java

+        // results won't be returned since keepOnCompletion is true
+        assertThat(id, is(not(emptyOrNullString())));
+
+        // issue an "async get" request with no Content-Type


elasticsearchmachine · 2024-12-09T12:10:12Z

💚 Backport successful

Status	Branch	Result
✅	8.x

I've updated the listener for GET /_query/async/{id} to EsqlResponseListener, so it now accepts parameters (delimiter, drop_null_columns and format) like the POST /_query API. Additionally, I have added tests to verify the correctness of the code. You can now set the format in the request parameters to specify the return style. Closes elastic#110926

kanoshiou · 2024-12-09T12:17:17Z

@bpintea, Thank you so much for taking the time to review my code. I truly appreciate your patience and attention to detail. This has been an unforgettable experience, and I have learned a lot.😊

I've updated the listener for GET /_query/async/{id} to EsqlResponseListener, so it now accepts parameters (delimiter, drop_null_columns and format) like the POST /_query API. Additionally, I have added tests to verify the correctness of the code. You can now set the format in the request parameters to specify the return style. Closes #110926 Co-authored-by: kanoshiou <[email protected]>

Async request in txt mode returns the async parameters as headers

462a8e5

elasticsearchmachine added needs:triage Requires assignment of a team area label v8.16.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jul 19, 2024

Add changelog

33e8b1f

gbanasiak added the :Analytics/ES|QL AKA ESQL label Jul 23, 2024

elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) and removed needs:triage Requires assignment of a team area label labels Jul 23, 2024

kanoshiou marked this pull request as draft July 26, 2024 01:52

kanoshiou added 8 commits July 30, 2024 22:59

Typo

f0285ea

Change listener of /_query/async/{id} to EsqlResponseListener to …

f77b893

…support format and delimiter in params

Change HTTP header names access modifiers to private

e81487a

Revert "Change HTTP header names access modifiers to private"

c27716f

This reverts commit e81487a.

Response media type defaults to JSON if there is no specific media ty…

9a010a5

…pe in request

Add tests for async query in text mode

12ef64d

Get requests do not need checkNonNullMediaType

b774b61

Update doc

c3c1396

kanoshiou marked this pull request as ready for review July 31, 2024 16:29

kanoshiou added 2 commits August 1, 2024 09:12

Merge branch 'main' into async-parameters-returns-as-headers-in-text-…

e94872a

…mode

Merge branch 'main' into async-parameters-returns-as-headers-in-text-…

3f48c01

…mode

mark-vieira added v9.0.0 and removed v8.16.0 labels Sep 11, 2024

Merge remote-tracking branch 'origin/main' into async-parameters-retu…

d6bd2e2

…rns-as-headers-in-text-mode

bpintea reviewed Nov 27, 2024

View reviewed changes

bpintea self-assigned this Dec 4, 2024

bpintea added >feature auto-backport Automatically create backport pull requests when merged v8.18.0 labels Dec 4, 2024

bpintea reviewed Dec 4, 2024

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/EsqlMediaTypeParser.java Outdated Show resolved Hide resolved

...ugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/RestEsqlTestCase.java Outdated Show resolved Hide resolved

Update

2904aa2

Merge branch 'main' into async-parameters-returns-as-headers-in-text-…

b0da5b5

…mode

bpintea added 2 commits December 5, 2024 10:54

Allow an ASYNC GET with no media type spec

444ed89

Default to JSON in this case.

Merge branch 'main' into fork/kanoshiou/async-parameters-returns-as-h…

389dd8a

…eaders-in-text-mode

Testcases for request with no Content-type header

5b02528

Merge branch 'main' into async-parameters-returns-as-headers-in-text-…

201a45d

…mode

Merge branch 'main' into async-parameters-returns-as-headers-in-text-…

0fb5625

…mode

bpintea reviewed Dec 9, 2024

View reviewed changes

bpintea approved these changes Dec 9, 2024

View reviewed changes

bpintea merged commit 67ee034 into elastic:main Dec 9, 2024
17 checks passed

bpintea mentioned this pull request Dec 9, 2024

[8.x] ESQL: Enable async get to support formatting (#111104) #118257

Merged


		}

		static Request prepareRequestWithOptions(RequestObjectBuilder requestObject, Mode mode) throws IOException {

ESQL: Enable async get to support formatting #111104

ESQL: Enable async get to support formatting #111104

Uh oh!

Conversation

kanoshiou commented Jul 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 23, 2024

Uh oh!

bpintea commented Jul 25, 2024

Uh oh!

kanoshiou commented Jul 25, 2024

Uh oh!

kanoshiou commented Jul 31, 2024

Uh oh!

kanoshiou commented Oct 10, 2024

Uh oh!

bpintea commented Nov 27, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bpintea left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kanoshiou commented Dec 4, 2024

Uh oh!

bpintea commented Dec 4, 2024

Uh oh!

kanoshiou commented Dec 4, 2024

Uh oh!

bpintea commented Dec 5, 2024

Uh oh!

bpintea commented Dec 5, 2024

Uh oh!

kanoshiou commented Dec 5, 2024

Uh oh!

kanoshiou commented Dec 6, 2024

Uh oh!

bpintea commented Dec 6, 2024

Uh oh!

bpintea commented Dec 9, 2024

Uh oh!

bpintea commented Dec 9, 2024

Uh oh!

bpintea left a comment

Choose a reason for hiding this comment

Uh oh!

bpintea Dec 6, 2024

Choose a reason for hiding this comment

Uh oh!

bpintea Dec 6, 2024

Choose a reason for hiding this comment

Uh oh!

bpintea Dec 6, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Dec 9, 2024

💚 Backport successful

Uh oh!

kanoshiou commented Dec 9, 2024

Uh oh!

Uh oh!

kanoshiou commented Jul 19, 2024 •

edited

Loading