Fix async stop sometimes not properly collecting result

[smalyshev]

If the request finishes before the stop is called, but the result is not stored yet, we rely on async functionality to collect the response. I though that we are guaranteed that the data would be stored before the final handler is completed, but looks like it's not always the case. So this explicitly checks the task is finished before trying to collect.

Fixes #121249

[elasticsearchmachine]

Hi @smalyshev, I've created a changelog YAML for you.

[elasticsearchmachine]

Hi @smalyshev, I've updated the changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[dnhatn]

@smalyshev
I think the problem is that we wrap runAfter to remove the async listener too early:

elasticsearch/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/TransportEsqlQueryAction.java

Line 222 in 2fbec77

asyncListeners.put(asyncExecutionId, subListener);

.

This is because the listener does not include storing results to the index:

elasticsearch/x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/async/AsyncTaskManagementService.java

Line 191 in 67ee034

wrapStoringListener(searchTask, waitForCompletionTimeout, keepAlive, keepOnCompletion, listener)

.

We could avoid this issue if the runAfter wrap before dispatch to request to asyncTaskManagementService

elasticsearch/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/TransportEsqlQueryAction.java

Line 186 in 2fbec77

listener

). However, at that point, we don't yet have the async ID or executionInfo.

I'm a bit hesitant about this proposal in this PR because it would deduplicate the get-result logic more.

[smalyshev]

@dnhatn We don't change much logic-wise - we fetch the task ID anyway, the only change is that we'd wait for it to finish on one of the branches. As you note, if we move listener wrap earlier, we'd have to also move execution info creation and it may cause other problems?

[dnhatn]

One ask, but LGTM. Thanks @smalyshev.

[dnhatn]

Since this is related to security, can we expose and use AsyncTaskIndexService#getTaskAndCheckAuthentication instead of duplicating it here?

[smalyshev]

Oh, I didn't notice this exists. This is even better.

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.18	Commit could not be cherrypicked due to conflicts
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 121843

[smalyshev]

💚 All backports created successfully

Status	Branch	Result
✅	8.x
✅	9.0
✅	8.18

Questions ?

Please refer to the Backport tool documentation