feat: add a user-configurable timeout parameter to the _resolve/cluster API

[pawankartik-elastic]

Previously, should a remote be unresponsive, _resolve/cluster would wait until netty stepped in and terminated the connection. We initially responded to this issue by switching the disconnect strategy. However, this was problematic because it defeated the whole purpose of this API call—re-establish connection if and when possible. We now attempt to respond to it by adding a user-configurable GET parameter. This PR also reverses the problematic disconnect strategy.

Example:

GET _resolve/cluster/*:*?timeout=4s

[elasticsearchmachine]

Hi @pawankartik-elastic, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[quux00]

First round feedback.

In addition to specific code comments, two other notes:

1. I don't see any tests for this. Can we add automated tests?

2. Did you do any manual testing? In past experience with doing timeout based returns, I've seen errors/instability in Elasticsearch when the transport action ends earlier while an outstanding network call is still pending. For example - let's add a manual "sleep" to remote clusters that lasts longer than the timeout. What does the output look like for those and do they report errors in the logs or have instability issue when they actually return a response to a coordinator that is no longer listening.

[quux00]

Next round review.

[quux00]

The Description also still references a 9 second timeout (in the error message), so I'm confused about 9s vs 30s internal transport/network timeouts.

Didn't mean to approve, just comment

Comments addressed

[quux00]

Good test improvement. Additional suggestions left.

[pawankartik-elastic]

Cleaning up the description and dropping additional info here.

Q: What happens on a timeout?
User sees a response similar to this:

{
    "remote1": {
        "connected": false,
        "skip_unavailable": false,
        "error": "Request timed out before receiving a response from the remote cluster"
    }
}

where the "error" field denotes that the request timed out.

Then:
Scenario 1: If the remote continues to stay unresponsive, transport layer code then logs a warning (since the connection gets cut off due to handshake exception):

[WARN ][o.e.t.SniffConnectionStrategy] [node-1] fetching nodes from external cluster [remote1] failedorg.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9301] handshake_timeout[SOME_TIMEOUT_VALUE_IN_SECONDS]

Scenario 2: If the remote responds, the timeout has already happened by then (by calling into onFailure()) which sets the isDone variable to true in the timeout listener. Should the remote respond back, since isDone is set, no attempt is made to utilise the result since the listener is perceived as consumed/used. In this case, the transport layer logged nothing during the testing since the remote connection is fine from its perspective.

[quux00]

Approved.

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 120542

[pawankartik-elastic]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation