Support partial sort fields in TopN pushdown

[craigtaverner]

The work in #112938 allowed for pushing down SORTS to Lucene even when the sort expression came from an EVAL, as long as is was possible to push it down. It laid the groundwork for partial pushdown, where if the first n expressions could be pushed down, but not all could, we should be able to push down those n expressions. Since the coordinator-node TopNExec would still exist to merge partially sorted results from data nodes, this is sufficient to also perform the final sort on partially sorted data from Lucene. This feature therefor only required removing the limitation.

Benchmark	before this PR	after this PR	Comments
distanceSort-esql	180	180
distanceSort-esql-partial	14467.9	180
distanceFilterSort-esql	26	26
distanceFilterSort-esql-partial	5800	25

The four queries benchmarked here are:

• distanceSort-esql

FROM osmgeopoints
| EVAL distance = ST_Distance(location, TO_GEOPOINT("POINT(7.0 55.0)")),
| SORT distance ASC
| LIMIT 10

• distanceSort-esql-partial

FROM osmgeopoints
| EVAL distance = ST_Distance(location, TO_GEOPOINT("POINT(7.0 55.0)")),
       loc = location::string
| SORT distance ASC, loc ASC
| LIMIT 10

• distanceFilterSort-esql

FROM osmgeopoints
 | EVAL distance = ST_Distance(location, TO_GEOPOINT(\"POINT(7.0 55.0)\")),
| WHERE distance <= 400000
| SORT distance ASC
| LIMIT 10

• distanceFilterSort-esql-partial

FROM osmgeopoints
 | EVAL distance = ST_Distance(location, TO_GEOPOINT(\"POINT(7.0 55.0)\")),
        loc = location::string
| WHERE distance <= 400000
| SORT distance ASC, loc ASC
| LIMIT 10

Fixes #114515

[elasticsearchmachine]

Hi @craigtaverner, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

Hi @craigtaverner, I've updated the changelog YAML for you.

[elasticsearchmachine]

💔 Backport failed

The backport operation could not be completed due to the following error:

An unexpected error occurred when attempting to backport this PR.

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 116043

[alex-spies]

@craigtaverner , it looks like the backport broke and this never made it into 8.x

Should we backport this to 8.x (label says 8.17, but that doesn't look necessarily right due to the parity between 8.18 and 9.0)?

[craigtaverner]

Did the backport manually in #129199