Epoch Millis Rounding Down and Not Up

[john-wagster]

Related to: #109542

Dug into the above mentioned bug wherein a date range check for ">" was failing because the underlying date was being parsed as off by 1 millisecond in some cases.

operations to reproduce

curl -XPUT --header 'Content-Type: application/json' "http://localhost:9200/test" -d '{
  "mappings": {
    "properties": {
      "mydate": {
        "type": "date",
        "format": "yyyy-MM-ddXXX"
      }
    }
  }
}'

curl -XPUT --header 'Content-Type: application/json' "http://localhost:9200/test/_doc/1?refresh" -d '{
  "mydate": "1969-12-31Z"
}'

# 1969-12-31Z as ms since epoch incorrectly returns 1 hit
curl -XGET --header 'Content-Type: application/json' "http://localhost:9200/test/_search?filter_path=hits.total.value" -d '{
  "query": {
    "range": {
      "mydate": {
        "gt": -86400000,
        "format": "epoch_millis"
      }
    }
  }
}'

I found that this was because negative epoch milliseconds were being rounded down by 999_999 nano seconds rather than up as per the missing date component logic documentation

This was because naively when nano seconds are defaulted to 999_999 we subsequently parse a negative epoch milli value and have to convert negative epoch millis into a (seconds, nanos) tuple. The math to do so never accounted for the intention of rounding and instead correctly was using the intended behavior of fractional milliseconds and pushing the time value toward negative infinity. For rounding up though this was likely not intended.

To compensate for this I've introduced a small hack wherein we detect the intention of rounding up as the default behavior by setting the default to an invalid value of -999_999 instead of 999_999. This allows the parser to detect whether the date was intended to round up by 999_999 or shift toward negative infinity by 999_999 fractional milliseconds.

[elasticsearchmachine]

Hi @john-wagster, I've created a changelog YAML for you.

[john-wagster]

@elasticmachine update branch

[john-wagster]

@elasticmachine update branch

[john-wagster]

@elasticmachine update branch

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Hi @john-wagster, I've updated the changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

Taking a fresh look at this, I wonder if this line is the root of the problem. Why do we extract nanos after making secondsAndMillis negative? If we extracted the millis portion before negation, it would always be positive, and we could always add nanos to it? I think it would be much easier to reason about if we didn't have these intermediate negatives since ultimately nanos must be positive.

[rjernst]

If we make nanos always positive as suggested above, I don't think it matters whether nanos was defaulted or came from the parse, we should always add it? Then the original "nanos != 0" branch below should work as intended (with a slight modification since it would now be positive), and I don't think we need roundUp at all?

[john-wagster]

here's what I tried based on what you were suggesting and this causes failures in the parsing, which I need to look into some more (try to do that if I can get a bit of time between now and Monday):

@@ -187,23 +187,14 @@ class EpochTime {
             long seconds;
             long nanos;
             if (isNegative != null) {
+                nanos = secondsAndMillis % 1000 * 1_000_000;
                 secondsAndMillis = -secondsAndMillis;
                 seconds = secondsAndMillis / 1_000;
-                nanos = secondsAndMillis % 1000 * 1_000_000;
                 // `secondsAndMillis < 0` implies negative timestamp; so `nanos < 0`
                 if (nanosOfMilli != null) {
-                    if (roundUp) {
-                        // these are not the nanos you think they are; these are "round up nanos" not the fractional part of the input
-                        // this is the case where we defaulted the value to 999_999 and the intention for rounding is that the value
-                        // moves closer to positive infinity
-                        nanos += nanosOfMilli;
-                    } else {
-                        // aggregate fractional part of the input; subtract b/c `nanos < 0`
-                        // this is the case where the user has supplied a nanos value and we'll want to shift toward negative infinity
                         nanos -= nanosOfMilli;
-                    }
                 }
-                if (nanos < 0) {
+                if (nanos != 0) {
                     // nanos must be positive. B/c the timestamp is represented by the
                     // (seconds, nanos) tuple, seconds moves 1s toward negative-infinity
                     // and nanos moves 1s toward positive-infinity

[john-wagster]

to clarify I tried both nanos += nanosOfMilli; and nanos -= nanosOfMilli;

[john-wagster]

Looked into this some more (and will probably spend a little more time tomorrow to better explain it), but the problem with this suggestion is that something like -123.123456 (without rounding) gets an incorrect value of nanos without the above selection logic with a negative nanos essentially it adds nanos when it should have subtracted. Probably easier to walk through it together.

[john-wagster]

discussed w @rjernst a good bit here and the game plan right now is to hold this PR open for a couple of days to noodle on any clean up / optimization to this code and otherwise to move forward with what we have here.

[john-wagster]

@elasticmachine update branch

[john-wagster]

@elasticmachine update branch

[john-wagster]

@rjernst can I get a +1 on this PR whenever it's convenient if it all looks good to you at this point and I'll wait to merge till Thursday in case there's any good suggestions or iteration here.

[rjernst]

LGTM. Thanks for splitting out the tests.

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.x