BYWEEKNO recurrence issue

[minichma]

The following cases related to BYWEEKNO fail:

RRULE:FREQ=YEARLY;BYWEEKNO=1,2;UNTIL=20170101T000000Z
DTSTART:20130108T000000
Expected INSTANCES:20130108T000000,20131231T000000,20140107T000000,20141230T000000,20150106T000000,20160105T000000,20160112T000000
Actual   INSTANCES:20130115T000000,20130122T000000,20140114T000000,20140121T000000,20150113T000000,20150120T000000,20160112T000000,20160119T000000

RRULE:FREQ=YEARLY;BYWEEKNO=-1,-2
DTSTART:20131224T000000
Expected INSTANCES:20131224T000000
Actual   INSTANCES: *** MALFORMEDDATA: An input string was not correctly formed or a component has missing or extra properties

RRULE:FREQ=YEARLY;BYWEEKNO=-1,-2;COUNT=6
DTSTART:20270102
Expected INSTANCES:20270102,20271225,20280101,20281223,20281230
Actual INSTANCES:20271225,20280101,20281223,20281230,20291222,20291229

Warning

The RFC is somewhat ambiguous when it comes to BYWEEKNO. In particular it is unclear how to deal with recurrences in the first and last week of the year if those cross the year boundary. Different libraries implement this case differently. See related discussions here:

• Test cases related to BYWEEKNO + BYMONTHDAY, etc. could be wrong ical-org/ical.net#791
• https://stackoverflow.com/questions/48064349/yearly-rrule-interpretation-when-more-than-one-of-bymonth-byweekno-byyearday-a

The test cases in this issue are based on the implementation that libical has followed so far, i.e. that in case BYWEEKNO is present, the individual years are assumed to start on the first day of week no. 1 (and end the day before the first day of the following year). Whether or not there is consent on this behavior throughout the libical project may require further discussions.

Reproduced on Ubuntu with ICU and Windows without ICU.

These are more test cases related to WEEKNO which may or may not have the same root cause. I'm also not 100% sure that my interpretation of the expected outcome is correct. The test cases are based on the assumption that 2010-01-02 belongs to week 53 of 2009 (not 2010). So the yearly interval starts counting from 2009 and after 6y we are at 2015, not 2016.

# The first date belongs to 2009, so after 6 years it will be 2015, which has a week 53.
RRULE:FREQ=YEARLY;BYWEEKNO=53;BYDAY=TU,SA;INTERVAL=6;UNTIL=20170101T000000Z
DTSTART:20100102T000000
INSTANCES:20100102T000000,20151229T000000,20160102T000000

# The first date belongs to 2009, so after 5 years it will be 2014, which has NO week 53.
RRULE:FREQ=YEARLY;BYWEEKNO=53;BYDAY=TU,SA;INTERVAL=5;UNTIL=20170101T000000Z
DTSTART:20100102T000000
INSTANCES:20100102T000000

# The first date belongs to 2009, so after 7 years it will be 2014, which has NO week 53.
RRULE:FREQ=YEARLY;BYWEEKNO=53;BYDAY=TU,SA;INTERVAL=7;UNTIL=20170101T000000Z
DTSTART:20100102T000000
INSTANCES:20100102T000000

# DTSTART is in 2024 but the week belongs to 2025, so after 3 years it will be 2028.
RRULE:FREQ=YEARLY;BYWEEKNO=1;BYDAY=MO,TU;INTERVAL=3;UNTIL=20320101
DTSTART:20241231
INSTANCES:20241231,20280103,20280104,20301230,20301231

Test outcome:

Test case at line 478 failed.
RRULE:FREQ=YEARLY;BYWEEKNO=53;BYDAY=TU,SA;INTERVAL=6;UNTIL=20170101T000000Z
DTSTART:20100102T000000
Expected INSTANCES:20100102T000000,20151229T000000,20160102T000000
Actual   INSTANCES:

Test case at line 483 failed.
RRULE:FREQ=YEARLY;BYWEEKNO=53;BYDAY=TU,SA;INTERVAL=5;UNTIL=20170101T000000Z
DTSTART:20100102T000000
Expected INSTANCES:20100102T000000
Actual   INSTANCES:20151229T000000,20160102T000000

Test case at line 488 failed.
RRULE:FREQ=YEARLY;BYWEEKNO=53;BYDAY=TU,SA;INTERVAL=7;UNTIL=20170101T000000Z
DTSTART:20100102T000000
Expected INSTANCES:20100102T000000
Actual   INSTANCES:

Test case at line 493 failed.
RRULE:FREQ=YEARLY;BYWEEKNO=1;BYDAY=MO,TU;INTERVAL=3;UNTIL=20320101
DTSTART:20241231
Expected INSTANCES:20241231,20280103,20280104,20301230,20301231
Actual   INSTANCES:20270104,20270105,20291231,20300101

Some related tests have been added in #889 for cases that are explicitly marked as not implemented as of today.

[minichma]

see #794

[CMendia]

As you've pointed out here and in your comments at ical-org/ical.net#791, our difficulties stem from ambiguities in the RFC relating to BYWEEKNO handling. In particular, there is little information on how ISO 8601 weeks should interact with "normal" calendar years. (The interaction of BYMONTH and BYWEEKNO values in a yearly recurrence is also poorly specified, which is probably why we don't implement it, but I think that is a slightly different issue.)

I think there are two reasonably consistent (though inelegant) ways of interpreting the RFC:

1. We only consider weeks within a given calendar year, so 20100102 would not belong to any week because it is before the start of the calendar year's week numbering. In handling BYWEEKNO values, we would only consider dates from first day of the first week of a calendar year to the end of the calendar year; for example, for 2016, we would only generate dates between 2016-01-04 and 2016-12-31.
2. We handle BYWEEKNO values using ISO 8601 "week years", entirely separate from our handling of other BY values using calendar years. Under this interpretation, when we generate the recurrence set for a yearly recurrence starting at 20100102, which belongs to the ISO 8601 week year 2009, would go through the ISO 8601 week years 2009, 2010, etc. to match BYWEEKNO values, whereas we would go through the calendar years 2010, 2011, etc. to match BYMONTH values.

We currently do neither of these, which leads to the inconsistencies you've shown above. (I think the first two test cases were due to a genuine bug, though, which should be fixed by #941.) Instead, we iterate through calendar years and try to add on ISO 8601 weeks by generating occurrences outside the calendar year.

There are two parts of the RFC that I think could indirectly clarify things (or at least help to guess the intentions of the RFC authors):

The BYWEEKNO rule part specifies a COMMA-separated list of
ordinals specifying weeks of the year. Valid values are 1 to 53
or -53 to -1. This corresponds to weeks according to week
numbering as defined in [ISO.8601.2004]. A week is defined as a
seven day period, starting on the day of the week defined to be
the week start (see WKST). Week number one of the calendar year
is the first week that contains at least four (4) days in that
calendar year.

If multiple BYxxx rule parts are specified, then after evaluating
the specified FREQ and INTERVAL rule parts, the BYxxx rule parts
are applied to the current set of evaluated occurrences in the
following order: ...

(my emphasis)

In my view, both of these point towards interpretation # 1 above. The first quote does not define week numbers outside of the calendar year (although it does mention ISO 8601, it only says that the BYWEEKNO values "correspond" to the week numbering, which seems less definite). The second quote seems more decisive: as far as I can tell, there is no reason to interpret the evaluation of a yearly recurrence with an INTERVAL of n differently when BYWEEKNO is present, so the following test case

RRULE:FREQ=YEARLY;BYWEEKNO=53;BYDAY=TU,SA;INTERVAL=6;UNTIL=20170101T000000Z
DTSTART:20100102T000000

would first be expanded to obtain "every five years, starting with 2010", and then be filtered with BYWEEKNO, just like it would if we had no BYWEEKNO parameter. I believe JsCalendar also uses interpretation # 1.

However, there are a few practical reasons that I cannot be confident this is the correct interpretation. The first is that it would cause half of week 1 or week 52 to be "unreachable" by any recurrence rule. This does not really make sense to me, and seems undesirable for any user of recurrence by week number. The second is that any one who wanted to use ISO 8601 weeks would also probably want to use ISO 8601 week years, not calendar years. I do not think there is a situation where someone would want an event that occurred on the first ISO 8601 week of each year (presumably meaning ISO 8601 week years), except when that week did not occur at the start of the calendar year.

Perhaps interpretation # 1 is correct though, and iCalendar is not really intended to be used with ISO 8601 weeks for all practical purposes (in which case why was BYWEEKNO not left out?). The ISO 8601 week numbering scheme is quite different from calendar years, so that it would be very difficult to design a system where they worked together in an intuitive way.

Another issue (relevant to the examples above, I think) is that the RFC does not clearly indicate what information ought to be derived from DTSTART, so we don't know whether we should extract the weekno + week day, or perhaps month + month day (as one would normally expect). This also makes it difficult to add handling for yearly recurrences with, e.g., weekno and monthday.

[winterz]

I'll leave this one up to @CMendia and @minichma
I don't think this is a version4 release blocker. I might be wrong.

[winterz]

@CMendia would you be willing to work on interpretation #1 to solve this one?