Fix ordering duplication on non-unique field

[TheSuperiorStanislav]

Description

Always add "pk" as last parameter for ordering.
For example, we have following QS

pk	name
1	Bob
2	Dan
3	Bob
4	Joe

And we want to order it by name and take 2nd element.
We can get following results: qs.order_by('name')

pk	name
1	Bob
3	Bob
2	Dan
4	Joe

Or we can get:

pk	name
3	Bob
1	Bob
2	Dan
4	Joe

As you see, QS is correctly ordered, but order is not consistent, and
pagination is also inconsistent since we are using fields which doesn't
have unique restriction. So we always add "pk" as last ordering param.

Unfornualy I couldn't reproduce this behavior for tests, might be the is that in tests sqlite in memory is used. But on postgress result is stable. Here is an example:

from django_extensions.db.models import TimeStampedModel


class BaseModel(TimeStampedModel):
    """Base model for apps' models."""

    class Meta:
        abstract = True

class Genre(BaseModel):
    """Model defines data for genre from MAL."""
    name = CICharField(
        max_length=255,
        unique=True,
        verbose_name=_("Name"),
    )

    class Type(models.TextChoices):
        """Define possible genre types."""

        GENRES = "GENRES", _("Genres")
        EXPLICIT_GENRES = "EXPLICIT_GENRES", _("Explicit genres")
        THEMES = "THEMES", _("Themes")
        DEMOGRAPHICS = "DEMOGRAPHICS", _("Demographics")

    type = models.CharField(
        choices=Type.choices,
        max_length=15,
        verbose_name=_("Type"),
    )

    class Meta:
        verbose_name = _("Genre")
        verbose_name_plural = _("Genres")

print(
    list(Genre.objects.order_by("type").values_list("pk", "type")[0:2]), 
    list(Genre.objects.order_by("type").values_list("pk", "type")[2:4])
)
# Prints
# [(1454, 'DEMOGRAPHICS'), (1455, 'DEMOGRAPHICS')] [(1454, 'DEMOGRAPHICS'), (1460, 'DEMOGRAPHICS')]
print(
    list(Genre.objects.order_by("type", "pk").values_list("pk", "type")[0:2]), 
    list(Genre.objects.order_by("type", "pk").values_list("pk", "type")[2:4])
)
# Prints
# [(1750, 'DEMOGRAPHICS'), (1763, 'DEMOGRAPHICS')] [(1778, 'DEMOGRAPHICS'), (1785, 'DEMOGRAPHICS')]

[kevin-brown]

I'm not entirely convinced that this is desired behavior. Yes, the ordering without pk is unexpected but arguably what's worse is not being able to rely on the ordering that you set in a view is the ordering that you get on the queryset level.

If anything, I'd be more in favor of calling this out in the ordering documentation as something people should be aware of in case of equal ordering.

[auvipy]

I'm not entirely convinced that this is desired behavior. Yes, the ordering without pk is unexpected but arguably what's worse is not being able to rely on the ordering that you set in a view is the ordering that you get on the queryset level.

If anything, I'd be more in favor of calling this out in the ordering documentation as something people should be aware of in case of equal ordering.

-- From Kevin Brown's review

[TheSuperiorStanislav]

I'm not entirely convinced that this is desired behavior. Yes, the ordering without pk is unexpected but arguably what's worse is not being able to rely on the ordering that you set in a view is the ordering that you get on the queryset level.

If anything, I'd be more in favor of calling this out in the ordering documentation as something people should be aware of in case of equal ordering.

So how should this one be fixed outside drf? Tell front-end to always add pk to ordering? Create custom OrderingFilter class which will add pk to the end? This problem is just not obvious until you actually encounter it)

[kevin-brown]

So how should this one be fixed outside drf?

This very much so depends on who is the consumer of your API and if your API is new or existing. Let's walk through the options given by you.

Tell front-end to always add pk to ordering?

This is the standard move in APIs that are already existing. You don't necessarily tell them to add pk (unless you're exposing that directly), you tell them to add id or my_object_id or whatever identifier field you are exposing. DRF supports this out of the box today, no new release needed. And this is how a lot of APIs work that you'll find out on the web.

Create custom OrderingFilter class which will add pk to the end?

If you're the only consumer, you can break your contract, or you're working with a brand new API, this is probably the way to go.

The challenge here is that DRF expects that ordering fields will be unique or nearly unique. And for cases where this matters a lot, such as for cursor pagination, DRF tries to detect the common cases where this goes wrong (double underscore ordering fields) and raise a warning about it. But in most cases the uniqueness of the ordering is critical so DRF leans towards flexibility instead.

[auvipy]

closing as per design decision by maintainers as not needed