Improve set_rollback() behaviour

[adamchainz]

Description

Fixes #6921.

Added tests that fail before and pass afterwards.

Remove the check for connection.in_atomic_block to determine if the current request is under a transaction.atomic from ATOMIC_REQUESTS. Instead, duplicate the method that Django itself uses in BaseHandler.

This requires fetching the actual view function from as_view(), as seen by the URL resolver / BaseHandler. Since this requires request, I've also changed the accesses in get_exception_handler_context to be direct attribute accesses rather than getattr(). It seems the getattr defaults not accessible since self.request, self.args, and self.kwargs are always set in dispatch() before handle_exception() can ever be called. This is useful since request is always needed for the new set_rollback logic.

This also fixes a bug with multi-DB compatibility - previously set_rollback would only be called on the default DB when there are multiple DB's.

[adamchainz]

So there are a bunch of test failures, all:

AttributeError: 'NoneType' object has no attribute 'func'

This is all due to use of RequestFactory, which doesn't set resolver_match on its requests as it bypasses URL resolution. In a real world Django app, URL resolution doesn't get bypassed like this.

This could be fixed by moving all those tests to use the test Client, which does set resolver_match, but it's a lot of work.

[auvipy]

I can help with updating the tests, can you share your suggested changes for the tests? I can come up with a PR

[rpkilby]

This could be fixed by moving all those tests to use the test Client, which does set resolver_match, but it's a lot of work.

An off the cuff reaction, but I don't think we'd want to go this route. Ignoring the headache of updating the test suite, wouldn't this imply that users are unable to test their views with the request factory? Users would have to use the test client instead, right?

Would it be sufficient to do a best effort to get the non_atomic_requests flag? e.g., from this:

    view_func = request.resolver_match.func
    non_atomic_requests = getattr(view_func, '_non_atomic_requests', set())

to this (and document the deviation from the BaseHandler):

    try:
        non_atomic_requests = request.resolver_match.func._non_atomic_requests
    except AttributeError:
        non_atomic_requests = set()

[adamchainz]

An off the cuff reaction, but I don't think we'd want to go this route. Ignoring the headache of updating the test suite, wouldn't this imply that users are unable to test their views with the request factory? Users would have to use the test client instead, right?

You're right, it's probably not a good idea.

Would it be sufficient to do a best effort to get the non_atomic_requests flag?

I guess so - I've updated the PR accordingly.

[rpkilby]

Thanks for the update. Added to the 3.11 milestone.

[tomchristie]

I'm a bit wary of this. Why is connection.in_atomic_block not sufficient, since it seems like it'd be the right thing to be doing? What does connection.in_atomic_block return when inside a non_atomic_request decorated view?

Is it possible to isolate the multi-DB fix in this PR from the _non_atomic_requests change in the PR, or are they tightly linked?

[adamchainz]

Why is connection.in_atomic_block not sufficient, since it seems like it'd be the right thing to be doing? What does connection.in_atomic_block return when inside a non_atomic_request decorated view?

connection.in_atomic_block means "is the default DB in a transaction?" not "is any DB in a transaction started by ATOMIC_REQUESTS?"

This has a number of subtleties. connection.in_atomic_block can return True in a view with @non_atomic_requests if there is an atomic block from another source than ATOMIC_REQUESTS. This is most likely to be from Django's TestCase, but could also be a custom view decorator or middleware for managing transactions.

More information on the specific case I had was:

• Move a client's app to ATOMIC_REQUESTS = True on default DB because they had a number of errors due to not using transactions
• Wrap some views with @non_atomic_requests because they weren't safe for ATOMIC_REQUESTS, and instead manually decorate with @atomic internally
• Have unit tests for those views using django's TestCase, which sets up two atomic blocks around tests.
• Those unit tests crash. DRF's set_rollback() sees connection.in_atomic_block is True, despite the transaction not coming from ATOMIC_REQUESTS. Tests checking error paths in the views cause attempt to rollback the transaction from TestCase, which TestCase also tries to rollback, which breaks the whole test run due to unbalanced transactions.

Is it possible to isolate the multi-DB fix in this PR from the _non_atomic_requests change in the PR, or are they tightly linked?

The multi-DB fix comes "for free" because set_rollback() here now copies what Django does in BaseHandler. I think moving away from this is riskier than trying to make this patch focussed only on the single DB case.

I have monkey patched the implementation in this PR into my client's app to fix things there and there have been no issues for 3 months now.

[tomchristie]

Gotcha. Is there any more graceful way we can do connection.in_atomic_block on a per-db basis?

[tomchristie]

Eg, is something along these lines possible instead?...

for db in connections.all():
    if db.settings_dict['ATOMIC_REQUESTS'] and db.in_atomic_block:
        transaction.set_rollback(True, using=db.alias)

[adamchainz]

Yes that should work but it wouldn't fix my bug with testing the @non_atomic_requests views under TestCase

[tomchristie]

it wouldn't fix my bug with testing the @non_atomic_requests views under TestCase

Sorry, walk me through that. Do you mean would be broken for @non_atomic_requests views, or that it would be broken for test cases of @non_atomic_requests views?

[adamchainz]

The latter (TransactionTestCase should still work).

[tomchristie]

Okey dokes. So would we be able to update the PR to use the style in the comment above, and switch any test cases to TransactionTestCase if required?

[adamchainz]

Done.

However this doesn't fix #6921. I will still need that patch in place for my client because their tests use TestCase on @non_atomic_requests views. I remembered while writing that commit that even raise Http404 counts as an error on DRF ;)

[tomchristie]

"I will still need that patch in place for my client because their tests use TestCase on @non_atomic_requests views."

Okay, that seems to fall within documented expected behavior...

https://docs.djangoproject.com/en/2.2/topics/testing/tools/#django.test.TransactionTestCase "Django’s TestCase class is a more commonly used subclass of TransactionTestCase that makes use of database transaction facilities to speed up the process of resetting the database to a known state at the beginning of each test. A consequence of this, however, is that some database behaviors cannot be tested within a Django TestCase class. For instance, you cannot test that a block of code is executing within a transaction, as is required when using select_for_update(). In those cases, you should use TransactionTestCase."

[tomchristie]

I'd like to release 3.11, and I'm not clear enough about this one yet, so going to drop this from the milestone.

If we absolutely need to we can always include it on a point release so long as we make sure to handle the potential signature change on the exception handler in a backwards compat way.

[tomchristie]

Perhaps we ought to be a bit more specific here and link to the https://docs.djangoproject.com/en/2.2/topics/testing/tools/#django.test.TransactionTestCase docs?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[auvipy]

OK Adam, can you please come back to it and make it merge able again? And perhaps from where anyone should retake it?