So, through trial and error, I have learned a case in our codebase where we can run into TransactionManagementErrors (I’ll refer to them henceforth as "TME"s for short), and I have fixed the bugs that cause them to happen. It took me quite some time to understand what was really going on, and I think part of my confusion related to the wording of the error, which referred to not being able to execute queries until the end of the atomic transaction block. I found that confusing, because we have tons of queries that occur under the context of an atomic transaction that raise no exceptions at all.
Anyway, I worked out that in the vast majority of our cases, the error happened when a query traversed a reverse relation before the record was saved (which was happening in some @property methods we have to keep changing data up-to-date.
Last week, I found an edge case that produced a TME. It stemmed from an exception raised during a save that contained an invalid value. Our code saves the error and tries to find more errors (to reduce iterations of data debugging). I didn’t dig too deep on it, but it seemed like every query after that exception raised during the save would produce a TME.
In a general sense, that makes sense to me. I fixed the bug that attempted to load invalid data, but this scenario made me curious about the ultimate nature of TMEs. I read that there are some ORM features that can(/could?) raise TMEs because of auto-commit behavior. That also makes sense.
However, when I started debugging the issue last week, I couldn’t recall my original understanding about traversing reverse relations that I had worked out over a year ago, and I’d had trouble finding where I’d jotted it down. Usually, I retain what I’ve learned better when I understand things at a deeper level, so I’m wondering if there’s a better conceptual way I should use to think about TMEs so that I can better understand how they get raised?
Like in the case of the exception during save I encountered last week (which incidentally was about trying to set a number field to an empty string): what about that exception caused subsequent queries to raise a TME? Is it a state issue that had to do with staging commits?
I think this is the case covered by the note section titled “Avoid catching exceptions inside atomic !” in the docs for Controlling transactions explicitly. Specifically:
If you attempt to run database queries before the rollback happens, Django will raise a TransactionManagementError.
Does this sound like what you’re looking for? (The entire note, not just this sentence.)
It’s helpful in a general sense, as we do adhere to its advice:
If necessary, add an extra atomic block for this purpose.
And it’s informative as to the potential for unexpected behavior. Incidentally, we don’t handle (i.e. try to work around) the errors. We just collect errors and raise them at the end of the load that reads the input file. We skip any data associated with errors (e.g. IntegrityErrors). If any error occurs, we always eventually raise an exception.
But it doesn’t answer the question I’ve asked here. What it answers is how to treat the errors. It doesn’t help me understand the error better or the scenarios that raise the error.
For example, it doesn’t explain anything about how it will be raised if you attempt to traverse a reverse relation in a model object before it has been saved, which is something I had to work out on my own - and that wasn’t a situation where any other exception occurred, so it’s not helpful in debugging why a TME is raised without any other preceding exception. It just mentions not being able to execute queries until the end of the atomic transaction block - which, without any other information (from the perspective of a developer using Django), makes no sense, because there are lots of queries inside an Atomic Transaction.
That section and other documentation sections focus on providing guidance about what to do, but it doesn’t go into a lot of depth as to the how or why WRT TMEs. Like, what in the documentation would allow someone to understand TMEs coming from traversing reverse relations in an unsaved model object. I’m sure someone may be able to work it out (like I did). It’s just rather confusing.
Issue of terminology here: In this context, the word “handle” here means “catch”, not “work around”. By you catching the exception, you’re preventing Django from doing the transaction cleanup before you issue your queries.
If you’re doing any db queries inside the “except” block, the transaction has not yet been rolled back.
In other words, I would expect this:
with transaction.atomic():
try:
some db operation that fails
except:
issue a query
to throw a TPE at issue a query. Why? Because the transaction that I’m currently in has not yet been rolled back. (And notice how this is different from the examples in that section of the docs.)
What the docs are saying is that you should be doing this:
try:
with transaction.atomic():
some db operation that fails
except:
issue a query
That is my understanding of what the green note box is saying.
Yes, if an error does not occur, you can do whatever you want within the transaction block. But once you’ve thrown an exception, you shouldn’t do anything else related to the database until that transaction has been rolled back.
Note: This is something that is enforced at the database level in PostgreSQL. If you have an error inside a transaction, subsequent SQL statements will error until you rollback that transaction.
I just realized something. When I said:
we do adhere to its advice:
If necessary, add an extra atomic block for this purpose.
What I was referring to (and was incorrectly inferring from the doc), was that this was talking about an outer atomic transaction block for the ultimate rollback of everything processed in our input file loop. But reading between the doc lines and what you’ve said, I think one way of interpreting what you’re saying (though I’m not certain this is what you intended to convey), is that we should have a catch and rollback inside the loop where we’re iterating over the lines of the input file. Rollback the one line of data loaded and then continue to find more errors…
That’s great. But I feel like it still doesn’t help me understand all the situations that can cause TME’s. The example I’m citing (traversing a reverse relation on an unsaved model object) doesn’t seem to require a different exception to have already been raised… But I suppose it’s possible I’m wrong about that. I don’t think I am though. I refactored my code to avoid the possibility of the derived class property methods from executing when the model object hasn’t been saved yet and it prevented those TME errors.
So I think there’s more to it than what you do in an except block - and that’s what I’d like to understand better.
You’re right on both counts.
-
No, I wasn’t intending to say that, because it hadn’t really sunk in that this is what you were doing.
-
Yes, this should allow you to process all lines to catch all errors - at which point you rollback the outer transaction to prevent any of the processed lines from being added/updated.
I don’t see anywhere in the Django source code where this would cause that exception to be thrown - at least not as the first exception. If you search for raise TransactionManagementError, you’ll see all the conditions where it can occur, and none of them are associated with the related object manager.
What I could imagine is that if you have multiple such relationships that you are traversing in a single expression, one might throw a more expected error (ValueError in the case of a reverse lookup on an unsaved instance), and the second then throwing the TME. But that’s conjecture - I’m not sure how that would be tested aside from looking at the actual query being issued and trying it independently in the database’s shell.
A little, but not much.
What I’m seeing includes the possibility that it’s thrown by a select_for_update or rollback call outside a transaction, using on_commit in a case where you’re manually managing the transaction, and trying to execute DDL in a transaction on databases that don’t support it.
(At least this is as far as it goes with the Django-provided backends. There’s always the possibility that a third-party package uses that exception for other purposes.)
OK. I’ll go with the assumption that you’re right and that my recollection is faulty about those TMEs related to reverse relation queries. There must have been a previous exception that caused the TMEs to get raised when those reverse relations were later traversed - and I’m just not recalling it. It was a year or more ago when I implemented that fix.
In fact, looking at that code code from a year ago, it does perform a rollback. It’s different code from the infile loop that raised the TMEs last week. Creating those in-loop transaction blocks should make the overall strategy of trying to catch as many errors in 1 go as possible really solid.
Thanks for the help.